Post author here. You're dead on about this. I could try to turn this into another reason why git + pathogen is lacking as a plugin management system (you have to know a lot about git!), but that is weak, so a better tack is to just agree.
Question: is there an equally easy way to delete submodules? They're easy as pie to add, and you rightly point out that they are easy to update, but I never figured out an easy way to remove them. I was likely missing something, though.
