I agree with you. Partly this is because mobile browser vendors have made some really annoying decisions (zoom doesn't reflow text).

But an upvote gives you an [unvote] button, a downvote gives you an [undown] button.