Much of UX design is suck. People can't tell a difference, not at least in detail. If I were to change color of a button, maybe you'd think no one cared.
I think you just proved the point, most people will not notice, nor care, about a change in fonts from one family to another if they are similar.
If one were to change a button from red to a slightly different red, which is a detail, then most will not notice nor care. Change it from red to green then there could possibly be a change in response; be it good or bad.
Even if they don't care or consciously notice, the small details do make the experience. Even the reader who can't name even one font of knows what serifs are will experience the different fonts differently. One site will make his reading a little less comfortable, maybe make him tense some small facial muscles, and after a few minutes he just closes the tab because for some reason he doesn't even care to articulate he just feel slightly annoyed with the site. Another site has a design with a calming and harmonic color scheme, few distractions, a font that can be read without strain and our reader finishing his reading in a good mode and adds the site to his RSS feed. :)
Of course the most perfect site won't save crappy content, but the ugly site's content may never be read. It's similar to all the non verbal clues in human interaction. The tone of your voice and even the color of your shirt do make a difference.
But you are talking about several elements across the board that I would agree would have some sort of impact.
I'm just saying that changing from one font to another that are almost identical, and to the layman are identical, will not have much impact in the long run.
But such things do make a difference.