The downside, for me, is that I very consciously enable tracking this way for mailings where I want my reads tracked.

When you actually want to be marketed to, it makes it easier for the sender to tailor their content -- and I do for some markets want to get the tailored stream.