I can't find the link, unfortunately, but the Congressional Budget Office released an analysis a few years ago comparing the FairTax with a flat tax and I believe another proposal (possibly a VAT system) and found that, while the FairTax would greatly benefit the poor (via the flat "prebate" checks) and the rich (because of the elimination of capital gains taxes and the fact that they spend a smaller proportion of their income), it would increase the burden on the middle class substantially. That kind of killed it for me.

Edit: Found it. It wasn't the CBO, it was a Bush-appointed advisory panel. See the "Chapter Eight - Nine" pdf, page 20:

http://govinfo.library.unt.edu/taxreformpanel/final-report/i...