Yep, I was doing it wrong. You're supposed to pass a state object as the first parameter, I was passing null. So this fixes 2 things, prevents me from having to do XHR on initial page load AND I don't need to do an XHR on back/forward navigation either, I can just use the state object to store my data. Awesome!
See edit, try checking for the state attribute on your event object to filter out spurious popstate events.
An other option would be to use a shim library (History.js) handling that kind of crap (and smoothing out implementation details issues) for you.
[0] http://www.google.com/support/forum/p/Chrome/thread?tid=28ed... [1] http://hacks.mozilla.org/2011/03/history-api-changes-in-fire...