The main differences are:

1. MessagePack includes less data types. Several of the BSON-only data types are specific to MongoDB, but others - such as separate types for binary data and UTF-8 text - are IMO quite useful.

2. MessagePack includes several optimizations in the "type byte" to efficiently represent small values.

A while back I designed my own serialization format that combined features from both, but I never finished an implementation.

do you have a specification or even some prototype on github? would be interesting to see

I dug the spec I wrote off my backup hard disk and posted it here: https://gist.github.com/2907123

I also planned to define a standard set of tags for the "tagged value" type, but never quite got around to that.

BSON is really only used by MongoDB.