As an IT guy, I see hundreds of these connectors on a regular basis, so I've noticed something...if it's falling out too easily, odds are that the port has picked up a small piece of debris somehow, and the magnet isn't getting good contact. Take a close look at the plug and the port and brush them out a bit, and see if that helps.