This is the way of thinking about it that works best for me. As you fall into the gravity well of the planet from behind it, the planet is moving in the same direction as you. That means you spend more time in its gravity well catching up to it than if it was stationary. As you pass the planet you have picked up as speed boost from its pull. But as you travel away from it, because you are going faster you spend less time in its influence, so the deceleration is less than the acceleration you picked up on the approach.
As I try to wrap my brain around this could I assume then in order for this to work, on approach to the planet you have to be going in the same direction as its orbit?
Theoretically no, but if you're going the opposite direction the approach velocity is vastly higher and thus you need a (1) very massive planet and (2) much lower periapsis to have meaningful gravitational slingshot effect. In practice, between atmospheres (the big planets being Jupiter, Saturn, and Neptune), and (for non-gaseous planets) planets not being perfectly smooth spheres, that probably decreases the tolerances below any acceptable level in this solar system.
