I need to do that every time and remember.

From my experience, you loose access when the last backup you made and synced was made before the key you need now was added.

I.e. this doesn't work in practice.