> it is possible to store index to some object only for that object to be replaced with something else entirely later.

That's what generational arenas are for, at the cost of having to check for index validity on every access. But that cost is only in comparison to "keep a pointer in a field" with no additional logic, which is bug-prone.