Agreed, but it's possible, up to a point: oral history, apprenticeship, observation all serve. It is much less efficient, much less reliable, and probably impractical for higher-order things (such as theories of relativity) but it does work as a bootstrap. Human culture did well enough for a long time before writing.

I think a bigger reason that intelligent animals haven't progressed as far as us (qualify 'progress' however you like) is more to do with evolutionary pressure from humans changing the environment faster than they can adapt to it. That extends to populations of humans at different stages of societal evolution as well: cultures are snuffed out very quickly when more advanced groups make their presence felt.