The build system needs to know what source files it needs to rebuild and for which ones it can just reuse previously-generated build artifacts, typically by detecting what changed in the source.

What changed in the source outside of your project is driven by whether the semver changed. Semver not changing means that the source didn't change and you can link existing binaries as-is.

Then there is the whole business of dynamic linking.