The check happens at a lower level than the Finder, I believe.

I think it checks things with the quarantine flag set on the executable or app. This is set by your browser and other similar software when downloading an app. Once cleared, the notarization check isn't done.

Brew doesn't set the quarantine flag in the first place as far as I know.