Hardware is hard. We would have happily done this as a native iPhone or Watch app if it were possible. We explored that first. But, unfortunately both iOS and watchOS don't support it. In particular, anytime you listen to audio (a YouTube video, phone call, etc) it turns off the microphone and you have to remember to turn it back on.
If Limitless gets PMF, anyone who puts Whisper into an iPhone app will do what you’re doing basically for free.