And even when we set it, we don't get it, because we blithely read a "latency" label in a GUI instead of measuring the round-trip latency on the specific device in question.
That wouldn't be correct either, at least half the time. Problem is that "latency" is used with different meanings, at least two:
1. time from an acoustic pressure wave reaching a transducer (microphone), being converted to a digital representation, being processed by a computer, being converted back to an analog representation and finally causing a new acoustic pressure wave care of another transducer (speaker).
2. time between when someone uses some kind of physical control (mouse, MIDI keyboard, touch surface, many others) to indicate that they would like something to happen (a new note, a change in a parameter) and an acoustic pressure wave emerging somewhere that reflects that change.
The first one is "roundtrip" latency; the second one is playback latency.