If you look at a lot of audio amplifier ICs, they simply drive a 4-ohm speaker at a higher current and power than they do an 8-ohm speaker. The rest of the circuit needs to be designed around the anticipated power draw.

Matching impedance is necessary for connecting high-impedance output stages like tubes to low-impedance circuits like speakers. But transistor amps are generally already operating with low impedance outputs.