It's about copy/paste-ability. If you're copy/pasting in to a terminal, with $HOST you can type

    HOST=myhost.tld

first, then just paste, and everything will work.

With <host>, if you paste the command without editing it, then it will do something different and write some files to disk (unless quoted).