Taking a copy does not affect the potential market of the copyrighted work in any way. Sharing a copy is another matter. Youtube-dl facilitates taking a copy only, it is not a tool for sharing. This case is substantially different than Napster, I disagree. In some circumstances, you must be able to take a copy in order to exercise your allowed "transformative" fair use rights. You cannot sample a record without a way to take a copy.
That's not covered under "Fair Use": see [1] for fair-use criteria. It is definitely a violation of copyright, yea.
[1] http://fairusetube.org/guide-to-youtube-removals/3-deciding-...