It uses DNS for only discovering the other containers within the same custom network, if query is not found the DNS redirects the queries to your own DNS server, I don't see how it can be worse than /etc/hosts solution.
Exactly, obviously /etc/hosts will still take precedence, but instead of munging your /etc/hosts when starting a container you can just use their DNS server. I don't see a problem with this.
Because /etc/nsswitch.conf exists to do exactly what's needed here. Now there's an extra layer of need-to-know that adds confusion. I can almost guarantee that there's going to be a major outage somewhere, someplace because of this change.
Docker assumes it cannot trust the container's DNS resolver to respect TTL and cache in a compliant way. So it guarantees stable name->ip mapping for each container. As a result, when you point a service endpoint at another container, it's the IP/container mapping that is changed, which is a much more reliable and atomic change. I would definitely never rely on changing DNS records to orchestrate changes to my production stack, that would be way too brittle.
The real problem is that you can't trust the program's resolver, either. Java will behave differently than Go, which behaves differently from Python, and so on.
This will end well...