I wonder why this pattern of managing distributed system at OS level is not widely adopted? and we end up with various customized systems handling distributed system issues at application level.

Hard to say, here is an interesting take on it from a few years back: https://cs.brown.edu/people/malte/pub/papers/2013-apsys-dios...

… legacy applications and dominance of the win32 and posix APIs.

It’s proven easier to implement many of the features of Plan9 lineage and Erlang via containers and orchestrators, rather than porting software and reteaching developers.