Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Thanks for the pointer. Is there any comparison with C/C++ I/O in various operating systems?

Note, that unlike Hadoop, original Google's map reduce system was written in C++.



I/O is I/O: the big differences are lack of certain APIs (which could make it poorly suited to some -- but not majority -- of I/O intensive use cases; needing -- afaik this isn't resolved with NIO.2 -- needing to use native bindings for kernel AIO is one example), poor APIs (ByteBuffer, I'm looking at you, but here's an alternative: https://github.com/airlift/slice), and the fact that by IO (obviously) involves making JNI calls which do have a latency penalty, but one that is dominated by costs of I/O itself (if code is well written, we're speaking nanoseconds here -- while even a FusionIO device would have latency measured in microseconds). I would say for bulk throughput oriented transfer from disk to socket via sendfile(), I'd say the costs of using Java are not noticeable at all.

Re: Google. MapReduce paper was published in 2004, only a few years after Java 1.4 was released. The infrastructure it is built on top of -- and like M/R itself -- were most certainly written before Java 1.4 was released and likely at a time where running Java on Linux meant using Blackdown -- which had its own issues. Java 1.4 is when java.util.concurrent and non-blocking I/O were introduced; prior to this, writing scalable socket in Java was far more difficult. It would also be until Java 1.6 that epoll() would be supported by Java on Linux, etc...

The other big part is that Map/Reduce is not in a vacuum: while Map/Reduce workloads are mostly I/O dominated, other pieces of infrastructure built on those working blocks are very memory intensive. Google "Java GC" to why that is still an issue with Java (despite having an extremely advanced concurrent garbage collector) In addition, I'd machine in the case of Google the performance advantages of C++ (more about things like being able to lay memory out in precise ways and being friendly to the CPU caches, rather than about pure performance) really do matter -- as Joshua Bloch put it "inner loops of indexers would likely stay in C++".

Finally Google does have at least some infrastructural pieces that in Java: I do know that there are first-class supported APIs for BigTable/Spanner and Map/Reduce in Java, there is also the FlumeJava paper (an EDSL for Map/Reduce in Java), and while I believe C++ infrastructure has superseded MegaStore it is one piece of Java being used for Google's core infrastructure.

That's not to even mention Google's well known exteandnsive use of Java for web-applications/middle-ware including AdSense/AdWords billing and management and gmail.

In the end, you can certainly write I/O intensive applications with Java, but with some caveats: be sure it's actually I/O intensive and the non-I/O intensive parts perform fine in Java, be wary of GC hell, know how to avoid excess/implicit copying (pretty much a case in any languages), and be sure that the APIs you plan to use are available.

In the end, I don't think it's an either/or answer: given (hate to use a buzzword, but it certainly applies) service oriented architecture, you can implement various pieces of a system in languages best suited for that task. This is becoming easier in the H* world now that protobufs is the "native" RPC language as opposed to Java serialization.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: