Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

While the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context.

In 2007-2010, when Hadoop first started to gain momentum it was very useful because disk sizes were smaller, 64 bit machines weren't ubiquitous and (perhaps most importantly) SSDs were insanely expensive for anything more than tiny amounts of data.

That meant if you had more than a couple of terabytes of data you either invested in a SAN, or you started looking at ways to split your data across multiple machines.

HDFS grew out of those constraints, and once you have data distributed like that, with each machine having a decently powerful CPU as well, Map/Reduce is a sensible way of dealing with it.



It still is a valid pattern. As disks get bigger, the harder it is to move the data and the more efficient it is to move the compute to the data.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: