Unix command line faster than Hadoop

If you’ve paid any attention to cloud computing, you’ve probably heard of Hadoop, a cloud-based tool for analysing large data sets. You might think it would be the be all and end all for data analysis, but in reality modern PCs can outperform it for some tasks.

Adam Drake downloaded a database of results from 2 million chess games and used some clever Unix shell script programming to analyse the data. He was able to get his results on a laptop more than 200 times faster than by using a Hadoop cluster. The data set wasn’t large by Hadoop standards but it does prove a point – modern computers are amazingly powerful when used with the right tools. For example, the Quantum XA trading engine at the TSX, which is one of the world’s fastest, runs on off-the-shelf x86 servers running Linux.

Leave a Reply