Installing Storm on Ubuntu
2013/01/25 1 Comment
Storm is an open source ETL created by Nathan Marz in late 2011. Unlike Hadoop where data are processed offline in big batches, Storm takes another approach by aggregating streaming data on the fly so that aggregated data are immediately available. It is scalable, fault tolerant (no data loss guarantee) and the benchmarks showed that every node can process over a million tuples per seconds.
We describe below the different steps to install Storm in Ubuntu Linux describing the issues we had during the process.
- g++ (sudo apt-get install g++)
- uuid-dev (sudo apt-get install uuid-dev)
- leiningen 1.7. (sudo apt-get install leiningen)
- Java (JDK from http://www.oracle.com/technetwork/java/javase/downloads/index.html). Make sure to have JAVA_HOME set to your JDK directory
- Maven (install it from maven.apache.org)
- Git (sudo apt-get install git)
We recommend you download the stable version from the download page: http://storm-project.net/downloads.html.
First install ZeroMQ as follows:
If you have this error:
Problem with the SSL CA cert (path? access rights?) while accessing...
You might want to ignore SSL verification as follows:
$ git config --global http.sslVerify false
And then re-run:
Now, if you are a Ubuntu user or a Mac user you might have the following error:
make: *** No rule to make target `classdist_noinst.stamp', needed by `org/zeromq/ZMQ.class'. Stop.
This is a known bug found by ebroder (https://github.com/zeromq/jzmq/issues/114). To fix it, edit the file jzmq/src/Makefile.am and replace “
classdist_noinst.stamp” by “
and then re-run:
Add the bin directory to your PATH.
Running the Storm Starter examples
It contains some examples to play with. It can be downloaded as follows:
git clone http://github.com/nathanmarz/storm-starter
Build the project:
$ lein deps $ lein compile $ lein jar
Then run the ExclamationCount example:
$ java -cp $STORM_PATH/lib/*:$STORM_PATH/storm-0.8.2.jar:\ storm-starter-0.0.1-SNAPSHOT.jar \ storm.starter.ExclamationTopology
Assuming STORM_PATH is set to your storm directory.