Running Storm in local mode using docker

©Cid Miranda— https://www.flickr.com/photos/cidska/

First, we need to install docker, i am using ubuntu 20.04. https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04

sudo snap install docker

Download the storm docker image:

sudo docker pull storm

Apache Zookeeper is a must for running a Storm cluster. Start it first. Since the Zookeeper “fails fast” it’s better to always restart it.

sudo docker run -d --restart always --name some-zookeeper zookeeper

The Nimbus daemon has to be connected with the Zookeeper. It’s also a “fail fast” system.

sudo docker run -d --restart always --name some-nimbus --link some-zookeeper:zookeeper storm storm nimbus

Persisting data

sudo docker run -it -v /logs -v /data storm storm nimbus

Finally start a single Supervisor node. It will talk to the Nimbus and Zookeeper.

sudo docker run -d --restart always --name supervisor --link some-zookeeper:zookeeper --link some-nimbus:nimbus storm storm supervisor

Now you can submit a topology to our cluster.

sudo docker run --link some-nimbus:nimbus --rm -it  -v $(pwd):/path/ storm storm jar /path/target/YOURJAR-SNAPSHOT.jar  com.b2wdigital.MAIN_CLASSL "/path/config/CONFIG_FILE.yml"

Optionally, you can start the Storm UI.

sudo docker run -d -p 8080:8080 --restart always --name ui --link some-nimbus:nimbus storm storm ui

List all dockers containers

docker ps

To access a container type

sudo docker exec -it CONTAINER_ID /bin/bash

The supervisor controls the topologies, to see the logs you need to acess the supervisor.

sudo docker exec -it d0a9495c68a2 /bin/bash

To list all topologies

storm list

To kill a topology

storm kill topology_name