Hadoop in Docker

Hadoop in Docker

Protocol for Installation on OpenSuse 15.1

Docker Intro

First create btrfs fs on /var/lib/docker to get decent snapshot support, then

sudo zypper install docker python3-docker-compose
sudo usermod -G docker -a $USER
sudo systemctl enable docker
sudo systemctl start docker
sudo systemctl status docker

You need to relogin or do ssh localhost to activate the group change. Then you can start shell in an opensuse image

docker run --name bigtop -it --entrypoint /bin/bash opensuse/leap:42.3

If you exit and want to restart this container restart and attach

docker container restart bigtop
docker attach bigtop

If you want to start another shell in the running container

docker container exec -it bigtop /bin/bash

If you dont know the name of the container, use this to list them all

docker ps -a

If you want to add a volume to your existing container, that is not as easy as I thought:

• stop container (if it is not stopped by just exiting the running progrem or shell)
• commit container to new image
• run new container with changed options using the commited image

docker stop bigtop
docker commit bigtop bigtop-image
docker run --name bigtop2 -v /mountpoint -ti --entrypoint /bin/bash bigtop-image

Once you know all is ok, you can replace the old container

docker rm bigtop
docker rename bigtop2 bigtop

Now you have a docker container where you can start bigtop setup in and know how to reconfigure it if something is misssing

Bigtop

Bigtop from Repo

Add repository to opensuse 42.3 (suggest to do that in docker image created above)

zypper ar https://artfiles.org/apache.org/bigtop/bigtop-1.4.0/repos/opensuse42.3/bigtop.repo
zypper in bigtop-utils bigtop-jsvc bigtop-tomcat bigtop-groovy

Configure tomcat users

cp -av  /usr/lib/bigtop-tomcat/conf.template /usr/lib/bigtop-tomcat/conf
vi /usr/lib/bigtop-tomcat/conf/tomcat-users.xml

I added this just before the final "</tomcat-users>" line. No idea if this is really ok or necessary

 <role rolename="tomcat"/>
 <role rolename="manager"/>
 <role rolename="manager-gui"/>
 <user username="tomcat" password="my-PW" roles="tomcat,manager,manager-gui"/>
 <user username="joachim" password="my-PW" roles="tomcat,manager,manager-gui"/>
 <user username="julian" password="his-PW" roles="tomcat,manager,manager-gui"/>

Open port 8080 (and maybe 8009 and 8005?) for the container and start /usr/lib/bigtop-tomcat/bin/catalina.sh run

docker run --name bigtop-catalina -p 8080:8080 -p 8005:8005 -p8009:8009 -ti --entrypoint /bin/bash bigtop-image /usr/lib/bigtop-tomcat/bin/catalina.sh run

Check if tomcat is running fine by deploying a "hello world" war

zypper install wget
cd /usr/lib/bigtop-tomcat
mkdir -p webapps
cd webapps
wget https://tomcat.apache.org/tomcat-8.0-doc/appdev/sample/sample.war

Tomcat should find the war file, expand and deploy it automatically. Find it at http://localhost:8080/sample

Groovy that comes with bigtop uses utility "which" and needs a JDK. Install them, but since java-1_8_0-openjdk-1.8.0 building fails with java exception (seems this is due to downloaded java lib, not installed java, so this maybe is not necessary), trying with Oracles GPL java:

cd /usr/local
wget https://download.java.net/java/GA/jdk15.0.1/51f4f36ad4ef43e39d0dfdbaf6549e32/9/GPL/openjdk-15.0.1_linux-x64_bin.tar.gz
tar xf openjdk-15.0.1_linux-x64_bin.tar.gz

edit /home/bigtop/.profile: add these two lines

export JAVA_HOME=/usr/local/jdk-15.0.1
export PATH=$JAVA_HOME/bin:$PATH

zypper install which
/usr/lib/bigtop-groovy/bin/groovy -version

Bigtop 1.4.0 from Tar

Enable docker to run from within containers (bind mount socket and command). Check if docker group has same id on host and container. Change /etc/group in container if needed and restart

docker run --name bigtop-catalina -p 8080:8080 -p 8005:8005 -p8009:8009 -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -ti --entrypoint /bin/bash bigtop-catalina-image /usr/lib/bigtop-tomcat/bin/catalina.sh run

zypper in tar curl unzip rpm-build
sed -i 's/en_US.UTF-8//' /etc/sysconfig/language
mkuser -m -G docker bigtop
su - bigtop
wget https://apache.lauf-forum.at/bigtop/bigtop-1.4.0/bigtop-1.4.0-project.tar.gz
tar xzvf bigtop-1.4.0-project.tar.gz
cd bigtop-1.4.0

Now you can call ./gradlew with options as described in BIGTOP Quickstart Guide. E.g.:

./gradlew hadoop-pkg-ind -POS=opensuse-42.3

Bigtop 1.4.0 2nd Try

docker run --name bigtop-prep -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -ti opensuse/leap:42.3 /bin/bash
zypper in -y vim less
zypper in -y tar curl unzip rpm-build
cd /usr/local
curl https://download.java.net/java/GA/jdk15.0.1/51f4f36ad4ef43e39d0dfdbaf6549e32/9/GPL/openjdk-15.0.1_linux-x64_bin.tar.gz -o openjdk-15.0.1_linux-x64_bin.tar.gz
tar xf openjdk-15.0.1_linux-x64_bin.tar.gz
echo 'export JAVA_HOME=/usr/local/jdk-15.0.1' >>/etc/profile.local
echo 'export PATH=$JAVA_HOME/bin:$PATH'  >>/etc/profile.local
sed -i 's/en_US.UTF-8//' /etc/sysconfig/language
groupadd -g `ls -l /var/run/docker.sock | while read x x x g x; do echo $g; done` docker
useradd -m -G docker bigtop
su - bigtop
curl https://apache.lauf-forum.at/bigtop/bigtop-1.4.0/bigtop-1.4.0-project.tar.gz -o bigtop-1.4.0-project.tar.gz
tar xf bigtop-1.4.0-project.tar.gz
ln -sf bigtop-1.4.0 bigtop
cd bigtop
./gradlew --console=plain hadoop-pkg-ind -POS=opensuse-42.3

Bigtop 1.4.0 3rd Try

Now back with distro java

docker run --name bigtop-prep -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -ti opensuse/leap:42.3 /bin/bash
zypper in -y vim less
zypper in -y tar curl unzip rpm-build java-1_8_0-openjdk-devel
sed -i 's/en_US.UTF-8//' /etc/sysconfig/language
groupadd -g `ls -l /var/run/docker.sock | while read x x x g x; do echo $g; done` docker
useradd -m -G docker bigtop
su - bigtop
curl https://apache.lauf-forum.at/bigtop/bigtop-1.4.0/bigtop-1.4.0-project.tar.gz -o bigtop-1.4.0-project.tar.gz
tar xf bigtop-1.4.0-project.tar.gz
ln -sf bigtop-1.4.0 bigtop
cd bigtop
./gradlew --console=plain hadoop-pkg-ind -POS=opensuse-42.3

No matter which os I try, all fail with

Error: No such container:path: e4e2ac52a1c5d559460ab73c92a0f70d39f67fd196d979ec38b18a425121f1dd:/bigtop

or similar

Bigtop 1.4.0 4th Try

Now with rpm's installed again

docker run --name bigtop-prep -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -ti opensuse/leap:42.3 /bin/bash
zypper in -y vim less
zypper in -y tar curl unzip rpm-build java-1_8_0-openjdk-devel
sed -i 's/en_US.UTF-8//' /etc/sysconfig/language
groupadd -g `ls -l /var/run/docker.sock | while read x x x g x; do echo $g; done` docker
useradd -m -G docker bigtop

zypper ar -G https://artfiles.org/apache.org/bigtop/bigtop-1.4.0/repos/opensuse42.3/bigtop.repo
zypper in -y which bigtop-utils bigtop-jsvc bigtop-tomcat bigtop-groovy

cp -av  /usr/lib/bigtop-tomcat/conf.template /usr/lib/bigtop-tomcat/conf
sed -i '/<\/tomcat-users>/i \ \ <role rolename="manager"/>' /usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <role rolename="manager-gui"/>' /usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <user username="tomcat" password="my-PW" roles="tomcat,manager,manager-gui"/>' /usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <user username="joachim" password="my-PW" roles="tomcat,manager,manager-gui"/>' /usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <user username="julian" password="his-PW" roles="tomcat,manager,manager-gui"/>' /usr/lib/bigtop-tomcat/conf/tomcat-users.xml

/usr/lib/bigtop-tomcat/bin/startup.sh

su - bigtop
curl https://apache.lauf-forum.at/bigtop/bigtop-1.4.0/bigtop-1.4.0-project.tar.gz -o bigtop-1.4.0-project.tar.gz
tar xf bigtop-1.4.0-project.tar.gz
ln -sf bigtop-1.4.0 bigtop
cd bigtop
./gradlew --console=plain hadoop-pkg-ind -POS=opensuse-42.3