Systemd unit’s crash course

Intro

Either you are aware of it now or you will be later. There is a lot of mixed feelings around the implementation of systemd. As a sysadmin I am kind of bias and I for one love it and want to show you how easy you can automate your services / containers whether its docker or native python apps/services.

Systemd will help you set dependencies for our services/daemons/scripts, timers, start (pre-start) , stop (post-stop) , restart , update +++ and more…

With all these options this sounds complicated right? Well, its not! Its only as complicated as you make it! Keep reading and you wont be sorry. There’s a lot of “compressed” information here, so if you dont get it the first time, read it again (and again). But dont despair, its all connected in the end.

Easy start

Lets start with something simple:

[Unit]
Description=Plexconnect
After=plexmediaserver.service

[Service]
Type=simple
ExecStart=/usr/bin/python /opt/PlexConnect/PlexConnect.py
User=root
Group=root
Restart=on-failure
RestartSec=15
StartLimitInterval=10s

[Install]
WantedBy=multi-user.target

This is a unit file called “plexconnect.service”. Its jobb is to start a python service at boot and keep it running / logging. For systemd to recognize unit files they need to end in .service or .timer (more about timers later).  They live in “/etc/systemd/system”.

Now lets brake down the individual components of the file:

[Unit]
Description=Plexconnect

This is a simple description and will not impact anything except maybe your sanity when you have a bunch of them and might not remember why you made exactly that unit.

After=plexmediaserver.service

Now this is more important. This tells systemd to wait for plexmediaserver.service to be running before starting this unit. Now if you want another (or more) services to be started before this one you could just specify them on the same line like so:

After=plexmediaserver.service network.target

But since I know plexmediaserver.service has network.target defined already there is no need to specify it again in something that starts AFTER.

[Service]
Type=simple

Now we start to specify what we want systemd to to. “Type=simple” means that systemd expects the file executed to be the main and/or only program. This works for most commands you want to execute as long as they dont fork (divide) in to other processes and leave the “command” you specify. More on that subject later. (For bashscripts use “Type=oneshot”)

ExecStart=/usr/bin/python /opt/PlexConnect/PlexConnect.py

This is the main function / command of the unit file. As you can see you need to use the full path names. This would be the equivalent of launching “python /opt/PlexConnect/PlexConnect.py” from a console.

User=root
Group=root

You guessed it, this tells systemd which user to “be” when running the command(s). If the user is “root” you dont need these lines. I just added them to show you that this is an option. (It does not produce any errors to keep them)

Restart=on-failure
RestartSec=15

Tells systemd to restart the service if it should stop or fail unexpected. It will restart 15sec after failure/stop.

[Install]
WantedBy=multi-user.target

Here its where it gets a bit complicated. This is the target  we want the service to be loaded before. Very simplified: This means we want the service up and running about the same time you reach the login screen.

You can use this “template” to set up your own units for your own python (and other) programs to run at startup and get logged in journalctl (more later).

So how can we use this with docker?

It’s assumed you have docker up and running. If you dont, read this crash course:

Docker Crash Course

Lets make our self a unit file:

sudo nano /etc/systemd/system/docker-portainer.service

Paste this:

[Unit]
Description=Portainer
Documentation=https://github.com/portainer/portainer
After=network.target docker.socket
Requires=docker.socket

[Service]
RestartSec=10
Restart=always

Environment="NAME=portainer"
Environment="IMG=portainer/portainer"

# Pull new image for updates
ExecStartPre=-/usr/bin/docker pull $IMG

# Clean-up bad state if still hanging around
ExecStartPre=-/usr/bin/docker rm -f $NAME

# Main process
ExecStart=/usr/bin/docker run \
--name $NAME \
-p 9000:9000 \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/docker/portainer:/data \
$IMG

# Large image causes a timeout because of the time it takes
# to download and extract.
TimeoutStartSec=infinity

# Stop Service
ExecStop=/usr/bin/docker stop $NAME

[Install]
WantedBy=multi-user.target

(You exit (when ready) and save with Ctrl-x, y, enter)
Lines starting with an “#” are comments and will not be interpreted by systemd. 

Again, lets brake it down to pieces:

[Unit]
Description=Portainer
Documentation=https://github.com/portainer/portainer
After=network.target docker.socket

Again we give it a description, but this time we add a link to the github page as well. Again, its not mandatory, but helps you keep your sanity in the end. We also tell systemd about the dependencies network.target and docker.socket.

Requires=docker.socket

This sets docker as a dependence, which is important, as we are going to use it in our command(s).

Restart=always

This tells systemd to restart the service no matter how it stops. This means if you use “docker stop $containername”, systemd will still restart it. You can choose “Restart=on-failure” but then you can override systemd with docker, and as I am a big fan of “one rule, to rulle them all”, I dont (So there you have it, the debate about systemd shown in a choice (in a nutshell),as I mentioned in the beginning).

Environment="NAME=portainer"
Environment="IMG=portainer/portainer"

These are environment variables we use to make replication of the Unit or change of container name / image source easy for our selves (We’ll get back to this). Note that this has nothing to do with the environment variables passed to docker (It’l all fall in to place soon).

# Pull new image for updates
ExecStartPre=-/usr/bin/docker pull $IMG

# Clean-up bad state if still hanging around
ExecStartPre=-/usr/bin/docker rm -f $NAME

Here you can see our environment variables in action. its basically the same as writing:

ExecStartPre=-/usr/bin/docker pull portainer/portainer

&

ExecStartPre=-/usr/bin/docker rm -f portainer

These are commands run before our actual service (ExecStart=). The first command pulls/updates the container image. The second on the other hand is more important. As you might remember from our previous crash course you cant start two containers with the same “–name”. This command removes any previous container with the same name ($NAME=portainer). If you dont use VOLUMES or experiment inside your containers, this will remove your changes. But as most containers use volumes
(-v) this is usually (almost never) not an issue.

NOTE: The “=-” at the end of ExecStartPre. The “-” means the command is allowed to fail and the main process will still start.

# Main process
ExecStart=/usr/bin/docker run \
--name $NAME \
-p 9000:9000 \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/docker/portainer:/data \
$IMG

This is where the magic happens. We start our container like we would on the command line (actually its nothing magical about it 🙂 ). One thing, never use  the “-d” variable as that will –daemonize the container and the process will fork. Meaning systemd with the “Type=simple” will think its stopped and get stuck in a loop killing and restarting.

# Large image causes a timeout because of the time it takes
# to download and extract.
TimeoutStartSec=infinity

The comments explains it all. You can also use variables like: “120sec” or “2min”

# Stop Service
ExecStop=/usr/bin/docker stop $NAME

This tells systemd what command to run to stop the service. (if nothing is specified its basically a Ctrl-c exit).

Lets go already!

Thats it, we are finally through explaining the content of these unit files. Now we can finally se systemd in action (in addition to every time you boot). Lets start with the basics. After you created the unit you need to let systemd know about it.

sudo systemctl daemon-reload

This command reloads the content of all the units and needs to be run every time you make a change to a unit. Now lets start the service/container.

sudo systemctl start docker-portainer.service

Depending on your internet connection and CPU this might take some time (seconds). You will be back in your bash like nothing happened. But it did, lets take a look:

sudo systemctl status docker-portainer.service

Will print:

docker-portainer.service - Portainer
Loaded: loaded (/etc/systemd/system/docker-portainer.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2017-12-13 21:09:12 CET; 54s ago
Docs: https://github.com/portainer/portainer
Process: 28563 ExecStartPre=/usr/bin/docker rm -f $NAME (code=exited, status=1/FAILURE)
Process: 28523 ExecStartPre=/usr/bin/docker pull $IMG (code=exited, status=0/SUCCESS)
Main PID: 28569 (docker)
Tasks: 10 (limit: 4915)
CGroup: /system.slice/docker-portainer.service
└─28569 /usr/bin/docker run --name portainer -p 9000:9000 -v /var/run/docker.sock:/var/run/do

Dec 13 21:09:10 idea docker[28523]: d1e017099d17: Download complete
Dec 13 21:09:11 idea docker[28523]: d1e017099d17: Pull complete
Dec 13 21:09:11 idea docker[28523]: f96966278ba2: Verifying Checksum
Dec 13 21:09:11 idea docker[28523]: f96966278ba2: Download complete
Dec 13 21:09:12 idea docker[28523]: f96966278ba2: Pull complete
Dec 13 21:09:12 idea docker[28523]: Digest: sha256:261d02375566fd24f3fc0fb358bb5e4c394d82b3e44efb77e1609
Dec 13 21:09:12 idea docker[28523]: Status: Downloaded newer image for portainer/portainer:latest
Dec 13 21:09:12 idea docker[28563]: Error: No such container: portainer
Dec 13 21:09:12 idea systemd[1]: Started Portainer.
Dec 13 21:09:12 idea docker[28569]: 2017/12/13 20:09:12 Starting Portainer 1.15.5 on :9000

Take note of the line:

Active: active (running) since Wed 2017-12-13 21:09:12 CET; 54s ago

Success! Open your browser and navigate to localhost:9000 . You will see portainer is up and running. Portainer is a great tool for docker and you can do many things! Go read more on the homepage:
https://github.com/portainer/portainer

Now look at the line:

Process: 28563 ExecStartPre=/usr/bin/docker rm -f $NAME (code=exited, status=1/FAILURE)

The prosess: ExecStartPre=-/usr/bin/docker rm -f portainer failed. Not so strange as we did not have a container with that name on the system. As mentioned, this will not impact the main process as this pre-prosess was allowed to fail.

Lets run:

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6d1fad8bc34c portainer/portainer "/portainer" 3 seconds ago Up 3 seconds 0.0.0.0:9000->9000/tcp portainer

All good!

Now lets stop our service:

sudo systemctl stop docker-portainer.service

Lets have another look:

sudo systemctl status docker-portainer.service

This is how it looks now:

docker-portainer.service - Portainer
Loaded: loaded (/etc/systemd/system/docker-portainer.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2017-12-13 21:25:10 CET; 35s ago
Docs: https://github.com/portainer/portainer
Process: 29147 ExecStop=/usr/bin/docker stop $NAME (code=exited, status=0/SUCCESS)
Process: 28569 ExecStart=/usr/bin/docker run --name $NAME -p 9000:9000 -v /var/run/docker.sock:/var/ru
Process: 28563 ExecStartPre=/usr/bin/docker rm -f $NAME (code=exited, status=1/FAILURE)
Process: 28523 ExecStartPre=/usr/bin/docker pull $IMG (code=exited, status=0/SUCCESS)
Main PID: 28569 (code=exited, status=2)

Dec 13 21:09:12 idea docker[28523]: Digest: sha256:261d02375566fd24f3fc0fb358bb5e4c394d82b3e44efb77e1609
Dec 13 21:09:12 idea docker[28523]: Status: Downloaded newer image for portainer/portainer:latest
Dec 13 21:09:12 idea docker[28563]: Error: No such container: portainer
Dec 13 21:09:12 idea systemd[1]: Started Portainer.
Dec 13 21:09:12 idea docker[28569]: 2017/12/13 20:09:12 Starting Portainer 1.15.5 on :9000
Dec 13 21:25:10 idea systemd[1]: Stopping Portainer...
Dec 13 21:25:10 idea docker[29147]: portainer
Dec 13 21:25:10 idea systemd[1]: docker-portainer.service: Main process exited, code=exited, status=2/IN
Dec 13 21:25:10 idea systemd[1]: docker-portainer.service: Failed with result 'exit-code'.
Dec 13 21:25:10 idea systemd[1]: Stopped Portainer.

Take no note of the failed state, but take note of the:

Process: 29147 ExecStop=/usr/bin/docker stop $NAME (code=exited, status=0/SUCCESS)

The process exited correctly!

Now we want to tell systemd to start the service at boot:

sudo systemctl enable docker-portainer.service

Thats it! It will now run and update at every boot and systemd will keep it alive an logg every output. You can disable it with replacing “enable” with “disable”.

Now lets look at the logg:

journalctl -u docker-portainer.service

outputs:

-- Logs begin at Wed 2017-11-29 10:56:15 CET, end at Wed 2017-12-13 21:29:04 CET. --
Dec 13 21:09:06 idea systemd[1]: Starting Portainer...
Dec 13 21:09:06 idea docker[28523]: Using default tag: latest
Dec 13 21:09:09 idea docker[28523]: latest: Pulling from portainer/portainer
Dec 13 21:09:09 idea docker[28523]: d1e017099d17: Pulling fs layer
Dec 13 21:09:09 idea docker[28523]: f96966278ba2: Pulling fs layer
Dec 13 21:09:10 idea docker[28523]: d1e017099d17: Verifying Checksum
Dec 13 21:09:10 idea docker[28523]: d1e017099d17: Download complete
Dec 13 21:09:11 idea docker[28523]: d1e017099d17: Pull complete
Dec 13 21:09:11 idea docker[28523]: f96966278ba2: Verifying Checksum
Dec 13 21:09:11 idea docker[28523]: f96966278ba2: Download complete
Dec 13 21:09:12 idea docker[28523]: f96966278ba2: Pull complete
Dec 13 21:09:12 idea docker[28523]: Digest: sha256:261d02375566fd24f3fc0fb358bb5e4c394d82b3e44efb77e1609
Dec 13 21:09:12 idea docker[28523]: Status: Downloaded newer image for portainer/portainer:latest
Dec 13 21:09:12 idea docker[28563]: Error: No such container: portainer
Dec 13 21:09:12 idea systemd[1]: Started Portainer.
Dec 13 21:09:12 idea docker[28569]: 2017/12/13 20:09:12 Starting Portainer 1.15.5 on :9000
Dec 13 21:25:10 idea systemd[1]: Stopping Portainer...
Dec 13 21:25:10 idea docker[29147]: portainer
Dec 13 21:25:10 idea systemd[1]: docker-portainer.service: Main process exited, code=exited, status=2/IN
Dec 13 21:25:10 idea systemd[1]: docker-portainer.service: Failed with result 'exit-code'.
Dec 13 21:25:10 idea systemd[1]: Stopped Portainer.
Dec 13 21:28:45 idea systemd[1]: Starting Portainer...
Dec 13 21:28:45 idea docker[29309]: Using default tag: latest
Dec 13 21:28:47 idea docker[29309]: latest: Pulling from portainer/portainer
Dec 13 21:28:48 idea docker[29309]: Digest: sha256:261d02375566fd24f3fc0fb358bb5e4c394d82b3e44efb77e1609
Dec 13 21:28:48 idea docker[29309]: Status: Image is up to date for portainer/portainer:latest
Dec 13 21:28:48 idea docker[29325]: portainer
Dec 13 21:28:48 idea systemd[1]: Started Portainer.
Dec 13 21:28:48 idea docker[29331]: 2017/12/13 20:28:48 Starting Portainer 1.15.5 on :9000
lines 1-30/30 (END)

The end

As you can see, you have full control of the container now! BTW its no coincidence I choose portainer for this crash course. It also has logs of your running containers. On top of that it has tools like console for easy access to your containers. Definitely worth a look! Thats it for now. I might have gotten a bit intense towards the end here but you stuck to it! If there’s anything unclear, or something fishy, just drop me a comment and Ill get back to you asap!

As you can imagine I have only scratched the surface of systemd and journalctl, you can find more information on these sites:

https://www.freedesktop.org/software/systemd/man/systemd.unit.html

https://www.freedesktop.org/software/systemd/man/journalctl.html

Next time i will show you how to make a unit and a timer that backs up and updates your docker volumes and units. Until next time, happy docking!

 

Leave a Comment!