Getting notifications from your services

running some services and want to be notified if anything goes wrong? let’s get started.
this will require sendmail to be installed and working on your system.

keep in mind

if you’re working on a windows based system to use this on a unix based system, your line endings will be fucky. use a tool like notepad++ to change them over.
these notifications work quite well with something like auto restart services.

1. identify your executable’s service file by running sudo systemctl status myApp. the loaded line will tell you where to locate it.
2. open your executable’s service file in your editor of choice.
3. in the [Unit] block, add this: OnFailure=crashmailserv@%n.service
4. in a new text file, input this:
   [Unit]
Description=status email for %i to user

[Service]
Type=oneshot
ExecStart=/etc/script/crashmail you@domain.tld %i
User=monitor
Group=systemd-journal
5. save that text file as crashmailserv@.service and put it in /etc/systemd/system
6. in a new text file, input this:
   #!/bin/bash

/usr/sbin/sendmail -t <<ERRMAIL
To: $1
From: servmon <monitor@$HOSTNAME>
Subject: "$2" has crashed, and won't resurrect
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
$(journalctl -u "$2" -e -n 20)
ERRMAIL
7. save this as crashmail in /etc/script
8. run the following commands:
   sudo useradd -g systemd-journal -G sudo monitor
sudo chmod +x /etc/script/crashmail
sudo systemctl daemon-reload

what did we just do?

i’ll take you through it bit by bit.

1. using systemctl’s OnFailure functionality, requested that systemctl start a defined service when the subject service displays a failure status.
2. created a new service (crashmailserv@.service) which runs once and then exits (Type=oneshot), and in doing so runs a defined executable (/etc/script/crashmail) as a defined user and under a defined group
3. created a new executable (crashmail) which runs in a shell and uses sendmail to send an email containing information about the subject service
4. added a user called monitor under whom these services and commands will be executed
5. added the executable permission to our crashmail applet (without it, nothing would be able to execute it like a script)
6. reloaded systemctl‘s loaded unit files and reference information to make it pull in the modifications we just made

what’s all this %n, %i and $1 stuff?

in short, they are dynamic variables which are used to represent aspects of the subjects at hand.
the %n in the OnFailure line tells systemctl to replace that %n with the name of the current subject service, let’s say mysql, so we end up with systemctl start crashmailserv@mysql.service.service
the %i in our crashmailserv@.service tells systemctl to replace that %i with the name of the invoking service, in this case mysql.service as mysql has theoretically reached a failure status.
ExecStart in our crash service file tells systemctl what to do when the service starts, so now we’ve replaced our variables, we get ExecStart=/etc/script/crashmail you@domain.tld mysql.service
and now we’re on to our crashmail, which tells sendmail to send an email.
in a shell script like this, $1 and $2 refer to the first and second argument, respectively, to be passed with the command. now as we’ve discovered above, these are your email address, and the subject service.
so, replacing our variables again:
To: you@domain.tld
From: servmon <monitor@$HOSTNAME>
Subject: "mysql.service" has crashed, and won't resurrect
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
$(journalctl -u "mysql.service" -e -n 20)
journalctl calls the system log journal, -u "mysql.service" filters it by only mysql’s entries, -e jumps directly to the end of the journal (ie the most current entries) and -n 20 makes the command output only the last 20 lines.
all of this together, when a failure is reported in a subject service, asks systemctl to start a oneshot service, which starts an executable which asks sendmail to send an email informing you the subject service has crashed, and gives you the last 20 lines which might help you work out why it’s crashed.