running some services and want to be notified if anything goes wrong? let’s get started.
this will require sendmail to be installed and working on your system.

keep in mind

if you’re working on a windows based system to use this on a unix based system, your line endings will be fucky. use a tool like notepad++ to change them over.
these notifications work quite well with something like auto restart services.

1. identify your executable’s service file by running sudo systemctl status myApp. the loaded line will tell you where to locate it.
2. open your executable’s service file in your editor of choice.
3. in the [Unit] block, add this: OnFailure=crashmailserv@%n.service
4. in a new text file, input this:
   [Unit]
Description=status email for %i to user

[Service]
Type=oneshot
ExecStart=/etc/script/crashmail you@domain.tld %i
User=monitor
Group=systemd-journal
5. save that text file as crashmailserv@.service and put it in /etc/systemd/system
6. in a new text file, input this:
   #!/bin/bash

/usr/sbin/sendmail -t <<ERRMAIL
To: $1
From: servmon <monitor@$HOSTNAME>
Subject: "$2" has crashed, and won't resurrect
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
$(journalctl -u "$2" -e -n 20)
ERRMAIL
7. save this as crashmail in /etc/script
8. run the following commands:
   sudo useradd -g systemd-journal -G sudo monitor
sudo chmod +x /etc/script/crashmail
sudo systemctl daemon-reload

what did we just do?

i’ll take you through it bit by bit.

1. using systemctl’s OnFailure functionality, requested that systemctl start a defined service when the subject service displays a failure status.
2. created a new service (crashmailserv@.service) which runs once and then exits (Type=oneshot), and in doing so runs a defined executable (/etc/script/crashmail) as a defined user and under a defined group
3. created a new executable (crashmail) which runs in a shell and uses sendmail to send an email containing information about the subject service
4. added a user called monitor under whom these services and commands will be executed
5. added the executable permission to our crashmail applet (without it, nothing would be able to execute it like a script)
6. reloaded systemctl‘s loaded unit files and reference information to make it pull in the modifications we just made

what’s all this %n, %i and $1 stuff?

in short, they are dynamic variables which are used to represent aspects of the subjects at hand.
the %n in the OnFailure line tells systemctl to replace that %n with the name of the current subject service, let’s say mysql, so we end up with systemctl start crashmailserv@mysql.service.service
the %i in our crashmailserv@.service tells systemctl to replace that %i with the name of the invoking service, in this case mysql.service as mysql has theoretically reached a failure status.
ExecStart in our crash service file tells systemctl what to do when the service starts, so now we’ve replaced our variables, we get ExecStart=/etc/script/crashmail you@domain.tld mysql.service
and now we’re on to our crashmail, which tells sendmail to send an email.
in a shell script like this, $1 and $2 refer to the first and second argument, respectively, to be passed with the command. now as we’ve discovered above, these are your email address, and the subject service.
so, replacing our variables again:
To: you@domain.tld
From: servmon <monitor@$HOSTNAME>
Subject: "mysql.service" has crashed, and won't resurrect
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
$(journalctl -u "mysql.service" -e -n 20)
journalctl calls the system log journal, -u "mysql.service" filters it by only mysql’s entries, -e jumps directly to the end of the journal (ie the most current entries) and -n 20 makes the command output only the last 20 lines.
all of this together, when a failure is reported in a subject service, asks systemctl to start a oneshot service, which starts an executable which asks sendmail to send an email informing you the subject service has crashed, and gives you the last 20 lines which might help you work out why it’s crashed.

if you have some important apps running in the context of a service, meaning you can start them by running systemctl start myApp you can also configure them to automatically restart if they die.

1. locate your unit’s service file by running sudo systemctl status myApp.
2. read the loaded line, this tells you where systemctl found the service file to load.
3. open this service file in your editor of choice.
4. in the [Service] unit, add this:
   Restart=always
5. in the [Unit] unit, add these:
   StartLimitIntervalSec=10
StartLimitBurst=5
6. run sudo systemctl daemon-reload

that’s it. your service will now restart itself if it ever dies, and if it dies and doesn’t successfully start due to error 5 times in 10 seconds, it will stop automatically trying to restart itself.

i’ve recently become aware of how to make and write scripts, which honestly is just lovely. it makes running things for an idiot like me very, very user friendly. scripts are essentially an order-sensitive list of commands for your shell to execute when you ask it to, so they’re very versatile.
if, like me, you only have a vague idea what you are doing, here’s a few small things which you may find helpful.

• find an editor you like and stick with it. notepad++ is a great one for various reasons, not least that you can format what you’re writing based on the language you are writing it in, which can make errors nice and easy to spot.
  • hint: shell script has its own language setting in the format menu.
• if writing a linux/unix script on windows, note that windows characterizes line endings differently than unix, which invariably results in this: /bin/bash^M: bad interpreter on execution.
  this can be fixed in notepad++ by right clicking the line terminator picker in the bottom right Windows (CR LF) and switching it to Unix (LF)
• can’t run your script? you probably didn’t make it executable. try chmod +x /path/to/script.sh
• using a script to run an executable? you need to include the full path, so if you’re used to running systemctl status myApp you’ll instead have to write /bin/systemctl status myApp
  • don’t know where your executable is? try which myApp to ask the system to show you where it’s located.
• all executable scripts must start with #!/bin/bash (dependant on environment, usually similar). this tells the executor what to use to execute it.
• want your script to tell you what it’s doing as it’s doing it? make the first line in your script set -xv
• because scripts are essentially lines of commands, they can operate and run commands in reference to a folder, like you can execute commands inside a folder. this means you can include a cd command to move to a particular directory in order to run a particular command in context of that directory.
• got your scripts somewhere and want to be able to run them directly from the terminal? open /home/youruser/.profile in your text editor of choice and add this line at the end: export PATH=$PATH:/path/to/your/scripts