(This came up in a discussion on email@example.com)
When converting from sysvinit scripts to systemd init files, the default practice seems to be to start services without forking, and to use Type=simple in the service description.
What Type=simple does is, well, simple. from systemd.service(5):
If set to simple (the default value if neither Type= nor BusName= are specified), it is expected that the process configured with ExecStart= is the main process of the service. In this mode, if the process offers functionality to other processes on the system, its communication channels should be installed before the daemon is started up (e.g. sockets set up by systemd, via socket activation), as systemd will immediately proceed starting follow-up units.
In other words, systemd just runs the command described in ExecStart=, and it’s done: it considers the service is started.
Unfortunately, this causes a regression compared to the sysvinit behaviour, as described in #778913: if there’s a configuration error, the process will start and exit almost immediately. But from systemd’s point-of-view, the service will have been started successfully, and the error only shows in the logs:
root@debian:~# systemctl start ssh root@debian:~# echo $? 0 root@debian:~# systemctl status ssh ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled) Active: failed (Result: start-limit) since mer. 2015-05-13 09:32:16 CEST; 7s ago Process: 2522 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255) Main PID: 2522 (code=exited, status=255)
mai 13 09:32:16 debian systemd: ssh.service: main process exited, code=exited, status=255/n/a mai 13 09:32:16 debian systemd: Unit ssh.service entered failed state. mai 13 09:32:16 debian systemd: ssh.service start request repeated too quickly, refusing to start. mai 13 09:32:16 debian systemd: Failed to start OpenBSD Secure Shell server. mai 13 09:32:16 debian systemd: Unit ssh.service entered failed state.
With sysvinit, this error is detected before the fork(), so it shows during startup:
root@debian:~# service ssh start [....] Starting OpenBSD Secure Shell server: sshd/etc/ssh/sshd_config: line 4: Bad configuration option: blah /etc/ssh/sshd_config: terminating, 1 bad configuration options failed! root@debian:~#
It’s not trivial to fix that. The implicit behaviour of sysvinit is that fork() sort-of signals the end of service initialization. The systemd way to do that would be to use Type=notify, and have the service signals that it’s ready using systemd-notify(1) or sd_notify(3) (or to use socket activation, but that’s another story). However that requires changes to the service. Returning to the sysvinit behaviour by using Type=forking would help, but is not really a solution: but what if some of the initialization happens *after* the fork? This is actually the case for sshd, where the socket is bound after the fork (see
strace -f -e trace=process,network /usr/sbin/sshd), so if another process is listening on port 22 and preventing sshd to successfully start, it would not be detected.
I wonder if systemd shouldn’t do more to detect problems during services initialization, as the transition to proper notification using sd_notify will likely take some time. A possibility would be to wait 100 or 200ms after the start to ensure that the service doesn’t exit almost immediately. But that’s not really a solution for several obvious reasons. A more hackish, but still less dirty solution could be to poll the state of processes inside the cgroup, and assume that the service is started only when all processes are sleeping. Still, that wouldn’t be entirely satisfying…