One of the design goals of systemd is to check if services are running, and restart them automatically if they aren't. One of the criticisms is that this should be done in the daemon, not in systemd (for various reasons, one of which is addressed here).

In some cases it's easy to detect if a daemon has failed: when the daemon no longer has any processes running, you know it has failed. That's a fairly trivial case of course, and you'd hope the daemon is well written so it doesn't crash.

But that's not really what you want, what you want is to make sure the service is still up and responding correctly. There is no automated way to do that. In other words, you need to have deeper knowledge of the daemon in order to make sure it is still running.

One idea: you could have the service send a message to systemd every few seconds, letting systemd know it is alive. This is a heartbeat, but it is again relying on the writer of the daemon to make sure he does it correctly. If you've seen a lot of code, then you know that someone will set the heartbeat message to send in a separate thread, and you'll end up with a situation where the entire program has deadlocked, and nothing is running except the heartbeat thread.

These are the limitations systemd lives with: the limitations any monitoring system would live with; that automatic monitoring of processes is not really possible. If processes are crashing often, the answer isn't to improve monitoring, it's to fix the process.

I've written other entries (some positive some negative) about systemd here. The big problem I see with systemd is outlined here.

Systemd pid 1 takes up 1.5 megabytes on my Fedora system hard drive. For comparison, sysvinit compiles down to 40k. For security purposes, smaller is better, but getting the interfaces right is far more important.

So what is systemd doing with that 1.5 megabytes of compiled code? To answer that question, we find a good place to start, src/core/main.c, which is 2085 lines long. That is not necessarily too long, but if you are a fan of short functions, you will be horrified to learn that the main function is 825 lines long.

Typically a long function starts out well organized, but as people add to it over time, it becomes more and more chaotic, and that is starting to happen here. It is not always clear the order of why things happen, for example. If I were going to refactor this, it would look something like this:

main() {
    init();
    main_loop();
    cleanup();
}

So what is it doing with all those 825 lines of code? I went through and made a rough outline, which I wrote in code form. It could be further simplified, of course, but this will give you a basic understanding of what is in the main function, and it's better than 825 lines of chaotic code:

int main(void) {
 
  getTimestamp();
  parseSomeCommandlineArgs();
  setPIDNameToSystemd();
  setUMaskToZeroIfRequired();
 
  initFilesystems();
  initSELinux();
  initSMACK();
  initIMA();
  initKernelClocks();
  initLogging();
 
  initDefaultUnit();
  initJoinControllers();
 
  mountKernelVirtualFilesystems();
  initSignalHandlers();
  parse_Config_Commandline_Environment_and_KernelCommandLine();
  daemonize();
 
  initIfRunningAsManagerSystem();
  initIfRunningAsManagerUser();
  managerSetup();
 
  setCapabilityBounding();
  enforceSyscallArchitectures();
 
  queueDefaultJob();
 
  do{
      runMainLoop();
  }while(handleStateChange());
 
  reexecuteIfNeeded();
 
  cleanupAndShutdown();
}

I tried pasting the original function here so you could compare, but Slashdot apparently doesn't allow posts that long and chopped it off. If you want to see it, you'll have to put some extra effort in and click on this link.

When code gets too long, you need to break it into smaller pieces, so the human brain can see how the code is organized. When you refactor like this, it's important to make sure that the refactored code shows the overall structure, rather than obscuring it.

As always, please comment or correct me below, I am interested in hearing what you think!
(I've written other reviews, too, check them out).

$ dig -t ns fr. 198.41.0.4
 
; <<>> DiG 9.9.5-9+deb8u3-Debian <<>> -t ns fr. 198.41.0.4
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 12051
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;fr. IN NS
 
;; Query time: 0 msec
;; SERVER: 192.168.6.254#53(192.168.6.254)
;; WHEN: Tue Dec 01 15:12:39 CET 2015
;; MSG SIZE rcvd: 31
 
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 46972
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;198.41.0.4. IN NS
 
;; AUTHORITY SECTION:
. 42336 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2015120100 1800 900 604800 86400
 
;; Query time: 0 msec
;; SERVER: 192.168.6.254#53(192.168.6.254)
;; WHEN: Tue Dec 01 15:12:39 CET 2015
;; MSG SIZE rcvd: 114

198.41.0.4 is a.root-servers.net.

Other country domains, uk, de and so on seem to still exist.

COP21 or Daech?

In some ways, I think of Lennart Poettering as a product manager. He figures out what people want, then builds it. The architecture is not always the best, but 'customers' don't tend to care. The point here is that some people do want some of the features of systemd, and Lennart figured that out.

Lennart's system is sometimes laughably bad (for example, he'd prefer to get rid of BSD because he doesn't know how to write portable software), but he has done the community a service by bringing up the topic of system init. More and more, we are seeing thought provoking commentary, ideas on what a good init should be like. This wouldn't have happened if Lennart hadn't built systemd.

This is how the community should be. One person sees the problem, and tries to solve it. Other people come and say, "I can think of a way to improve that." Eventually we arrive at a solution that is elegant, and effective. Init systems have been controversial for decades, which is why BSD and SysVinit are different. Every major Unix vendor has tried to come up with a solution to that problem, and in each case come up with something rather mediocre. Until recently, the Linux and BSD communities have kept what they had, preferring not to change until something comes that is clearly better. When the better answer comes along, it will be obvious.

Systemd won't be here for the long term, but it's effects will be felt through spurring the community to solve this problem that has plagued the community for decades, again, analyzing the problem with things like this (I'd also like to think my own analysis is worth something).

I think a good start would be to add functions to POSIX like "daemonize()" and "start_dameonize()" "complete_daemonize()" but maybe I'm wrong.

Book about programming.

As an intermezzo while investigating the stability of APIs (since that's not a fun thing to do), here is an analysis of some code: examining random-seed.c, pulled September 1st, 2015 ( link here, for as long as github is around).

When you come across a piece of code, the first thing you ask is, "what does this do?" In this piece of code, you can see the main() on line 34, so we know it's a stand-alone utility. That is the only function in a 165 line file, so come on Poettering, break your code into easy to read functions. It's not 1985 anymore.

Since it's not clear what the utility does from the comments, a web search might be helpful, which finds the man page, which doesn't tell you much. For example, on line 44 of random-seed.c, we can see that a command-line argument is required. However, the man page doesn't tell you what options are available, it's biggest use is telling you where the seed is stored on the filesystem (if you're wondering where that file location is defined, it's in the Makefile.am). The documentation is bad.

The program itself isn't helpful either, it doesn't respond to the --help option. If you give it any option that is not correct, it responds "Unknown verb '%s'." So keep guessing random verbs until you get the correct option.

Moving along to my main complaint, you have to read many lines of this file before you understand what it does. A program like this can be mostly self-documenting, if you structure main correctly. Here is an example of how it could be.

int main(int argc, char **argv) {
      init();
      parseCommandLineArgs();

    if(shouldLoad) {
              writeSeedFromFileToDevUrandom();
    }
    else if(shouldSave) {
              writeSeedFromDevToFile();
    }
      return 0;
}

This can still be improved. There can be comments explaining why things happen. It is much easier to see what the program does when it's broken up like this. The main algorithm should be clear.

One more serious criticism about this file, hard-coding the output file in Makefile.am is double bad: first because it shouldn't be hardcoded, it should be a command-line option. Second because build files should store as little information as possible.

That's a lot of criticism, but they're relatively minor. With a few fixes, the code is cleaned up. The big problems in systemd, the things that really worry me, were mentioned in journal 9. Those are the things that will cause serious degradation over time, and problems for all of us.

This entry discusses the benefits of good interfaces, then gives a (overly simple) metric for how to recognize them. Then it ties it back to systemd.

The 'unix way' is built around solid interfaces. The actual code to Unix is mostly gone (unless you use SCO or something), but the interface lives on with multiple (mostly compatible) implementations. The interface long outlasted the code. (NB: Unix hasn't always had good interfaces in every place, Unix isn't perfect.)

Another example of a good interface is the Berkeley networking API. It's not user-friendly, but as a low-level 'plumbing' API, it is extremely flexible and effective. It spread and now every OS has an implementation. (That is, every OS more complex than some stuff at http://wiki.osdev.org ). The original code is gone, but the interface will be around for a long, long time.

There are plenty of examples of interfaces in Unix. It doesn't have to be an API.

If you consider a programming language to be an interface between a computer and a human, then think of the C compiler. The original C compiler is no longer in use, but the interface was designed clearly enough that even intentionally bad IOCC code from 30 years ago still compiles.

Cron is another interface, as are the basic CLI commands like ls, mv, ps, grep, etc (which have now taken over almost everywhere....the interfaces, not the original code). The pipe is a powerful interface that seems obvious now, but wasn't at the time......the pipe makes it possible to take the output from any program and send it as input to any other program. That is impressive.

"Robert Metcalf [the inventor of Ethernet] says that if something comes along to replace Ethernet, it will be called Ethernet', so therefore Ethernet will never die. Unix has already undergone several such transformations." -- Ken Thompson

How do you recognize a good API? It's tough, but one thing is sure, a good interface allows easy swapping of components. If it doesn't allow easy swapping of components, it's not a good interface.

"Linux has always been multiple components that you can chose which one suits you best - whether its vi or emacs, gnome or kde, sendmail or postfix, apache or nginx, etc. This is a good thing, where you can swap out component A for B for any reason, and keeps the project competing with each other to get better and better." -- gbjbaanb.

When your component can be easily replaced, because the interfaces are good, it can only compete on its technical merits. That is how you will recognize the best interfaces, because when they are evaluated and put into use by skilled programmers, the best ones will come to the fore.

Time to wrap this back to systemd.

Systemd has gained traction as a init engine because the "unit file" interface provides benefits that a lot of people like. If the "unit file" interface proves durable, then it will last longer than the code. Interfaces can be immortal, code can not.

Throughout systemd there is a lack of understanding of proper interfaces. Making the GUI depend on a particular init system is a particularly obvious example of poor design, but the code was written from a 'code first' perspective rather than an 'interface first' perspective.

Lennart Poettering is a fine programmer. (His code is readable, and let's be honest, anyone who can hack on the Linux kernel has skill. Furthermore he is passionate about it). I don't think he really understands interfaces, though. When he does, his code will move up to next level.

The previous entry investigated Gnome and logind, and found that practically speaking, Gnome does depend on logind. There are some alternatives, but they are not complete or well-maintained.

This entry discusses the dependency between logind and systemd.

Logind is deeply embedded in systemd. Here is a list of logind features, which seem like a grab-bag of vaguely related requirements. If anything, their relation seems to be they are goals of GnomeOS. Maybe Poettering looked at that list and tried to implement as many as possible to convince Gnome to adopt his software.

At a minimum we can say the name 'logind' is a poor choice for a name, since it does much more than that: the name doesn't describe what the product does.

Logind depends on systemd (for example) for calling unit scripts to shutdown or put the computer in sleep mode. In logind-dbus.c function execute_shutdown_or_sleep() you can see an example. Systemd also seems to be required for detecting hot-plugged devices and probably for other stuff. Note also there is a clear dependency on DBUS.

Logind is described as a tiny daemon. That's a lie, the thing has 15491 lines, with the largest file having 3023 lines (3023 lines is large but not necessarily bad).

As always, if you find any errors, please let me know. I'm trying to find the truth, not start fights.

Aide memoire

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#6729

> Bdale Garbee writes:
> > - - - start ballot - - -
> > We exercise our power to decide in cases of overlapping jurisdiction
> > (6.1.2) by asserting that the default init system for Linux
> > architectures in jessie should be
> > D systemd
> > U upstart
> > O openrc
> > V sysvinit (no change)
> > F requires further discussion
> > Should the project pass a General Resolution before the release of
> > "jessie" asserting a "position statement about issues of the day" on
> > init systems, that position replaces the outcome of this vote and is
> > adopted by the Technical Committee as its own decision.
> > - - - end ballot - - -
> I vote D U O V F.

On Sat, Feb 08, 2014 at 12:16:51PM -0800, Russ Allbery wrote:
> I vote:
> D U O V F

On Sat, Feb 08, 2014 at 02:18:39PM -0800, Steve Langasek wrote:
> I vote F U D O V

On Sat, Feb 08, 2014 at 02:51:13PM -0800, Don Armstrong wrote:
> I vote D > U > O > V > F.

On Sat, Feb 08, 2014 at 02:57:52PM -0800, Keith Packard wrote:
> I vote:
> 1. D
> 2. U
> 3. O
> 4. V
> 5. F

On Sun, Feb 09, 2014 at 01:04:31PM +0000, Colin Watson wrote:
> I vote UDOFV.

On Sun, Feb 09, 2014 at 07:15:58PM +0000, Ian Jackson wrote:
> I vote F, V, O, U, D.

On Tue, Feb 11, 2014 at 09:07:11AM +0100, Andreas Barth wrote:
> Thus voting U, F, D, O, V.

So that's all the votes in, by my count. Summary is:

    4x D U O V F (bdale, russ, keith, don)
          F U D O V (steve)
          U D O F V (colin)
          F V O U D (ian)
          U F D O V (andi)

Note that only Ian ranked sysvinit above upstart or systemd.

First sighting of a non-AC anti-systemd troll with the
"systemd doesn't log lie"

http://news.slashdot.org/comments.pl?sid=7312219&cid=49545987

Maybe it's an honest mistake, but how?

UPDATE: Some people have notified me of ConsoleKit2, which Gnome might be able to use instead of systemd. It is not mentioned in this post, so it's not clear that it actually works with Gnome. When I have time I'll dig in deeper to see if ConsoleKit2 works or not.

This post is flotsam related to some dependencies. If you note any mistakes, please let me know, I'm collecting information.

Gimp doesn't depend on systemd as far as I can tell. Gimp runs on Windows. It does depend on libgtk, whose dependency list is available.

There are reports that Gimp depends on systemd, but that seems to be an artifact of a package dependency tree on one system, not that Gimp actually needs systemd.

What about Gnome? Does Gnome depend on systemd? The answer is yes....., although there are alternatives. Systembsd, systemd-shim, and LoginKit.

LoginKit doesn't seem to be complete, like some files were missing from commit or something. The README is attractive, though. It seems to only be attempting to provide services for logind.

Systembsd provides hostenamed, localed, logind, and timedated.

Systemd-shim provides c-group services, some unit file services, power/sleep/reboot services, and ntpdate services.

Neither code base has been updated in four months, and the code in neither one looks particularly well organized.

I suspect there are other things systemd provides that Gnome relies on, but those aren't immediately apparent. I'm not sure where to look to figure that out.

Before logind, Gnome used ConsoleKit to provide login support. Fascinatingly, the commit to remove support for ConsoleKit was made by Florien Mullner.

A lot of this stuff depends directly on DBus. In my opinion that is a mistake; the communication mechanism should be separate from the API interface.

In the next journal entry I will investigate how deeply libsystemd is tied into the init system, or whether it is truly modular.

In this post, Lennart explains that he doesn't know how to write portable C code. He says:

The purpose of my post here is to explain how to write portable code, without #ifdefs on every other line. If that happens, you are doing it the hard way.

An easier way is to create an OS independent layer. Call it OSI (since no one is using that acronym anymore, right?). Create a header file that declares all your functions, then create an implementation file for each platform. Every time you find a piece of functionality that is different on different platforms, throw it into that header file.

For example, if you need to take a screenshot, it will be different on every platform you find. In the header file, create a function OSI_Screenshot(). Then in your OSI_Windows.c file, add an implementation for windows. In your OSI_Linux.c, add a screenshot implementation for Linux. In your OSI_OSX.c, add your implementation for OSX. This technique can also work for extremely incompatible platforms, with OSI_printf(), or OSI_malloc(), etc. If you use your compiler correctly, you can compile it without any extra overhead from the function call.

An example of this technique can be found in Android (look at sysdeps.h and sysdeps_win32.c), demonstrating that this technique works. They don't have #ifdefs scattered throughout their code. The Android example is tougher than it needs to be, because Android added the compatibility layer after the code was written. If they had started at the beginning, the code would have been much simpler.

The lack of knowledge (about how to write portable code) in the systemd team is causing them to make bad decisions, because the alternative seems too hard. For example:

Those who don't understand the failures of the past are destined to repeat them. Plenty of vendors try to focus on a single platform, and their work disappeared, to be replaced by something that was cross-platform compatible, and usually better work. The discipline required to keep things portable results in better code (there's a quote from Mythical Man Month that fits here but I'm too lazy to look it up). The Gnome developers understand the importance of portability but that's a story for another post.

(Note: When you write code, you're making a UI for programmers. Learn to do it well.)

Below you can see a traditional unix init script. It's long, but if you're familiar with shell-script you can figure out what is going on. There is a lot of redundancy here, most init scripts have a switch that runs an option based on the first command-line parameter, for example. One solution is to put common code in a function, but Poettering decided to use config files.

Let's examine the general tradeoff between putting code in a function, and using config files (a form of declarative programming). Config files are fine as long as there aren't too many special cases. If there are too many special cases, you end up with so many options and keywords that it would have been easier to just use a scripting language.

The good side is systemd saves a lot of typing. Way down at the bottom, is a unit file for the same init script. It's clearly shorter, and easier to type.

The bad side is it has arcane keywords which are are not discoverable merely by looking at the file. This is a pattern that repeats itself over and over in systemd, things are easier if you know how to do them, but the system itself is inscrutable without arcane knowledge.

Ideal systems fulfill the requirements while making it easy for those who want to dig deeper. The system opens like the petals of a rose.

#!/bin/bash
# Starts the abrt daemon
#
# chkconfig: 35 82 16
# description: Daemon to detect crashing apps
# processname: abrtd
### BEGIN INIT INFO
# Provides: abrt
# Required-Start: $syslog $local_fs
# Required-Stop: $syslog $local_fs
# Default-Stop: 0 1 2 6
# Default-Start: 3 5
# Short-Description: start and stop abrt daemon
# Description: Listen to and dispatch crash events
### END INIT INFO
 
# Source function library.
. /etc/rc.d/init.d/functions
ABRT_BIN="/usr/sbin/abrtd"
LOCK="/var/lock/subsys/abrtd"
OLD_LOCK="/var/lock/subsys/abrt"
RETVAL=0
 
#
# Set these variables if you are behind proxy
#
#export http_proxy=
#export https_proxy=
 
#
# See how we were called.
#
 
check() {
    # Check that we're a privileged user
    [ "`id -u`" = 0 ] || exit 4
 
    # Check if abrt is executable
    test -x $ABRT_BIN || exit 5
}
 
start() {
 
    check
 
    # Check if it is already running
    if [ ! -f $LOCK ] && [ ! -f $OLD_LOCK ]; then
        echo -n $"Starting abrt daemon: "
        daemon $ABRT_BIN
        RETVAL=$?
        [ $RETVAL -eq 0 ] && touch $LOCK
        echo
    fi
    return $RETVAL
}
 
stop() {
 
    check
 
    echo -n $"Stopping abrt daemon: "
    killproc $ABRT_BIN
    RETVAL=$?
    [ $RETVAL -eq 0 ] && rm -f $LOCK
    [ $RETVAL -eq 0 ] && rm -f $OLD_LOCK
    echo
    return $RETVAL
}
 
restart() {
    stop
    start
}
 
reload() {
    restart
}
 
case "$1" in
start)
    start
;;
stop)
    stop
;;
reload)
    reload
;;
force-reload)
    echo "$0: Unimplemented feature."
    RETVAL=3
;;
restart)
    restart
;;
condrestart)
    if [ -f $LOCK ]; then
        restart
    fi
    # update from older version
    if [ -f $OLD_LOCK ]; then
        restart
    fi
;;
status)
    status abrtd
    RETVAL=$?
;;
*)
    echo $"Usage: $0 {start|stop|status|restart|condrestart|reload|force-reload}"
    RETVAL=2
esac
 
exit $RETVAL

-----------------------------------------------------------------------------------------

[Unit]
Description=Daemon to detect crashing apps
After=syslog.target
 
[Service]
ExecStart=/usr/sbin/abrtd
Type=forking
 
[Install]
WantedBy=multi-user.target

Reference to the examples.

There's a Debian debate page, but it's disappointing and everything systemd does is listed with equal value. Thanks to Russ Albery for making a much more balanced assessment, explaining what he likes. The short answer to the question is: SystemD makes things much easier for people writing init scripts. It wasn't about cgroups, or speed, or login managers, it was about writing easy init scripts.

Here are the major complaints he has with the traditional startup system:

He furthermore points out these problems with startup scripts:

Those are the main things that systemd fixes for a distro builder, and probably why so many distros have switched to systemd, because it was built for them.

Systemd is a descendent of Apple's launchd, Lennart suggested the world watch this movie to learn about it, so why? What is good about launchd?

OSX has an excellent inter-processing message system, part of the mach kernel (which they got from CMU). Because of this, many times processes talk to each other through the standard messaging queues. If you use a library to launch a browser window, or query wifi strength, it will communicate through a message queue (although the library takes care of the details).

The interesting thing is that launchd can open a message queue before the recipient is running. So launchd doesn't launch services until they are needed, and if they aren't needed, they never get launched. The benefit is that resources are preserved, things don't get launched until someone sends a message.

The benefit here is dependency resolution. Instead of forcing a deep calculation of what depends on what, or forcing the server script to declare everything it depends on, you can say "manage all these thousand services!" and only the few that are needed will be used, when they are needed.

Another thing I think Lennart copied from launchd is the declarative config files. You don't need to write a shell script, you merely need to indicate what should be launched, and maybe give it configuration options like "relaunch on failure" or "launch completely before receiving message." Of course, some people like config files, others prefer scripts.....that's not an argument I want to get into here, I'm merely pointing out things that seem to have come from launchd. (

What is bad:

Launchd is utterly undiscoverable. It would have been nice of them to put a README in /etc/rc/ or something saying, "hey, all your startup scripts are gone, look elsewhere."

Launchd has a complex hierarchy of directories that it searches for startup scripts. They have justifications for why, and a reason behind each one.......but justifications and reasons are not the same as good design. Every bad design decision ever made had a justification and a reason.