One of the design goals of systemd is to check if services are running, and restart them automatically if they aren't. One of the criticisms is that this should be done in the daemon, not in systemd (for various reasons, one of which is addressed here).

In some cases it's easy to detect if a daemon has failed: when the daemon no longer has any processes running, you know it has failed. That's a fairly trivial case of course, and you'd hope the daemon is well written so it doesn't crash.

But that's not really what you want, what you want is to make sure the service is still up and responding correctly. There is no automated way to do that. In other words, you need to have deeper knowledge of the daemon in order to make sure it is still running.

One idea: you could have the service send a message to systemd every few seconds, letting systemd know it is alive. This is a heartbeat, but it is again relying on the writer of the daemon to make sure he does it correctly. If you've seen a lot of code, then you know that someone will set the heartbeat message to send in a separate thread, and you'll end up with a situation where the entire program has deadlocked, and nothing is running except the heartbeat thread.

These are the limitations systemd lives with: the limitations any monitoring system would live with; that automatic monitoring of processes is not really possible. If processes are crashing often, the answer isn't to improve monitoring, it's to fix the process.

I've written other entries (some positive some negative) about systemd here. The big problem I see with systemd is outlined here.

Systemd pid 1 takes up 1.5 megabytes on my Fedora system hard drive. For comparison, sysvinit compiles down to 40k. For security purposes, smaller is better, but getting the interfaces right is far more important.

So what is systemd doing with that 1.5 megabytes of compiled code? To answer that question, we find a good place to start, src/core/main.c, which is 2085 lines long. That is not necessarily too long, but if you are a fan of short functions, you will be horrified to learn that the main function is 825 lines long.

Typically a long function starts out well organized, but as people add to it over time, it becomes more and more chaotic, and that is starting to happen here. It is not always clear the order of why things happen, for example. If I were going to refactor this, it would look something like this:

main() {
    init();
    main_loop();
    cleanup();
}

So what is it doing with all those 825 lines of code? I went through and made a rough outline, which I wrote in code form. It could be further simplified, of course, but this will give you a basic understanding of what is in the main function, and it's better than 825 lines of chaotic code:

int main(void) {
 
  getTimestamp();
  parseSomeCommandlineArgs();
  setPIDNameToSystemd();
  setUMaskToZeroIfRequired();
 
  initFilesystems();
  initSELinux();
  initSMACK();
  initIMA();
  initKernelClocks();
  initLogging();
 
  initDefaultUnit();
  initJoinControllers();
 
  mountKernelVirtualFilesystems();
  initSignalHandlers();
  parse_Config_Commandline_Environment_and_KernelCommandLine();
  daemonize();
 
  initIfRunningAsManagerSystem();
  initIfRunningAsManagerUser();
  managerSetup();
 
  setCapabilityBounding();
  enforceSyscallArchitectures();
 
  queueDefaultJob();
 
  do{
      runMainLoop();
  }while(handleStateChange());
 
  reexecuteIfNeeded();
 
  cleanupAndShutdown();
}

I tried pasting the original function here so you could compare, but Slashdot apparently doesn't allow posts that long and chopped it off. If you want to see it, you'll have to put some extra effort in and click on this link.

When code gets too long, you need to break it into smaller pieces, so the human brain can see how the code is organized. When you refactor like this, it's important to make sure that the refactored code shows the overall structure, rather than obscuring it.

As always, please comment or correct me below, I am interested in hearing what you think!
(I've written other reviews, too, check them out).

In some ways, I think of Lennart Poettering as a product manager. He figures out what people want, then builds it. The architecture is not always the best, but 'customers' don't tend to care. The point here is that some people do want some of the features of systemd, and Lennart figured that out.

Lennart's system is sometimes laughably bad (for example, he'd prefer to get rid of BSD because he doesn't know how to write portable software), but he has done the community a service by bringing up the topic of system init. More and more, we are seeing thought provoking commentary, ideas on what a good init should be like. This wouldn't have happened if Lennart hadn't built systemd.

This is how the community should be. One person sees the problem, and tries to solve it. Other people come and say, "I can think of a way to improve that." Eventually we arrive at a solution that is elegant, and effective. Init systems have been controversial for decades, which is why BSD and SysVinit are different. Every major Unix vendor has tried to come up with a solution to that problem, and in each case come up with something rather mediocre. Until recently, the Linux and BSD communities have kept what they had, preferring not to change until something comes that is clearly better. When the better answer comes along, it will be obvious.

Systemd won't be here for the long term, but it's effects will be felt through spurring the community to solve this problem that has plagued the community for decades, again, analyzing the problem with things like this (I'd also like to think my own analysis is worth something).

I think a good start would be to add functions to POSIX like "daemonize()" and "start_dameonize()" "complete_daemonize()" but maybe I'm wrong.

Book about programming.

As an intermezzo while investigating the stability of APIs (since that's not a fun thing to do), here is an analysis of some code: examining random-seed.c, pulled September 1st, 2015 ( link here, for as long as github is around).

When you come across a piece of code, the first thing you ask is, "what does this do?" In this piece of code, you can see the main() on line 34, so we know it's a stand-alone utility. That is the only function in a 165 line file, so come on Poettering, break your code into easy to read functions. It's not 1985 anymore.

Since it's not clear what the utility does from the comments, a web search might be helpful, which finds the man page, which doesn't tell you much. For example, on line 44 of random-seed.c, we can see that a command-line argument is required. However, the man page doesn't tell you what options are available, it's biggest use is telling you where the seed is stored on the filesystem (if you're wondering where that file location is defined, it's in the Makefile.am). The documentation is bad.

The program itself isn't helpful either, it doesn't respond to the --help option. If you give it any option that is not correct, it responds "Unknown verb '%s'." So keep guessing random verbs until you get the correct option.

Moving along to my main complaint, you have to read many lines of this file before you understand what it does. A program like this can be mostly self-documenting, if you structure main correctly. Here is an example of how it could be.

int main(int argc, char **argv) {
      init();
      parseCommandLineArgs();

    if(shouldLoad) {
              writeSeedFromFileToDevUrandom();
    }
    else if(shouldSave) {
              writeSeedFromDevToFile();
    }
      return 0;
}

This can still be improved. There can be comments explaining why things happen. It is much easier to see what the program does when it's broken up like this. The main algorithm should be clear.

One more serious criticism about this file, hard-coding the output file in Makefile.am is double bad: first because it shouldn't be hardcoded, it should be a command-line option. Second because build files should store as little information as possible.

That's a lot of criticism, but they're relatively minor. With a few fixes, the code is cleaned up. The big problems in systemd, the things that really worry me, were mentioned in journal 9. Those are the things that will cause serious degradation over time, and problems for all of us.

This entry discusses the benefits of good interfaces, then gives a (overly simple) metric for how to recognize them. Then it ties it back to systemd.

The 'unix way' is built around solid interfaces. The actual code to Unix is mostly gone (unless you use SCO or something), but the interface lives on with multiple (mostly compatible) implementations. The interface long outlasted the code. (NB: Unix hasn't always had good interfaces in every place, Unix isn't perfect.)

Another example of a good interface is the Berkeley networking API. It's not user-friendly, but as a low-level 'plumbing' API, it is extremely flexible and effective. It spread and now every OS has an implementation. (That is, every OS more complex than some stuff at http://wiki.osdev.org ). The original code is gone, but the interface will be around for a long, long time.

There are plenty of examples of interfaces in Unix. It doesn't have to be an API.

If you consider a programming language to be an interface between a computer and a human, then think of the C compiler. The original C compiler is no longer in use, but the interface was designed clearly enough that even intentionally bad IOCC code from 30 years ago still compiles.

Cron is another interface, as are the basic CLI commands like ls, mv, ps, grep, etc (which have now taken over almost everywhere....the interfaces, not the original code). The pipe is a powerful interface that seems obvious now, but wasn't at the time......the pipe makes it possible to take the output from any program and send it as input to any other program. That is impressive.

"Robert Metcalf [the inventor of Ethernet] says that if something comes along to replace Ethernet, it will be called Ethernet', so therefore Ethernet will never die. Unix has already undergone several such transformations." -- Ken Thompson

How do you recognize a good API? It's tough, but one thing is sure, a good interface allows easy swapping of components. If it doesn't allow easy swapping of components, it's not a good interface.

"Linux has always been multiple components that you can chose which one suits you best - whether its vi or emacs, gnome or kde, sendmail or postfix, apache or nginx, etc. This is a good thing, where you can swap out component A for B for any reason, and keeps the project competing with each other to get better and better." -- gbjbaanb.

When your component can be easily replaced, because the interfaces are good, it can only compete on its technical merits. That is how you will recognize the best interfaces, because when they are evaluated and put into use by skilled programmers, the best ones will come to the fore.

Time to wrap this back to systemd.

Systemd has gained traction as a init engine because the "unit file" interface provides benefits that a lot of people like. If the "unit file" interface proves durable, then it will last longer than the code. Interfaces can be immortal, code can not.

Throughout systemd there is a lack of understanding of proper interfaces. Making the GUI depend on a particular init system is a particularly obvious example of poor design, but the code was written from a 'code first' perspective rather than an 'interface first' perspective.

Lennart Poettering is a fine programmer. (His code is readable, and let's be honest, anyone who can hack on the Linux kernel has skill. Furthermore he is passionate about it). I don't think he really understands interfaces, though. When he does, his code will move up to next level.

The previous entry investigated Gnome and logind, and found that practically speaking, Gnome does depend on logind. There are some alternatives, but they are not complete or well-maintained.

This entry discusses the dependency between logind and systemd.

Logind is deeply embedded in systemd. Here is a list of logind features, which seem like a grab-bag of vaguely related requirements. If anything, their relation seems to be they are goals of GnomeOS. Maybe Poettering looked at that list and tried to implement as many as possible to convince Gnome to adopt his software.

At a minimum we can say the name 'logind' is a poor choice for a name, since it does much more than that: the name doesn't describe what the product does.

Logind depends on systemd (for example) for calling unit scripts to shutdown or put the computer in sleep mode. In logind-dbus.c function execute_shutdown_or_sleep() you can see an example. Systemd also seems to be required for detecting hot-plugged devices and probably for other stuff. Note also there is a clear dependency on DBUS.

Logind is described as a tiny daemon. That's a lie, the thing has 15491 lines, with the largest file having 3023 lines (3023 lines is large but not necessarily bad).

As always, if you find any errors, please let me know. I'm trying to find the truth, not start fights.

UPDATE: Some people have notified me of ConsoleKit2, which Gnome might be able to use instead of systemd. It is not mentioned in this post, so it's not clear that it actually works with Gnome. When I have time I'll dig in deeper to see if ConsoleKit2 works or not.

This post is flotsam related to some dependencies. If you note any mistakes, please let me know, I'm collecting information.

Gimp doesn't depend on systemd as far as I can tell. Gimp runs on Windows. It does depend on libgtk, whose dependency list is available.

There are reports that Gimp depends on systemd, but that seems to be an artifact of a package dependency tree on one system, not that Gimp actually needs systemd.

What about Gnome? Does Gnome depend on systemd? The answer is yes....., although there are alternatives. Systembsd, systemd-shim, and LoginKit.

LoginKit doesn't seem to be complete, like some files were missing from commit or something. The README is attractive, though. It seems to only be attempting to provide services for logind.

Systembsd provides hostenamed, localed, logind, and timedated.

Systemd-shim provides c-group services, some unit file services, power/sleep/reboot services, and ntpdate services.

Neither code base has been updated in four months, and the code in neither one looks particularly well organized.

I suspect there are other things systemd provides that Gnome relies on, but those aren't immediately apparent. I'm not sure where to look to figure that out.

Before logind, Gnome used ConsoleKit to provide login support. Fascinatingly, the commit to remove support for ConsoleKit was made by Florien Mullner.

A lot of this stuff depends directly on DBus. In my opinion that is a mistake; the communication mechanism should be separate from the API interface.

In the next journal entry I will investigate how deeply libsystemd is tied into the init system, or whether it is truly modular.

In this post, Lennart explains that he doesn't know how to write portable C code. He says:

The purpose of my post here is to explain how to write portable code, without #ifdefs on every other line. If that happens, you are doing it the hard way.

An easier way is to create an OS independent layer. Call it OSI (since no one is using that acronym anymore, right?). Create a header file that declares all your functions, then create an implementation file for each platform. Every time you find a piece of functionality that is different on different platforms, throw it into that header file.

For example, if you need to take a screenshot, it will be different on every platform you find. In the header file, create a function OSI_Screenshot(). Then in your OSI_Windows.c file, add an implementation for windows. In your OSI_Linux.c, add a screenshot implementation for Linux. In your OSI_OSX.c, add your implementation for OSX. This technique can also work for extremely incompatible platforms, with OSI_printf(), or OSI_malloc(), etc. If you use your compiler correctly, you can compile it without any extra overhead from the function call.

An example of this technique can be found in Android (look at sysdeps.h and sysdeps_win32.c), demonstrating that this technique works. They don't have #ifdefs scattered throughout their code. The Android example is tougher than it needs to be, because Android added the compatibility layer after the code was written. If they had started at the beginning, the code would have been much simpler.

The lack of knowledge (about how to write portable code) in the systemd team is causing them to make bad decisions, because the alternative seems too hard. For example:

Those who don't understand the failures of the past are destined to repeat them. Plenty of vendors try to focus on a single platform, and their work disappeared, to be replaced by something that was cross-platform compatible, and usually better work. The discipline required to keep things portable results in better code (there's a quote from Mythical Man Month that fits here but I'm too lazy to look it up). The Gnome developers understand the importance of portability but that's a story for another post.

(Note: When you write code, you're making a UI for programmers. Learn to do it well.)

Below you can see a traditional unix init script. It's long, but if you're familiar with shell-script you can figure out what is going on. There is a lot of redundancy here, most init scripts have a switch that runs an option based on the first command-line parameter, for example. One solution is to put common code in a function, but Poettering decided to use config files.

Let's examine the general tradeoff between putting code in a function, and using config files (a form of declarative programming). Config files are fine as long as there aren't too many special cases. If there are too many special cases, you end up with so many options and keywords that it would have been easier to just use a scripting language.

The good side is systemd saves a lot of typing. Way down at the bottom, is a unit file for the same init script. It's clearly shorter, and easier to type.

The bad side is it has arcane keywords which are are not discoverable merely by looking at the file. This is a pattern that repeats itself over and over in systemd, things are easier if you know how to do them, but the system itself is inscrutable without arcane knowledge.

Ideal systems fulfill the requirements while making it easy for those who want to dig deeper. The system opens like the petals of a rose.

#!/bin/bash
# Starts the abrt daemon
#
# chkconfig: 35 82 16
# description: Daemon to detect crashing apps
# processname: abrtd
### BEGIN INIT INFO
# Provides: abrt
# Required-Start: $syslog $local_fs
# Required-Stop: $syslog $local_fs
# Default-Stop: 0 1 2 6
# Default-Start: 3 5
# Short-Description: start and stop abrt daemon
# Description: Listen to and dispatch crash events
### END INIT INFO
 
# Source function library.
. /etc/rc.d/init.d/functions
ABRT_BIN="/usr/sbin/abrtd"
LOCK="/var/lock/subsys/abrtd"
OLD_LOCK="/var/lock/subsys/abrt"
RETVAL=0
 
#
# Set these variables if you are behind proxy
#
#export http_proxy=
#export https_proxy=
 
#
# See how we were called.
#
 
check() {
    # Check that we're a privileged user
    [ "`id -u`" = 0 ] || exit 4
 
    # Check if abrt is executable
    test -x $ABRT_BIN || exit 5
}
 
start() {
 
    check
 
    # Check if it is already running
    if [ ! -f $LOCK ] && [ ! -f $OLD_LOCK ]; then
        echo -n $"Starting abrt daemon: "
        daemon $ABRT_BIN
        RETVAL=$?
        [ $RETVAL -eq 0 ] && touch $LOCK
        echo
    fi
    return $RETVAL
}
 
stop() {
 
    check
 
    echo -n $"Stopping abrt daemon: "
    killproc $ABRT_BIN
    RETVAL=$?
    [ $RETVAL -eq 0 ] && rm -f $LOCK
    [ $RETVAL -eq 0 ] && rm -f $OLD_LOCK
    echo
    return $RETVAL
}
 
restart() {
    stop
    start
}
 
reload() {
    restart
}
 
case "$1" in
start)
    start
;;
stop)
    stop
;;
reload)
    reload
;;
force-reload)
    echo "$0: Unimplemented feature."
    RETVAL=3
;;
restart)
    restart
;;
condrestart)
    if [ -f $LOCK ]; then
        restart
    fi
    # update from older version
    if [ -f $OLD_LOCK ]; then
        restart
    fi
;;
status)
    status abrtd
    RETVAL=$?
;;
*)
    echo $"Usage: $0 {start|stop|status|restart|condrestart|reload|force-reload}"
    RETVAL=2
esac
 
exit $RETVAL

-----------------------------------------------------------------------------------------

[Unit]
Description=Daemon to detect crashing apps
After=syslog.target
 
[Service]
ExecStart=/usr/sbin/abrtd
Type=forking
 
[Install]
WantedBy=multi-user.target

Reference to the examples.

There's a Debian debate page, but it's disappointing and everything systemd does is listed with equal value. Thanks to Russ Albery for making a much more balanced assessment, explaining what he likes. The short answer to the question is: SystemD makes things much easier for people writing init scripts. It wasn't about cgroups, or speed, or login managers, it was about writing easy init scripts.

Here are the major complaints he has with the traditional startup system:

He furthermore points out these problems with startup scripts:

Those are the main things that systemd fixes for a distro builder, and probably why so many distros have switched to systemd, because it was built for them.

Systemd is a descendent of Apple's launchd, Lennart suggested the world watch this movie to learn about it, so why? What is good about launchd?

OSX has an excellent inter-processing message system, part of the mach kernel (which they got from CMU). Because of this, many times processes talk to each other through the standard messaging queues. If you use a library to launch a browser window, or query wifi strength, it will communicate through a message queue (although the library takes care of the details).

The interesting thing is that launchd can open a message queue before the recipient is running. So launchd doesn't launch services until they are needed, and if they aren't needed, they never get launched. The benefit is that resources are preserved, things don't get launched until someone sends a message.

The benefit here is dependency resolution. Instead of forcing a deep calculation of what depends on what, or forcing the server script to declare everything it depends on, you can say "manage all these thousand services!" and only the few that are needed will be used, when they are needed.

Another thing I think Lennart copied from launchd is the declarative config files. You don't need to write a shell script, you merely need to indicate what should be launched, and maybe give it configuration options like "relaunch on failure" or "launch completely before receiving message." Of course, some people like config files, others prefer scripts.....that's not an argument I want to get into here, I'm merely pointing out things that seem to have come from launchd. (

What is bad:

Launchd is utterly undiscoverable. It would have been nice of them to put a README in /etc/rc/ or something saying, "hey, all your startup scripts are gone, look elsewhere."

Launchd has a complex hierarchy of directories that it searches for startup scripts. They have justifications for why, and a reason behind each one.......but justifications and reasons are not the same as good design. Every bad design decision ever made had a justification and a reason.

(DISCLAIMER: These are my notes and might be wrong.)

Lennart Poettering has never heard of Rob Pike's rules of programming. Let's examine why:

He said, "On my system the scripts in /etc/init.d call grep at least 77 times.... that has to be incredibly slow...., and should be rewritten in C."

You can time it yourself, with this command:

time for i in {1..77}; do grep user /etc/passwd > /dev/null; done

On my machine it takes 0.229 seconds.

Lennart didn't get the speed increase he wanted. From his benchmark report, "booting up with Upstart takes 27s, with systemd we reach 24s." This result isn't too bad: when I've made a similar mistake of writing code before timing anything, ignoring Pike's rules for programming, once I actually slowed the system down.

If you don't measure, you can't optimize.

For entertainment and enlightenment, here is a quote by Ken Thompson talking vaguely about Pike's rules for programming:

DISCLAIMER: THIS CODE REVIEW IS A LONG WORK IN PROGRESS, I COULD BE COMPLETELY WRONG IN ANYTHING I SAY.

To do a proper code review, you need to understand the purpose of the code, what all the stakeholders want. From my own perspective, init scripts work fine, but since Unix companies keep trying to create new init systems, they must have different needs than I do.

Here's a list of the stakeholders. I need to figure out what their goals are.
1) System admins.
2) Desktop users.
3) Distro builders.
5) Android (if systemd turns out to be good enough).
4) Programmers

My suspicion is that systemd has taken over because it makes things easier for 3.

At its core, Unix is a system for programmers. What other system comes with a compiler and multiple interpreters by default? Bash is so much more useable than DOS, or even powershell (yeah, go ahead and flame me but I'm right, Powershell doesn't even have < working). Unix was designed by programmers and for programmers.

The reason I'm talking about it is the traditional init process is incredibly discoverable. All you have to do is look in the /etc/rc directories, and you can figure out how your system boots. You can see it all. For a programmer it's easy. Poking around in the system to understand it is one thing that makes Unix great (and what I like about Slackware: the whole thing is lovingly crafted).

So that describes the approach I am taking to code review, and to the init process. Hopefully Systemd is an improvement.

On 9 October 2012, the game XCOM: Enemy launched... launched in the US. Unknown to me, the EU launch date was to several days layer, 12 October 2012. Maybe. Yet, 9 October 2012, I received an SMS from Gamemania.nl a dutch gaming retailer chain, that my copy was ready to be picked up. So I left work early that day to arrive 17:54 in front of the store. Doors pulled almost shut, store had already closed and refused to serve me. Very well, I thought there are other stores in the world, so I bought it the next day at Free Record Shop in Amsterdam. Then when I came home, I tried to install. First I had to install steam, which crashed, crashed and crashed some more but finally I got it working and had to create an account. Then activate my email. Then I installed the game and was told it was not released. What? If the game was not released, what was I holding in my hand? Note the error message mentioned nothing about a region or what would be the release date. Just not released. By google came to my aid and I found that throughout Europe, the game was available for sale but not yet ready for install. I read this from angry users posts. Not a single forum had an official answer yet. Not yet and counting. Even the official release date was less then clear. But I know my Internet, if Steam, Dutch retailing, 2kgames/firaxis couldn't/wouldn't help me, maybe some pirates would? thepiratebay itself is of course famously blocked in holland but there plenty of mirrors around. So I checked and yes, full downloads were available in various flavours for a total cost of ZERO bucks! And if you had issues, then the supplies answered your question in minutes. Not like the hours, days and counting before getting a reply from people I had payed money too. Many a reply to a piracy story has had comments similar to my story, so what is so special about it? Nothing. Just that after years of downloading, I have with MMO's gotten used again to paying and I didn't have any issue with paying for this game, if it had worked. But I do have an issue with paying 50 euro's for a game that can't be played and that now that I have read the forums I have seen is filled with bugs. Bugs the official forums have no answers for but that are fixed on piracy forums. To repeat myself, for this game the people that wanted me to pay did:

• Act as if my giving them money is a favor they are doing me and only when they feel like it, opening hours be damned.
• Not reply in a timely manner (or at all) to complaints
• Treat Europeans as second rate customers for no reason (what are they afraid of, that a world-wide release will overload the servers)
• Break consumer laws by selling a product not fit for its purpose (a game that can't even be installed is obviously not a fit product)

Meanwhile, the pirates offer:

• Early access
• No charge
• Free, fast useful support by computer experts.
• Service available any day of the week at any hour.

Sometimes the anti-piracy people complain the content industry can't compete with free. But come ON! I had PAYED already and the companies just said "no". Meanwhile the group that doesn't want money, said "yes". This is like paying a hooker to have an headache while your wife is stuffing your wallet full of money and begging for sex. Something ain't right!

And this is why I pirate. Because how else can I send the signal that I am not a sheep who will just keep turning the other cheek? Sure, there are sheep who advocate just that, just wait 3 days, it is not the developers fault etc etc. FUCK THAT! Nothing is every anybodies fault and I as a consumer should just take it all and keep quiet.

NO! And that is my reason why I post about being a pirate. Because just downloading alone isn't enough. Consumer boycotts don't work, there are to many sheep drowning out the silent protest of people like me who just see no other option but to not pay to make it clear I expect more service for my cash.

Because I see no other option. Mails go unanwered, forum posts get ignored, I can get my money back from the store and the sales clerk don't care, not his problem. How can I HURT that manager who thought it was a good idea to do a staggered release, hurt that Steam admin who didn't just flip a switch to prevent customers getting angry. How can I even get the companies involved to acknowledge my existence?

I can't. But I can keep my money in my pocket. That doesn't solve anything but it is a lot more fun having impotent nerd rage with cash then without.

Anyone want a beer over the backs of game developers who haven't learned that if they want an income, they need to tell their managers to not upset their customers?