Logging isn't done by or in PID1. It is done by the "journald" daemon that have its own pid. I must say, that whoever telling you about how systemd is designed and works, doesn't have a clue what they are talking about. I strongly suggest you check out the systemd homepage for some basic information about systemd.
http://www.freedesktop.org/wik...

People are hard to satisfy: some complain that systemd replaces daemons, and now you complain about it _doesn't_ replace syslog for writing text logs?

People who wants simple text-logs can still use rsyslog or whatever they have always used, and have "journald" forward all the messages to it. That way rsyslog can gain some of the systemd benefits like early boot logging, though not them all (rich meta data like monotonic stamps and whatnot). All in all you would be better off than simply using SysVinit and pure rsyslog.

Even without syslog, it is trivial to convert systemd's journal to text. You can even export it as JSON to preserve the rich meta data.

I really think the systemd developers did their homework very well with the design of the logging system. The more I use it, the more I like it. There is no going back for me to plain text logs.

The point is that most user processes doesn't communicate directly with the kernel, but with a cgroup manager using a higher level API, like systemd or cgmanager. So while the kernel works fine, the "communication link" or the mediator between processes and the kernel would be severed if the cgroup manager in PID2 died. That is highly likely to lead to an unstable system.

Anyway, looking at the design of systemd, you will see that it actually runs very little code in PID1, it just communicate via PID1 to the other helper processes. This enables PID1, the init part, to have total control and supervision of all processes. Making e.g. the cgroup manager run in PID2, wouldn't gain anything, but increase complexity, since there would need to be intense communication between PID1 and PID2 in order for the init system to use cgroup to track all processes.

I think the whole PID1 debate is based, not on hard facts or good analysis, but on a hatred against systemd. People seems to rely on random hate blogs on the net for what systemd is, instead of examining the systemd project for themselves.

Systemd's PID1 is actually rather small, especially compared to what functions it provides. Moving some core bits out of PID1 and place them in PID2 will just increase complexity with little if any gain.

The reality is, that the systemd designers made some good choices when making systemd. Personally I think it is hopeless for all the simple script based init-systems to have real feature parity with systemd, unless they make some of the same design decisions.

logind isn't a feature with "systemd" (the daemon), but with systemd (the project). "logind" is a consumer of the systemd-daemon's internal API however. That means you can use cgroups and other kernel and systemd features in user sessions. This is how eg. secure root-less X.org is possible with systemd.

I think many peoples idea what init is, have been clouded by the fact that traditional Linux init systems were so primitive. Certified Unix'en like Solaris and Mac OSX have abandoned crude Linux like init systems years ago.

Booting and initialization of a system is quite complex on modern OS's, so doing it by modules that aren't coordinated and aren't developed with the other modules in mind, really limits what the OS can do. Having modules like systemd's, that are designed to talk to each other, all other processes and the kernel, and is developed in a coherent manner, really can remove some old limits on how Linux works. Basic things like conferring "namespaces" and "capabilities" from the kernel to a service, just by adding a simple keyword in the service config file, shows the potential. But multi-seat computing, stateless booting and "zero config" boots are either realized already or being worked upon. With kdbus in the kernel, secure OS containers from basically unmodified Linux distros, will become a reality.

I also think it is good, that systemd now will reduce Linux fragmentation at some level at least, and freeing distro maintainers and developers from a lot of non-fun work like maintaining and debugging init-scripts, while making cross distro collaboration on e.g. extra hardened "Unit" files (service configs) possible.

Here is a youtube video where Lennart Poettering talks about why systemd goes beyond a simple init system:
https://www.youtube.com/watch?...

I strongly recommend reading as many relevant sections from this site:
http://www.freedesktop.org/wik...

There are also some design documents (like this old one about the journal)
https://docs.google.com/docume...

And of course more youtube talks from Fosdem etc:
https://www.youtube.com/watch?...

systemd's logging facilities are superior to old syslog in several ways. Fx. systemd and journald lives in initramfs while the system is booting, so it can collect logging info from before the root fs is even mounted. Since systemd actually have knowledge about mount points and files systems, it can also delay to the very last moment the things needed for journald to write to the log.

There is also kernel guarantee that the daemons/pid's/programs that appear in the log are the real ones since systemd have total control and supervision of all processes. So if "lp0: printer on fire" appears in the log, you will know whether it is a prank or a real message.

The structure and index makes it possible to store lots of interesting meta-data, like monotonic time-stamps, GUI, PID, command line, marks from where every boot started and ended ("journalctl -b -p err" shows all loglevel "error" messages generated last boot only). Using monotonic timestamps to compare two boot sequences on perhaps two similar machines, is quite interesting and very easy to do.

It also allows for multi-language log support, supplementary help files that can explain what the error message means, suggest how to solve the problem, etc.

Here is a list of fields in the journal. Because of the index, the journal have real time knowledge about what is written in the fields, so there is tab completion and extremely powerful sorting available:
http://www.freedesktop.org/sof...

If you care about logging with systemd, take a look a the original design document behind journal
https://docs.google.com/docume...
Notice how simplicity is design goal number 1.

If you intend to remain (a paid) Linux sysadmin in the future now where all major distro are starting to convert to systemd, you should really study systemd's journal.
It is much easier to "get" the power of a indexed and structured log file by trying it, than describing it. Fedora 20/CentOS 7 are reasonable choices to learn it on.

check out systemd's "nspawn" too: OS containers really are the future.

And that is part of the problem, that it is presented as a init vs init issue.

It has long since grown past that. Systemd, as a package, now holds udev, journald, its own cron replacement, a network manager, dhcp, ntp, inetd, etc etc etc.

There are several inaccuracies on that list; systemd has timers that may be used as a cron substitute, but really is a different design, so therefore cron can happily co-exist, and nobody forces anybody to use the systemd solution. Systemd doesn't have a replacement for a NTP server, it just has a simple sNTP client. Like the systemd dhcp client, it was made with a special user case in mind, namely OS containers, where existing solutions simply where too slow and noisy (when starting 1000 OS containers at the same time, it really matters if each dhcp client connection takes 60 seconds or 3). AFAIK, the dhcp code is mostly a library that can be ripped out of the systemd source tree.
Nobody forces anybody to use them; they are alternatives, not replacements.

Again, systemd doesn't have an "inetd" server. Systemd uses socket activation as pioneered by inetd, but there are important differences. Lennart has several blog pages about it, here are a couple: http://0pointer.de/blog/projec...
http://0pointer.de/blog/projec...

systemd can use "inetd" prepared services, but inetd may happily co-exist together with systemd, and inetd provides services that systemd doesn't provide such as TCPMUX and RPC services.

The journal was a part of systemd early on, because in order to have great service management, you need the best logging possible.

But the crowning achievement may well be logind, that more and more DEs are getting a dependency on.

And logind can only function with systemd as the init, full stop.

logind was part of systemd early on. It has never been a separate project. It has never been meant to work outside systemd. Systemd opponents seems to have this bizarre love-hate relationship with systemd code; they curse it and the open source developers who make it, claiming the code is bad etc. etc. But at the same time they lust for the code, claiming systemd should be split up in smaller modules so they can use it anyway.

It has been a seriously mistake for the systemd opponents that they have become so fixated on systemd's "logind". Why should they care about how systemd solves user session problems? Upstream projects like Gnome and KDE have warned about for years, that an _alternative_ to logind/ConsoleKit should be developed.

Upstream projects like Gnome and KDE becomes dependent on systemd's login, because it solves real problems, like no other maintained program does. If there was an alternative API, from a program with similar functionality, they would happily support that. But no-one seems to care enough to start developing such a program.

It has been said many, many times; the systemd opponents should start to solve their own problems that lie in front of them, instead of wasting breath on trash-talking systemd.
More code, less ranting.

First, the "kludges" you are talking about, is a minimal distro agnostic compatibility layer, so that upstreams projects like XFCE can finally get basic system information without searching 20 places just to display e.g. the distro version it is running on top of.

It is one of the many examples on how systemd solves real world problems that helps other Linux developers, which is why systemd is so popular among developers. It is quite shameful that SysVinit (and similar init systems) distros haven't done anything even as basic as that.

The "systemBSD" project can't really be ported to Linux, and the crucial part, "logind" doesn't work yet and may never do. Trying to clone a version of systemd's "logind" is also a the wrong thing to do: trying to present it self as a systemd logind for upstream projects, but without supplying the necessary functions will just mean errors and crashes. A "logind" using the systemd API isn't what upstream projects like Gnome and KDE etc. have asked for the last couple of years; they want an _alternative_ API that support at least the basic features that logind or or ConsoleKit have.

To sum up: what upstream projects wants are an _alternative_ to logind/ConsoleKit, that works and are maintained. It is the most minimal request they can formulate, and is essential for providing modern DE functionality to non-systemd distros.

But even that request are ignored by the systemd opponents, they prefer to trash-talk systemd instead of solving their own problems.

Systemd is brilliantly designed and engineered, which is why it has become so popular. Its backwards compatibility is top notch; there was no flag day where every service either was a systemd service, or it wouldn't function.

They studied "launchd" and Solaris "SMF" and other Unix init systems to make an init system that was better in everything, in every detail, than what was available for Linux. SysVinit OTHO, is a notoriously bad design, where code and config options are mixed together in free form executable config files. Systemd separates code from daemon config options, so the latter are in structured text files that are easy to parse for both humans and machines.

The systemd developers really cares about details too, so it has the best bash-completion I have ever seen. It is a joy to work with the systemd CLI tools.

Vague criticisms about systemd's design really isn't much of an argument. Systemd actually works extremely well in the real world.

If PID2 is responsible for critical features like eg. cgroups which affects all running processes, including PID1, then it won't make a difference.

I do really think that the systemd designers have actually done their homework quite well when they started out. PID1 is quite small and only contain what is needed, everything else runs as helper processes.

Besides that, systemd can supervise itself (PID1) by using a watchdog, so the system can react if PID1 doesn't answer the watchdog "ping".
http://0pointer.de/blog/projec...

Do you really think it is a serious argument is that you could re-implement "logind" as a bash script? We are talking serious hardcore system stuff here, which is why no-one have made an alternative to "logind" or "ConsoleKit" despite upstream projects have pleaded for such a program for several years.

You are seriously misinformed on how systemd works and what it can do:
It uses kernel namespaces and capabilities to protect the system; this is on top of SELinux etc.

Here is a general overview:
http://0pointer.de/blog/projec...

Here are some of the config options for the daemons you can use. See "CapabilityBoundingSet=" for one way of using kernel "capabilities":
http://0pointer.de/public/syst...

There are so many freaking cool security features in systemd. As time goes by, developers, distro maintainer, and systemd administrators, can add more and more options to the running processes, like "NoNewPrivileges=" to prevent privilege escalation, or "ProtectHome=" to prevent malware and exploited processes from stealing info from /home, even if they otherwise had permission to read in home.

All this great new stuff can be turned on and used by adding a simple keyword to a structured text file. As time goes by, systemd distros will become ever more hardened.

Well, it does need to be handled somewhere; if you want features, you will get complexity, it is that simple. But as explained, the features and complexity isn't running in PID1; PID1 (systemd) is just a hub for relaying those features to other processes.

I really think so much of the systemd opponents talk about "Unix way" and "PID1" should be simple, is hand waving to gloss over the fact that the non-systemd distros have no feature parity with systemd to speak of; SysVinit is crude and no one in their right mind would design a init system these days with executable config files. Service configuration files should be non-executable text only.

General and vague criticism against systemd really doesn't convince anybody. Anyway, the Linux community have spoken with a large majority of Linux distros using systemd in the future.

If SysVinit systems really have all the features of systemd, just much better because they are simpler, you would expect a "SysVinit" boom in the future with lots of developers and users.

Personally, I think the systemd opponents are too concerned with negative campaigns against systemd, that they entirely forget to code any alternatives, so I predict ever more distros like Slackware abandoning script based init systems; they simply don't have an alternative.

The journal is primarily append based, so it designed to be very robust against corruption, the log is also rotated whenever a corruption is detected. So usually the corruption you see are a single line that was malformed when writing because e.g. the daemon died unexpectedly; this happens all the time with text based logs, but because they have no inbuilt integrity checks like systemd, it doesn't get discovered.

So in most cases you don't really loose any log info, and the rest of the log remains fully readable.
I am sure that there are subtle bugs in systemd that end up causing file corruption, or enhances the risk of corruption when the systems suddenly shuts down. But I also think that the bugs can and will be fixed, so I really don't see it as a long term problem. The many advantages by having an indexed and structured logfiles far outweighs the theoretically small chance of file corruption on modern file systems.

I don't think it is practically possible separate text logs from the metadata and get the same functionality as systemd's journal. AFAIK, the journal is more or less just a textfile+index anyway. Try running "strings" on a journal, it will show a lot of the logging info.

You can't seriously claim that systemd provides nothing that can't be done by script based init systems, shell scripts and existing daemons, logind is just one example (ConsoleKit is a dead project with no alternative).

But it would be an interesting project to make a Linux SysVinit distro that tried get feature parity with systemd, so that daemons could utilize the kernel "namespaces" and "capabilities" to massively strengthen security, or have a total supervision chain of all processes, including PID1 etc. There is so much good stuff in systemd that the SysVinit-like init distros could clone.

Isn't that argument just trying to make a virtue out of the fact, that SysVinit and the like, are totally crude and primitive init systems that are unable to anything much of interest?

All the analyses I have seen shows that moving crucial processes into PID2, just makes everything more fragile and opens up security holes.

I think that there are actually very good design reasons for why systemd is designed like it is.
It only runs one process as PID1, the daemon "systemd" which is rather small. This daemon however, is capable of "talking" with with several other processes, which gives it many advantages, since it can act as a mediator between kernel features like cgroups and "capabilities" and the processes it controls.

Sometimes new stuff is actually much better than then old stuff. I was skeptical about binary logs until I actually tried it. The advantages of a indexed journal is overwhelmingly positive. "journalctl" is an extremely powerful logfilter exactly because of the indexed and structured logs.
Systemd's journal also collects all logs in the same place, so no need to use "last" to read the binary "wtmp" log files, or locate .xsessionerrors and what not; everything goes into the journal.

Also, all the usual text tools like grep, tee, sed etc. all works with the journal by the standard Unix concept of piping. "journalctl" simply enhances the Unix text tools.

Give "journalctl" a serious spin someday; you may like it.

Yes, freedom isn't a given thing. It may cost something to gain it or keep it.

Anyway, thank you for a pleasant conversation.

Hm, I would prefer less extreme options, but I certainly agree that some fragmentation is to be preferred to a very uniform Linux. I mean, I like systemd and I do think the systemd group have some very good ideas, but they certainly don't include e.g. KDE in that future as anything else than token gestures. I would be very unhappy if Gnome ruled the Linux desktop without real competition. I have no animosity against Gnome, I just love KDE.
So there...

I agree with the general gist of what you say; Linux will only be healthy with at least some internal competition of ideas and how to implement them. As it is now, people can move from e.g. Linux KDE to Linux Gnome, if the KDE direction bothers them. With only one DE on Linux, they can only move to other OS's if they don't like what they get.
The same thing also applies to low level libraries, programming languages etc.

But on the other hand, to reduce the fragmentation is also very worthwhile goal, so I would hope that systemd could be compromise between the two extremes. IMHO, I do think systemd so technically superior to all other script based init systems, that it barely has any real technical competition anyway.

Regarding systemd as long term solution, and lock in. In some ways I think systemd have paved the way making future init system shifts much easier; with rather firm external API's it is much easier to gain momentum if the new solution is backwards compatible.

The new direction could either be a fork that later changed in a new direction, or a whole new system, that just provided external compatibility like systemd did. I am not suggesting that especially the latter will be easy, but it is doable, and since almost all distros at that point would be using systemd, they could easily change init system.
It would not be a one man solution, however; organizing and cross distro cooperation would be the key to success

As it is now, I think that a fork of systemd is the most realistic long term solution, if one find it necessary that Linux have competing projects on every level.

To me systemd is the best thing ever happening to my distro since package management, but I have no problems with people having other ideas for what they like with their distros.

My rather bleak predictions all rest on the current lack of development and the many challenges ahead, with more developers and more cooperation most of the problems can be solved.

Regarding the many different Linux configurations, then I agree with you in principle. But I don't think the fragmentation of Linux has been really helpful either. It is clear that there now is a major push to reduce Linux fragmentation.

I think the only way the smaller distros will have a say in the new direction Linux is taking at the moment, is to organize and counter it with their own proposals.

I think the "every distro is a separate island" doing everything their own particular way, is something that will disappear. But perhaps that isn't so bad, maybe the interesting thing about different distros, aren't that they all place their shared libs in different subdirs, but rather, what software platform they deliver above the system level. Less Linux fragmentation will definitely make it easier for distro maintainers and upstream developers in many respects, so perhaps this will release energy to do more cool things, instead of patching up differences. I mean, a pure systemd version of Gentoo will still be Gentoo, it will just share some basic OS characteristics with other Linux distros that will make it easier for upstream projects to support it.

I still think there will be many, many different Linux distros in the future, catering for either the mass market, or specialist use, I just think they will be less fragmented and different at the core system level, thanks to systemd etc.

It will make it easier for the eudev fork to keep up with udev, but then someone will have to make a userspace kdbus alternative to what systemd does as PID1, that appears to be a non trivial thing to do, involving some really low level stuff. So it will require some serious brain and developer power going that way, with possible implication for the rest of the distro, and the unavoidable back-clash from downstream eudev users, since that will force changes in their distros too.

Even if you don't use PA, you still benefit from the colossal cleanup of ALSA and the audio drivers PA made. But my point was merely, that PA has been on all major desktop distro for years, despite a vocal minority's endless negative campaigning against it. When people realize they have been using PA for a long time without problems, then it sounds hollow when PA opponents rants that PA doesn't work, or that Lennart Poettering can't code etc. No one ever developed an alternative to PA, simply because it was an insanely hard task to make a system wide sound daemon for Linux, so the PA opponents couldn't point to another sound daemon that people should use instead. The bottom line is, that negative campaigning utterly failed to keep PA from being _the_ standard Linux sound daemon, in much the same way, negative campaigning have failed to make systemd the _the_ Linux init system.