Coward!

Yes, the journal really is brilliant. It just solves so many decade old logging Linux problems, and make hard stuff very easy.

I was sceptical about binary log files when I first heard about it, but just playing around with "journalctl" for 10 minutes convinced me wholly.

You are just talking bullshit here. I have been using systemd for +4 years now and it has been rock stable.
Besides, systemd systems are so much nicer to debug than distros glued together with shell scripts.

Just the fact that you can have full logging and the systemd tools working from initramfs is a vast improvement, and the systemd journal beats all other Linux logging options by a huge distance; field based filtering and monotonic timestamps are just great when debugging boot problems.

Being able to do a "journalctl -b -1 -p err" is so much better than faffing around with grep and regex. (the line shows all log entries from the previous boot with the syslog severity level "error" and above, try that with grep!).

You are seriously misinformed here; systemd provides a sNTPv4 client, not a ntp-server. It is a compile time option, so no distro ever needs to use it instead of their preferred sNTP-client. It is included in the systemd project for two main reasons; clock-less ARM boards and OS containers. Both have special timing needs since eg. an OS container can be "frozen" and "unfrozen" without warning. systemd provides them both with a solution so they don't gets confused by time jumps.

But perhaps you think choice is bad and there are too many sNTP clients so systemd developers should be banned from providing one?

Like what? systemd have extremely few external dependencies. And don't try the provable falsehood that systemd inserts "hard dependencies" in other projects like Gnome/KDE.
That Gnome have had problems supporting non-systemd distros was because those distros didn't care to maintain ConsoleKit. Gnome kept on supporting CK despite it having been abandoned for +1½ year with no upstream to provide bug-fixes or security fixes.

But thanks to systemd, there are now several alternatives to ConsoleKit. Choice is good.

Again, you are really misinformed here; how can systemd ever make it harder for non-systemd distros that are using mdev or vdev or eudev?

If a non-systemd distro wants to use unpredictable network names they can do so.
With systemd distros here is how you turn off predictable network interface names:
http://www.freedesktop.org/wik...

Again, thanks to systemd the Linux ecosystem went from just having udev and mdev, to also having eudev and vdev and probably several more. So if you like choice, praise systemd for providing it.

Not necessarily. The alternative to no laws isn't bad laws.

As it is now companies can spew out insecure products with impunity and even silently drop any security support for devices consumers have just bought, not forgetting the classic tactic of not acknowledging security problems and just plain ignoring them. This can't go on.

Really, there ought to be some sensible minimum standards for commercial products that can be connected to the internet. This could include that the company had a decent policy for security fixes and a published contact point for people reporting such problems.

And how about a pre-published, minimum security support length, so that people buying a smartphone/router/etc. will know in advance how many years it will be supported with security fixes. There are "use by" dates on food, why not on all internet connected devices.

Yeah, I even found a man page that list all the man pages : "man systemd.index" and a index for all unit directives "man systemd.directives" with cross-references to the relevant man pages. Really useful stuff that shows the developers cares about end users.

systemd is a seriously well documented project. At the moment there are around 202 man-pages.
http://www.freedesktop.org/sof...

Every man-page I have used is done by the book: good examples, relevant "See also" references, and sometimes even links to further information. I really like the fact that all config files have extensive man pages too (man journald.conf fx), and that they also provide "big picture" documentation like "man bootup".

I have often seen how a question on the systemd devel list resulted in clarification of the documentation, so they take their documentation seriously.

On top of the man pages, there is also a huge amount of information on the projects homepage, spanning from overview videos, to low level developer information.
http://www.freedesktop.org/wik...

Yes, those are the basics for systemd to work. Again, rather simple requirements. It doesn't state what goes into what, because this is something the distros dictate, not systemd. The systemd developers can merely suggest where to put things, but distros won't necessarily follow those suggestions.

systemd is designed with different distro policies in mind, so they provide "Presets" to help facilitate the different policies and needs of different distros:
http://freedesktop.org/wiki/So...

The FHS is primarily concerned about directory layout, not so much the content of these. And there is nothing in FHS 3.0 that contradict the slightest that user attached storage is mounted in the /run hierarchy.
All such disc layout reorgs of the distro I have tried, have been discussed on mailing lists etc. before they are implemented. Again, this is something the distros dictate, not systemd.

Come on. It is the distros that dictates exactly what features systemd have turned on, and how the FH layout is. There are no systemd developers sneaking into the distro repos at night and secretly turning on features without documenting them.

You may not follow the eg. development mailing list of your distro where such things are discussed, nor care about reading release notes. But Linux distros have always evolved this way, with new ways of doing things, often from release to release.

I fail to see how this in any way breaks any stable subsystem.
1. systemd-resolved is an entirely optional feature. If the distros don't want it, they can just turn it off.
2. Whatever its /etc /run requirements is, this is a new feature, not a change of something old and stable. So it doesn't break anything at all.

I really don't think you can give just a single non-contrieved example of how systemd are breaking stable subsystems for users without warning.

While more documentation is always better, I think the systemd project is way better documented than most other software projects, and certainly much better documented than many people think it is.

Running two NTP-clients at the same time was an amazingly stupid thing to do even pre-systemd. AFAIK, systemd per default tries to avoid this by using "conflict" statements in the service files, so if one NTP client or daemon is run, the others are suppressed.
I don't think eg. SysVinit or Upstart ever had such measurements against accidentally running two NTP daemons at once.

The bottom line; yes, systemd developers are working towards stateless boot and other similar optional features. Yes, this will require some reorgs of how especially config-files are kept. There are good technical reasons for this (crypto verification, factory reset etc), and the layout changes seem to benefit all user cases, including those that won't be using the new features. Even those who doesn't use systemd would probably benefit from a layout with a clear separation between user, system and distro default config files.

The basic requirements for these new features are sketched out here:
http://0pointer.net/blog/proje...
But remember, this is very much a work in progress for the future, when things are are starting to actually work somewhat, I am sure that a more "official" documentation will appear that spells out all the details.
And the there is no dictate to make these features work for either distros nor upstream projects. This work will only be done if people think it is beneficial to their distro/project. systemd will work without being able to perform a factory reset or a crypto verified /usr.

The systemd requirements for the file system is documented here:
http://www.freedesktop.org/wik...
These are rather simple requirements and probably very close to the old FHS 3.0 proposal, I think /run is the only new thing here compared to FHS 2.3

AFAIK, all other changes of the FHS are merely suggestions like getting extra OS container features when using btrfs, and in case of stateless boot, something for the future. Systemd will continue to work for Linux distros that don't care for stateless boot etc.

The kernel ring-buffer size can be user specified, so no need to ever lose such messages. The problem is that during boot, only messages directed at /dev/kmsg is recorded until rootfs is mounted and syslogd is running. This is hard to do something about since it is a kernel limitation that only one device at the time can own /dev/log. journald catches all messages, including those to /dev/kmsg and /dev/log, and it does this from early boot. So a journal based system will always contain more info than a pure syslog system.

Regarding silently dropping messages: AFAIK all syslog daemons does this per design (or lack thereof). The main problem is that calls to syslog() would be blocking if implemented in a safe manner.

You misunderstand; a reason why syslog messages doesn't contain monotonic timestamps etc. is that it makes the log entries extremely long and hard to read for humans. It is a kind of limitation of the "design".
journald doesn't have such limitations since it is field based, and the output format is easily adapted to what view you want. If you only want wall time stamps, you can get that, if you want monotonic time stamps, you get that, and only that.

So with the journal, you can add more fields in the future without making the log entries unwieldy, something impossible with syslog.

Regarding human readable: I only care about that for the subset of information I intend to read, and this is exactly what journalctl does. That way you can have logs that are easy to parse for both humans and machines; each consumer gets exactly the view of the data that they want in a form they can use.

Yes, the scaling thing is a good argument against using flat file text logs. You really need indexes when analysing such huge amount of data, which is exactly what people are doing at the moment; they invest in often proprietary commercial solutions in order to massage their flat text file log data into something usable.

RDBMS have been tried for a long time and they are not a good general purpose solution for local logging. They are also much more prone to corruption than the journal's simpler binary storage format.

While regex is a powerful tool, the old joke about trying to solve a problem using regex, and then discover you now have _two_ problems, is true too. Discounting the many variations of regex that each program like grep /egrep uses, it simply becomes unwieldy when you are using regex epressions that are +120 characters long. Making ad hoc queries with regex really is painful when the complexity rises beyond the basics. The systemd journal is so much better regarding this since it is field based. It is so trivial to make complex ad hoc queries without any need of using regex.

Really, stuff like:
"journalctl --since -5m"
"journalctl -b-1 -p warning"
"journalctl /usr/sbin/smartd --since 2015-05-10"

Is so much easier than trying to construct regexes that does the same.

I assume you are talking about FHS 3.0 here, but remember, that is only a proposal, not a standard. Also note how the FHS 3.0 process has been stalled for years.
The bottom line today is that there is FHS 2.3, and then everybody doing their own thing , perhaps with some cross-distro cooperating. The end result is no common formal standard for the FHS among the distros.

Personally I don't think there will be a FHS 3.0 until the dust has settled regarding the current layout reorg,

The reorg of the FHS that the systemd developer are doing, are caused by good technical reasons. Unix and later Linux have had several of these reorgs over the decades, and it is a process that probably never stops.

So until the FHS reorg to make about OS containers, and stateless boot work in a sensible manner, the Linux FHS will be in a slight flux. Not a big problem really.
I guess at some point somebody will make a FHS 3.X out of what the systemd distros does at some point in the future. Personally I prefer this approach of practise getting formalized into a standard. That way you are sure to have both a good working standard, and a standard that people actually use.

The same is very much true for systemd's journald files. They are just appended text files with another delimiter +index, so in worst case a hex editor or "strings" may read them.

"journalctl" is designed to cope with corrupted journals and read them very well. It should be noted here that corruption of journal files as discovered by the --verify option, mostly are very trivial errors like a monotonic timestamps of the wrong length etc. Such errors only affects a single log entry, and often even that log entry can be read without problems. Despite using journald since it became standard on Fedora (F15) I have never experienced any data loss nor having journal files that I was unable to read.

Come on, syslog(3) wasn't even signal-safe and designed as dog slow too. It was crap. People have been working ever since to make syslog thread safe (still interim), etc.

That syslogd can't log until rootfs is started and most of the machine is up, that it is dog slow and silently drops messages under load, that there is no common API for accessing the content of the data, that the data is basically free form, deviating from machine to machine (an industry exist dedicated to this problem alone) also making it impossible to make a distro agnostic GUI log viewer, that meta-data like monotonic timestamps are basically impossible to add without the log entries becoming unreadable long, that text logs simply don't scale (again, an industry exist to solve this problem, see Splunk etc.), that using regex becomes intolerable and unwieldy above a certain point, that in order to grep for something in the log file, you basically already have to know the name of the function and what the problem is, this makes syslog log files basically impossible to use for newbies, so they have to resort to trawling through the logs like reading a book.

Really, flat file, un-indexed log files meant to be read by a human SA is something that belongs to another era. What the present and the future needs are binary, structured and indexed logs with a stable API that can be parsed by machines and that works even at mega-scales.

And syslog gets the messages late, and may not get them at all.

Well, these are know problems with syslogd implementations; they silently drop messages under even moderate load. It is their ancient design, so using pure syslog you will never know whether something was logged or dropped. With journald you will at least be informed if log messages are dropped. The latter should be a rare occurrence since journald is so much faster than syslogd.

journald goes to great lengths by using per app log rate limiting in order not saturate syslogd, but the inherent limitations in syslog means that this will always be an best effort. Still, it probably isn't worse than using syslogd alone.

There are several problems with solution: it isn't backwards compatible with syslog (journald strives for 100% backwards compatibility).

It doesn't solve the problems that combining meta-data with log entries in flat text files, makes the text logs hard to read for humans since the log lines become far too long and complex.

It doesn't really gives any advantage of pure binary data. Remember grep, sed, tee and every other standard Linux text tool works with systemd's journal through the standard Linux/Unix concept of piping, so there is nothing lost by going pure binary.