Please don't confuse the term database with a RDBMS or similar. A database can be a simple text file with structured data like /etc/passwd.
http://en.wikipedia.org/wiki/F...
systemd's journal is by any definition a database file and since it contains logging info, I fail to see the problem in calling it a "log database".

Some use ACID RDBMS, some use non-ACID NoSQL databases, etc. It all depends on needs and to why they collect the logs in the first place.

You are of course wrong about this, Enterprise log analysers like Splunk uses simple files and indexes to store events etc. pretty much the same way that systemd does.

There is considerably overhead when using ACID RDBMS', so when things needs to be really fast you do stuff like systemd and Splunk does. Logstash (probably one of the more popular log-analysers) doesn't use a ACID compliant RDMBS as backend either (though it can if you want to).

The point is that the world of logging and databases have moved on considerably the last decade. The primary reason being more and more data that needs to be analysed (fast) for one reason or another (Business, real-time security etc.).

systemd's journal is a pretty significant upgrade to the otherwise rather fossilised world of Linux logging. Don't get me wrong, I have tremendous respect for the Rsyslog team, but they have been struggling for over a decade to solve just some of the problems systemd's journal now have solved.

And I have have never really seen any good argumentation against using binary, structured and indexed log files:

They can be read by all standard Linux text tools with piping.
There exist multiple independent readers for them.
The logs can be programmatically accessed through a myriad of languages.
They provide functionality that can't be matched with legacy syslog files.
There is no non-contrived scenario where they can't be read one way or another.
They can be exported in any format and have default export options for all relevant industry standards.
Unlike syslog output, they have a stable and documented API.

So there isn't any real downside to using binary journal log files, while there is considerably advantages.

So now you are going from saying that people do not use DB's for logging unless they are clueless; (Let me quote you on this:)

And yes, I have extensive experience with log analysis. You do not turn the logs into a database unless you are fully clueless.

..to that DB's can be a good thing?
And please remember that ACID doesn't give any protection against file corruption, nor does it prevent user space programs corrupting the logs by generating obviously impossible field values. But then again, neither does the journal nor plain text logs.

Again, for those who need to run a full DB for their logs, systemd's binary file format is great, because it allows the journal to be exported in industry standard formats like JSON. That way the remote log-sink database can receive and store rich meta data in a totally stable and structured way; changing hostnames, IP's, or even different wordings or even different and changing languages used by the daemon log output isn't a problem with the journal, since it is based on field values, not complex regex-ing of unstructured, undocumented, unstable, language specific words.

Oh, and tamper proof, cryptographically "sealed" logs too (FFS) if you want that.

If your remote log-sink solution isn't a full DB, you still gets all the benefits of receiving structured log entries with eg. full microsecond precision timestamps _and_ monotonic timestamps. It is trivial to convert eg. JSON output with defined field names to any other structured format, while converting and aggregating unstructured text files is a pain.

It is also great for those who only needs local logging; the journal has many of the advantages of a full DB, but without the complexity and overhead. Its append based file format is also much more robust against file level corruption than databases. Since the log files are structured and indexed with field values, they are easy to perform powerful, yet simple queries on.

How do you find all syslog entries with the priority level "error" generated by the previous boot only?
With the journal it is : "journalctl -b -1 -p err"

And how do you generate a full list of every executable, including their path?

Since you can "tab" trough the values in the journal this can easily be done: "journalctl -F _EXE "

And with the -x switch, the help database is activated, giving further explanation on what the log entry means, and gives direct links to upstream support, perhaps linking directly to a page that explains the error code etc:

Example:
# journalctl -b -x -u systemd-logind.service

mar 07 16:46:16 localhost systemd-logind[546]: New session 1 of user Peter H.S.
(log entry above, help database output below:)
-- Subject: A new session 1 has been created for user Peter H.S
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/m...
-- Documentation: http://www.freedesktop.org/wik...

There are seriously many people who just trawls through long log files (even using vi apparently), because it is hard to grep for anything unless you already know what to grep for, and because regex'es are well, regex'es; difficult to use, understand and to remember all the many variations. Basically, newbies can't read or filter Linux syslog log-files by other means than trawling through them.

They can't have a useful GUI either because it is impossible to make a distro-agnostic syslog gui. Again, a problem that systemd's journal solves.

The bottom line is, that the only virtue syslog files have, namely that they are human readable, is a serious hindrance for their use too: they can't add monotonic timestamps and micro-precision timestamps and other meta-data without being excessively difficult to read for humans.

With the journal you can give machine parsers exactly the structured log info they need and can benefit from, while still allowing easy readable log output: a simple cmd-line switch determines the format. So you get all the benefits of legacy text logs without their severe limitations. It is a win-win situation.

These days, using simple, unstructured text log files, is simply obsolete. The market and various industries have already decided on this issue. Logs are meant to analysed, and the sheer volume of them means they need to be structured, indexed and have rich logging info, just like systemd's journal. That way they can be aggregated and machine parsed with minimal effort.

I am not twisting your words. Let me quote you verbatim:

That does not make systemd better. It makes it worse as it is so complex is not had dependencies on tools. Of course this also means external contributors can easily be fended off, as nobody can afford the licenses on their private budget.

Isn't it straight up obvious that you claim that external contributors can be fended off because they can't afford licenses? Isn't this exactly what you are claiming?

Since this claim is totally untrue, isn't it then obvious that you are utterly confused about how Coverty and Jenkins work, and have no clue whatsoever on how systemd development is done?

Oh yes you were, let me quote you verbatim on this:

They aren't meant to be corrupted in this manner either, but apparently journald has a fsck option in an attempt to fix it. Crazy.

You thought that apparently journald had a fsck option, it doesn't, which just goes to show how little you know about systemd.

As a Sysadmin you should know how your logging system works and the limitations it has. This is just basic SA stuff.
In this case you should be able to discern about malformed log entries and actual file corruption of the logs. This is apparently something that is really hard for you to understand.

Yeah right, like you were the smart one; I am sure you are a hot shot Linux developer with a CS degree that has to fend off job offers every week from leading firms, because that pretty much describe the leading systemd developers, that includes among many, Greg KH, the kernel developer who maintains the stable Linux kernels etc. probably number two after Linus T. in the kernel group, or Poettering who has over a decades Linux developing experience etc.

I could go on, but the CV's of the systemd developers are really impressive, and many of them work for leading Linux distros like Red Hat, Canonical, Suse, Debian etc. ; if these Linux distros know nothing about logging, please write a peer reviewed paper that sets them all straight about this.

Good luck with your BSD adventure, be sure to tell them that BSD is all about choice, so that they should support several init-systems simultaneously and by the way, break up their source repos in many smaller independent groups or else they are "bloated" and "monolithic".

And don't cry when your BSD fork makes a systemd clone and throws away their old obsolete legacy script based init-system, because this is exactly what is going to happen, and yes, BSD's will get binary logging too, it is only a matter of time.
All these changes will be "forced down your throat" no matter how much you whine about.

BSD is for people who hate Linux, so you will feel right welcome there. It is just a matter of time before you will kowtow to the party-line about how GPL is bad because it doesn't allow BSD sponsors to close source the code etc.

Yes you are confused about several things, including licensing payment.
Your claim that people are discouraged from contributed because Coverty/Jenkins may discover that their patch have security issues or breaks the build is laughable; of course such patches should be rewritten, and the submitter would be glad they such problems was discovered. Asking the submitter to rewrite the patch, or that the developer with commit access fix it, is everyday work in FOSS land. It happens all the time and it discourages nobody; this is something the submitter learns from.
Really, should the alternative be to accept every broken patch just so not hurt the submitters feeling? I don't think anybody wants that.

Regarding your claim that the systemd developers don't play nice to others and suffer from "God complex" (talk about hyperboles), then this is plain wrong:

You can find lots of statements from people actually contributing, even small patches, that the systemd developers where really friendly and helpful. The proof is also in the numbers; there hundreds of such minor contributors to systemd, something that strongly indicate a good and including developer community. Compare that to the almost non-existing non-systemd developer eco systems.

Sure, the systemd developers sometimes says no to certain patches or features, but again, this isn't having a "God complex" but about project leadership and the technical know-how to reject things that are bad for the project. This happens in all FOSS projects.

Fedora - was working then broke again.

Ah, so you where trying to use zfs on Fedora, despite that this isn't supported. Problems may occur in such cases but the problems have nothing to do with systemd since systemd works fine with zfs.

But you really seems to be very confused since you erroneously talk about systemd contributors needing to pay licensing fees: "Of course this also means external contributors can easily be fended off, as nobody can afford the licenses on their private budget.

That is just totally misunderstanding how things works, so it really seems like you are very confused about even basic facts how systemd is developed.

Disagree or not; logging directly to a full DB is very common, which is why it is a selling point for every commercial syslog implementation I know of.

Regarding the indexing and structuring of log files, this is exactly what journald does. This dramatically reduces work needed to analyse them, even if you re-index them and convert them another structured file format like JSON.
Working with indexed and structured files with defined field values, is always better than working with unstructured, un-indexed text files.

systemd's journal log files will always have an advantage over the poorly defined syslogd text logs, no matter how you spin it.

Really, are you unaware how common it is to aggregate log files in databases? This is a major selling point for Rsyslog that it is able to do so.
In fact, Rainer started Rsyslog exactly in order to overcome the many deficiencies with syslog(3), including the severe limitations of unstructured text files.

systemd's journal is pretty much a stroke of genius in that it overcomes all the limitations of unstructured, unindexed text logs, while not being a full blown DB either (the journal files are basically appended text files with a different line delimiter and an index in front).

Log files keep on growing every year, because people log more and more, and systems are running more and more daemons. Analysing such logs means machine parsing, and structured and indexed log formats like the journal have a huge advantage for this kind of work.

Ah, so now facts and reality is propaganda. Take a look here
http://www.freedesktop.org/wik...
and here (about the file journal file format)
http://www.freedesktop.org/wik...

It is when the corruption means squat for the ability to actually read the logs.

For some reason you seem to think that "corruption" discovered by "journalctl" means the logs are unreadable, but as explained, they are often marked "corrupted" just because a single field value in a single log entry was discovered as impossible. This doesn't mean that the log file can't be read.

I don't care what you think, as long as you agree that the whole notion of not being able to use standard Linux text tools like grep together with systemd's journal is just plain wrong and a total non-issue.

There just aren't any good arguments against structured, indexed log files that can be programmatically accessed and has rich meta-data.

You misunderstand; there _isn't_ a "fsck" option for systemd journals and there won't be because that would be equivalent of "editing the log files".

You simply don't understand how journald works. Let me explain again: systemd marks certain log entries as "corrupted" if eg. they are plain wrong, that is the client is sending obviously wrong or malformed log entries. That way you avoid impossible field values that may otherwise have screwed the log watch or log analysing programs; think of it as discarding bad data. This is the most common "journal" corruption that people discover when using the journalctl "verify" option. It is usually totally harmless and usually limited to a single log entry. It happens all the time with syslogd text logs too, people just don't discover them because syslogd doesn't have log file integrity checks.

When journald discover such malformed entries, it log rotates as a simple preventive security measure.

Sure, the journald log files are sensitive to real file corruption too (but so is syslogd log files), so clean unmounts of file-systems is a good idea (it always is). But the whole issue is massively overblown by people who doesn't use systemd anyway.

I can only suggest that you start to read up on systemd and the journal. You seem utterly confused about even basic facts.

It would just be an optional KDE module; KDE already have a rather excellent GUI for configuring systemd (Kcmsystemd) that works as an optional control panel module.

You seem to be somewhat confused here:

Coverty is static code analysis that find potential stability and security problems. https://scan.coverity.com/
Think "Unix Lint".

Many FOSS projects like the Linux kernel and LibreOffice uses Coverty. It isn't a magic wand, but it plain works nevertheless. No systemd contributor needs a licence in order to contribute to systemd, even if they get commit access. Coverty can trivially be replaced or complemented by other static code analysers, or even be omitted; it is something helpful, not something systemd depend upon.

The Jenkins builder provides: "Continuous Integration with automated test execution", something that really improves code stability since it ensures that eg. a patch isn't committed if it breaks the build.

Again, no contributor needs any kinds of licence for this, and again, this is a nice to have thing, not something systemd depends upon.

You seem to be arguing against code discipline and the use of automated tools, just because systemd have them (while SysVinit doesn't). That is just bizarre.

Anyway, systemd's developer community is extremely vibrant and growing all the time; there are hundreds of people who have contributed already with new ones coming every month. So there doesn't seem to be anything that scares contributors away, unlike, sorry for putting the boot in, the tiny and deteriorating non-systemd developer eco-system.

The major point of systemd's binary logs are that they are both structured, indexed, and contain rich meta-data, something plain syslog logs isn't.

That again means that when you export your logs to a log-sink, you can do so in a format that fits directly into the log-sinks database like JSON format and preserve the rich meta-data like the "_MACHINE_ID" field, that uniquely are tying every log-entry to a certain machine. You can also be sure that that the program claiming to generate the entry actually is correct since journald provides kernel guarantee for this.

If you choose to keep the logs in journald's binary format on the log sink server, you gain the benefit of indexed fields when analysing data; a huge win when it comes to random access. So much faster than trawling through text logs (O(n) complexity and all that).

Every log entry can be traced back to a certain machine (not just hostname), many log fields have kernel guarantee for what they say, besides having integrity checking and even strong cryptographically "forward secure sealing" (FSS) security against tampering.
The journald logs are simply designed from the ground up to be read an analysed on other computers than the one they were generated on, either in their native format or as exported logs.

Since the journald can be accessed programmatically, it can aggregate and analyse logs across different languages and has strong immunity against changing wording of the daemons log output (it operates on fields, not words).

And since the journal logs are structured identically by a documented standard, they are trivial to aggregate, unlike the output of many different syslog implementations.

Since the journald binary logging file format is stable and fully documented, and can be accessed programmatically with language bindings, you don't need a specific binary like "journalctl" to read them. In fact, there are there is a Rsyslog module that allows it to directly read (and export with metadata) systemd journal's. There are also Python modules etc. that acts as journal readers etc.
In fact, it is quite possible to make a systemd journal reader that works on non-systemd platforms like MS-Windows or OSX.

The journald collate all logging on the Linux machine, that means everything can be trivially exported to a remote log-sink, including the kernel ring buffer and the binary utmp and wtmp log files that syslog doesn't know about, and it can include early boot and late shutdown log info because it can work in initramfs, something syslog can't.

How about journald being designed as signal-safe from the ground up (unlike syslog) and that it doesn't silently drop messages under load etc. etc.....

systemd's binary log format is simply a massive win for both the enterprise user and the Linux newbie and everything in between.

The arguments against it are based on contrived scenarios, like professional admins that doesn't have (systemd) boot medias and lack access to a pc, a usb stick and a internet connection so they can make one in 5 minutes.

All the standard Linux text tools like grep, tee, sed, sort etc. work with systemd's journal through the Linux/Unix concept of piping.

Sure, systemd's journal still doesn't have as many tool sets and projects around it as syslog. But that it is simply a matter of time.