The State of Linux Package Managers

I was pointed over to an editorial that is currently running on freshmeat. The author of the editorial takes issue with the current state of package managers for Linux and proposes a way to fix inadequacies. Here's a sample of the solution: "The solution to the problem seems to be to extend the autoconf approach to package systems. This requires providing the necessary tools, standards, and guidelines to software and/or package authors to enable a single source package to produce a wide variety of binary packages for the various platforms and packaging systems."

This already exists: OSD

The problem isn't developing a universal format, it's in getting everyone to support this format. I think a really good solution is already available in the OSD [marimba.com] standard. It's a standard developed by Marimba, Microsoft, Tivoli, and Novell which has been submitted to the W3C.

It's designed to be vendor neutral, and it's been written by firms that know a lot about installing software (in particular Marimba and Tivoli bear some focus).

The other nice thing is because it uses XML it's completely extensible.

Of course, the big problem is getting everyone to support it!

Well and good for the C-literate, but...

This approach may be fine for you and I; we're all comfortable with: ./configure; vi Makefile; make all; su # make install

Unfortunately, that isn't all that suitable for "naive lusers" who will react to this with a big Huh?!?

Rather than GNU Stow, I'd think the direction of BSD Ports [freebsd.org] would be suitable; that provides the merit of automating the process of setting up configuration info for lots of packages that hasn't yet been done with Stow. You may want to believe that

Dependencies could be added to Stow by someone without a lot of trouble.

I remain quite skeptical, as it has taken years for distributions like Red Hat, Slackware, and Debian to become richly functional.

Note that Ports, like Stow, uses nothing that anybody gets tangled into thinking is somehow "proprietary." (Not that RPM or DPKG actually use anything proprietary; it's mostly Slackware bigots, with emphasis on bigot, not on Slackware, that claim, dishonestly, that RPM/DPKG are somehow proprietary formats...)

But that misses the point.

Your proposal may be suitable for you and I, albeit marginally so, as I'd much rather that the administration of package installation for the 99% of packages where "default is fine" be dealt with by someone else; it is NOT, by any stretch of the imagination, suitable for making Linux deployable in other than highly UNIX literate environments.

Apt is *not* a package format.

Apt is a front end.

I don't see any realistic way around the consideration that Systems Integration Is Messy.

Whether we talk about DPKG, RPM, or BSD Ports, it's a given that the process of getting packages integrated into a particular distribution is a somewhat messy process. In all cases, there is some form of patch that gets applied to indicate precisely how they are to get installed.

It is getting increasingly common for Debian packagers ( e.g. - the human being that builds the patches required to integrate the package in with Debian) to have some degree of involvement with the "upstream" production of the original, authoritative source code tree.

When this happens, it is not unusual for there to be a ./debian subdirectory containing the "Debian-relevant" patches, and I've also seen ./SPECS directories with RPM .spec files. In cooperative development efforts, this is the point at which important cooperation takes place, as this means that there is some thought to systems integration in the original source code tree, which will make the job easier for everyone else.

It's not likely that the level of effort will actually diminish to zero, but if it becomes largely automated, and the human effort can be widely distributed, that makes the task not too herculean.

Re:Rpm works fine for me

I thought the same thing, untill I started using debians package manager. I still use RedHat on most of my machines, simply becouse debian can tend to stay in development forever, but once they come out with potato, I'm there..

It's the difference between a 10 speed and a Harley. Particularly the conflict managment, aka, you install package A. When you select it, it detects problems with package A, B, and C, which would also need to be upgraded due to conflicts, and gives you the ability to update them as well. And the package manager also handles updates as well, that puts RedHat's up2date and gnorpm using web search to shame..

Re:Well and good for the C-literate, but...

(try extracting a deb or rpm without the proper tools...

bluegreen:/var/cache/apt/archives> ar t apt_0.3.18_i386.deb
debian-binary
control.tar.gz
data.tar.gz

bluegreen:/var/cache/apt/archives> ar p apt_0.3.18_i386.deb control.tar.gz | tar ztv
drwxr-xr-x root/root 0 2000-02-13 05:01:14 ./
-rwxr-xr-x root/root 1361 2000-02-13 05:01:03 ./postinst
-rwxr-xr-x root/root 184 2000-02-13 05:01:03 ./prerm
-rwxr-xr-x root/root 534 2000-02-13 05:01:03 ./postrm
-rw-r--r-- root/root 29 2000-02-13 05:01:14 ./shlibs
-rw-r--r-- root/root 757 2000-02-13 05:01:14 ./control
-rw-r--r-- root/root 2707 2000-02-13 05:01:14 ./md5sums

bluegreen:/var/cache/apt/archives> ar p apt_0.3.18_i386.deb data.tar.gz | tar ztv
drwxr-xr-x root/root 0 2000-02-13 05:01:03 ./
drwxr-xr-x root/root 0 2000-02-13 05:00:59 ./usr/
drwxr-xr-x root/root 0 2000-02-13 05:01:02 ./usr/bin/
-rwxr-xr-x root/root 50776 2000-02-13 05:01:02 ./usr/bin/apt-cache
-rwxr-xr-x root/root 157576 2000-02-13 05:01:02 ./usr/bin/apt-cdrom
-rwxr-xr-x root/root 11148 2000-02-13 05:01:02 ./usr/bin/apt-config
-rwxr-xr-x root/root 129960 2000-02-13 05:01:02 ./usr/bin/apt-get
drwxr-xr-x root/root 0 2000-02-13 05:01:02 ./usr/lib/
drwxr-xr-x root/root 0 2000-02-13 05:00:58 ./usr/lib/apt/
drwxr-xr-x root/root 0 2000-02-13 05:01:02 ./usr/lib/apt/methods/
-rwxr-xr-x root/root 30288 2000-02-13 05:01:02 ./usr/lib/apt/methods/cdrom
-rwxr-xr-x root/root 17804 2000-02-13 05:01:02 ./usr/lib/apt/methods/copy
-rwxr-xr-x root/root 17108 2000-02-13 05:01:02 ./usr/lib/apt/methods/file
-rwxr-xr-x root/root 65508 2000-02-13 05:01:02 ./usr/lib/apt/methods/ftp
-rwxr-xr-x root/root 18652 2000-02-13 05:01:02 ./usr/lib/apt/methods/gzip
-rwxr-xr-x root/root 64632 2000-02-13 05:01:02 ./usr/lib/apt/methods/http
drwxr-xr-x root/root 0 2000-02-13 05:00:58 ./usr/lib/dpkg/
drwxr-xr-x root/root 0 2000-02-13 05:00:58 ./usr/lib/dpkg/methods/
drwxr-xr-x root/root 0 2000-02-13 05:00:59 ./usr/lib/dpkg/methods/apt/
.
.
.
(etc)

Daniel

Re:I can try...

Note that this isn't apt's fault; the problem is (perhaps) that the dependencies are incorrect. In particular, if libesd-alsa0 is a replacement for libesd0, and Conflicts: with it, it should also (..I think..) declare that it Provide:s libesd0. File a bug against libesd-alsa0 requesting that it provide libesd0 if my analysis is correct and you want this fixed in Potato.

Thanks,
Daniel

Re:autoconf, or what it could have been

There are an awful lot of one-off hacks that have no real internal consistancy.

Well, part of the problem is the lack of consistency between "UNIX-compatible" platforms. In particular, take a look at Motif; "Where's Motif" is a game somewhat like "Where's Waldo", except that it's not actually fun - OK, is it in /usr/dt, or /usr, or /usr/X11, or /usr/X11R N, or in some random location for third-party packages (although the "I installed package XXX in some random place" problem is generally handled in autoconf with a --with-XXX= YYY option)?

Note the quote at the beginning of the autoconf Info file:

A physicist, an engineer, and a computer scientist were discussing the nature of God. Surely a Physicist, said the physicist, because early in the Creation, God made Light; and you know, Maxwell's equations, the dual nature of electro-magnetic waves, the relativist consequences... An Engineer!, said the engineer, because before making Light, God split the Chaos into Land and Water; it takes a hell of an engineer to handle that big amount of mud, and orderly separation of solids from liquids... The computer scientist shouted: And the Chaos, where do you think it was coming from, hmm?

---Anonymous

autoconf is trying to cope with the chaos.

A complete "capabilities" API for UNIX-like systems. In other words, a programmatic way (from the language of choice) to determine how the local system compares to a set of metrics like "do I have gcc" or "is this Red Hat 6.1 or later" or "what is the standard include directory list for C++".

"Is this Red Hat 6.1 or later" isn't a capability; presumably the package cares because RH 6.1 or later behave differently from some other systems - but the package presumably cares about some particular difference, and that'd be the capability you'd want to check.

The API would, of course, have to be independent of the questions you ask it, so that arbitrary questions can be answered, perhaps with "I don't know" as an answer; the set of questions a package might need to ask about the system on which it's being built/installed is open-ended, so you can't just come up with a fixed set of questions that suffice for all packages.

Given that, either it would have to be able to somehow deduce the answers to those questions without the cooperation of the system - which means, in effect, reimplementing autoconf - or it'd have to assume that the OS and the third-party packages installed atop it would supply those answers, which would require that the OS know about this mechanism and come with answers and that third-party packages know about this mechanism and use some other API to supply answers.

(This would also require that programmers using third-party package X for their software be able to find all the questions to which third-party package X supplies answers - and hope that they don't need to ask a question about that package to which the third party in question failed to supply an answer.)

A configuration language (preferably built on something a little more powerful and flexibility than m4) which can be used to generate headers, Makefiles and other pre-processed items by using the above API.

Perhaps something along those lines (although not necessarily using an API of that sort) will come out of the Software Carpentry competition [codesourcery.com]. (And, if so, it'll use Python, as per the rules of the competition.)

Realistically, your average project should not have to look like more than:

buildmode: gnome_coding_standard
require: c(ansi,longlong), gtk
build_lib: fizzle(fizzle.c)
build: fazzle(fizzle, fazzle.c)

Unfortunately, many projects aren't necessarily "average". Ethereal, for example, doesn't "require" libpcap, it just disables packet capture if it's not present (and it has to go through some pain to try to find it); it doesn't "require" UCD or CMU SNMP, it just disables fancy dissection of SNMP packets if it doesn't find either of them, and it attempts to work with either of them; and it doesn't "require" libz, it just disables reading compressed capture files if it doesn't find it, and it requires not just libz, but a version of libz with particular functions in it, in order to support reading compressed capture files (as it doesn't just read the capture file sequentially).

This is not about the "State of Package Managers"

The article is on source packages, not the state of package managers.

Bruce

Advantage: Windows

Much as you may not want to admit it, this is one area where Windows products literally kick the crud out of the various free os's (osii?)

Not that there aren't any number of post-installation problems that can cause nightmares for Windows users; but generally, the installation of new software tends to go extrememly smoothly. This really doesn't have as much to do with MS as it does with InstallShield being the default end-all-be-all of installer builders for WinTel software, though some of the installer support included in W2K looks exceptionally neat, and a year or two ahead of what's available on Linux.

Your average user, when faced with RPM, DEB, tarballs, and the like will look at you and wonder what kind of crack you were smoking to come up with all these different ways to do the same thing, when all they want to do is just get something on thei machine so they can do X...

.deb, Apt

We already have a very good packaging system. Want to install something and everything it depends on?

# apt-get install foo

Want to remove some software?

# apt-get remove foo

Want to hack the source to something?

$ apt-get source foo

Want to compile your own debian package from source you've just downloaded and/or tweaked?

$ debuild

And given the large number of packages available, I don't even bother checking whether the package I want exists first, 80% of the time it does.

Advantage (?) Windows

Not that there aren't any number of post-installation problems that can cause nightmares for Windows users; but generally, the installation of new software tends to go extrememly smoothly.

Not in my experience.

Windows

1. Download archive.
2. Figure out if it is an archive or a self-extracting archive with a fully installed program inside or an archive or a self-extracting archive with an installer inside, or simply an all-in-one installer/archive, or maybe one of those rare single-file executables not archived at all.
3. If needed, extract the above-mentioned archive until you find an installer to run.
4. Run the installer.
5. Read the welcome message.
6. Close all your other running programs.
7. Read the license agreement. Jump through whatever hoop is required to prove you agree to it.
8. Click "Advanced" or "Custom" because "Typical" never works.
9. Redirect the installer to the "Program Files" directory on the drive that actually has free space on it.
10. Watch the pretty progress bar.
11. Read the readme, release notes, etc., etc., it throws up without asking.
12. Reboot.
13. Wonder why Random Unrelated Application suddenly doesn't work anymore, until you realize that the first thing overwrote some important .DLL in the C:\WINDOWS\SYSTEM folder without asking.

Red Hat Linux

1. Download archive.
2. rpm -ivh foo.rpm

There is a key difference between perceived ease-of-use and actual ease-of-use. Just because the installer has a pretty GUI with lots of colorful icons and progress bars doesn't mean it is actually any better. Give me RPM any day.

Us "experts" like package managers, too

However, there is a point where the newbies must learn how to do stuff as well, and RPM type things really don't teach much except rpm -Uvh and rpm -e :)

While I agree, as someone who knows a lot more then how to type those commands, anything that makes my life as a system administrator easier is a Very Good Thing. If I can install a package in a single RPM command (as opposed to reading the INSTALL file, diddling with configure options, and doing three different make commands), I'll gladly take it.

Uninstall!

What Linux needs is a way to uninstall applications. I don't mind compiling and installing stuff, or using .deb or .rpm packages, but I want to know that when I get rid of stuff, it gets rid of stuff.

Currently, uninstall options aren't all that promising. If you installed with 'make install', then good luck. If you still have the source around, maybe you can read the makefile and find out what went where. If you installed with RPM or the Debian package manager, you still have application-created data lying around.

I think most people have had the experience of doing a 'ls -a' in their ~ for the first time in a while and finding megs of old config data. When I uninstall enlightenment, I want it to take all seven megs of it's config info with it. Same goes for gimp or KDE.

autoconf, or what it could have been

When I first saw autoconf, I'd already dealt with metaconfig a bit, and autoconf seemed to be promissing a bit more modular and maintainable strucutre. It also was (at the time) a lot less interactive, which was good for a software configuration system.

Now, I long for what might have been if metaconfig had taken off. autoconf just isn't what it was craked up to be. There are an awful lot of one-off hacks that have no real internal consistancy. I once made the mistake of asking someone how I locate the Motif libraries in autoconf. I got several answers from "it should be where X is" to "you'll have to write your own command-line arguments, try doing something like what EMACS does". Granted, Motif is not at the heart of free software coding, but it seemed odd that a) such a popular library was not easy to locate and b) there was no standard way to say "search in these directories or as directed by these environment variables/command-line args for this library containting these headers and these functions". Many pieces of this exist, but none of it's coherent or complete.

I'd love to see two things:

1. A complete "capabilities" API for UNIX-like systems. In other words, a programmatic way (from the language of choice) to determine how the local system compares to a set of metrics like "do I have gcc" or "is this Red Hat 6.1 or later" or "what is the standard include directory list for C++".
   
2. A configuration language (preferably built on something a little more powerful and flexibility than m4) which can be used to generate headers, Makefiles and other pre-processed items by using the above API.

If someone were to ask my opinion, it should probably be based on one of the popular scripting languages (e.g. Perl, Python, Scheme, etc).

Realistically, your average project should not have to look like more than:

buildmode: gnome_coding_standard
require: c(ansi,longlong), gtk
build_lib: fizzle(fizzle.c)
build: fazzle(fizzle, fazzle.c)

That would indeed be sweet.

InstallShield type X utility is needed.

Now, for Linux to go mainstream, it is going to need some type of InstallShield type utility under X to do package management. I don't think this would be a very hard thing to do well, but this has not really been done yet because most Linux users do not need it. Most of us are happy compiling our software, and some of us just straight prefer it.

The average computer user simply can't handle the command line, let alone compiling things or even extracting files from a tarball. If we want a Mainstream Linux Desktop, we'll need this type of install utility.

"You ever have that feeling where you're not sure if you're dreaming or awake?"

PkgMaker

PkgMaker [slc.ut.us] is a tool I've written that can build packages for Solaris, HP-UX, binary tars, and RedHat RPMs. It uses a very simple model and can be easily extended for other package managers.

In writing PkgMaker I came to the same basic conclusions as Jeff did: adding a small amount of packaging information to a project's source would go a long way towards making packages easier.

CPAN!

Personally, I think CPAN makes a great model for what an end-all be-all package manager should be. Anything that handles dependencies and downloads automatically would be nice, but CPAN works SO WELL...

freebsd's style?

after spending over 5 yrs with linux, I wanted to open my viewpoints to other pc unices. started looking at freebsd and found that the 'pkg' and 'ports' notion was quite nice. I'm told debian's pkg mgr is somewhat like this set of concepts.

to the best of my knowledge, I am not aware of any linux distro that has an entire source tree structure such that you can 'gen a system' entirely from source - and painlessly, too. I think linux could benefit from freebsd in some ways.

I like having the ability to get just the binaries (pkg) as well as having the binaries be gen'd from source ON MY SYSTEM. no possibility of version skewing here!

so since linux can't decide on a common pkg scheme, why not take a slightly more neutral approach and just adopt the freebsd pkg/ports system?

--

Re:No.

Right idea, wrong tool. See encap http://encap.cso.uiuc.edu. Stow and encap were developed completely independent of each other, but came up with the right idea. The difference is epkg, the current encap implementation, is far more featureful and far faster than stow. It's really a generation ahead.

Rpm works fine for me

I use RPM on a linuxppc system. The majority of my problems come when there a ppc rpm for the most recent version of, for instance, binutils. I'll do a make and make install and *boom* suddenly it overwrites the rpm's files. Oh, but wait, some are in /usr/local/bin, some are in /usr/bin. Oh, and the rpm still thinks it's installed. Oh, and how do I now upgrade the rpm, or remove it without deleting the new binutils?

Just a few comments (also, rpm -qpl should put a header, so I can do rpm -qpl * instead of for x in *.rpm; "rpm -qpl" "$x" > "$x.lst"; done)

Jezzie
ls: .sig: File not found.

This situation comes up every time I ...

... show off one of my linux systems to a Windows-literate friend. Not a complete newbie, but someone who is used to downloading shareware and freeware utilities for Windows. Invariably they ask what the equivalent of of self-extracting .exe file is.

Now you and I may be happy with a uuencoded shell script, or wading through the 31 flavors of rpms on rpmfind.net, but coming from the Windows it looks very alien. Thre is an undeniable niceness to grabbing a zipfile, unloading it into a temp directory, running the program for while, deciding whether to keep it, or to delete the directory.

No dependency-foo, no Gnu-make-foo, no glibc-foo. Just unpack it and go. No silly compile from scratch and hope you have the right kernel, libraries, compiler and support packages.

RPMs, DEBs, source distribution with autoconf all give the user a LOT of power and niceties. But it is still an order of magnitude more complex than InstallShield looks to the average user under Windows.

Just some thought for food,

No.

What needs done is much simpler. Currently popular packaging systems need dumped in favor of GNU Stow [gnu.org]. Then we don't need to change automake and autoconf at all, because they work as-is.

Dependencies could be added to Stow by someone without a lot of trouble.

For those who don't want to download and install it to figure out what it does (althoug you should! It makes life very easy if you do any source installs), GNU Stow takes "packages" that have been installed in the standard manner (things placed properly in bin, lib, man, etc.) in their own directories (such as /usr/local/stow/) and makes links to the parent directory's bin, lib, etc. You can tell by a simple ls -l what a file belongs to. Since the links in the directories aren't the "real" files, you can delete and restore them with minimal trouble (I challenge someone with a conventional system to rm -rf /lib and restore it, without rebooting). You can even maintain multiple simultaneous versions of packages. Autoconf already makes this easy to use, simply supply the --prefix= parameter to your configure scripts.

No silly proprietary formats, nor waiting for someone to come out with the latest package in your favorite format, no trying to convert foreign packages to your system. Everything you can find in a tarball is now pre-packaged and waiting for you to install...

Why I like /usr/ports

This solution is very similar, but a little different to the /usr/ports solution in BSD. It would be easier to build this autoconf idea on top of ports than on top of the existing package managers, because they're already very similar.

Brief intro for those unfamiliar with *BSD: To install "gimp" on FreeBSD you do this: "cd /usr/ports/graphics/gimp ; make install" and away it goes--it downloads gimp from wherever it needs to, notices that it depends on GTK so it downloads that, etc., and builds each thing it needs in a giant make script until the whole thing is installed on the machine.

The FreshMeat editorial makes it sound like this is a brand new cool idea--it's not, all of the *BSD's have worked this way for years. I really like it.

I would love to see Linux support something like this. The closest is Debian's apt, which has a mode for fetching and installing from source, but it's not as simple and direct as this /usr/ports solution.

Some comments on this way of doing things:

-- I *love* being able to browse through the filesystem to find out what packages I could possibly install. It's a very natural thing to do: if I want to browse graphically, I do so via netscape or some filemanger. Mostly, being a geek, I use "ls", "cd", and "locate" to find out what packages i might want to install.

-- It's less to learn. If you are already going to have to learn how to do "make install" in order to get packages installed outside of your package management system (you just HAVE to have the version released yesterday) then you have already learned what you need to know to install any other package.

-- It does support a binary packages system. Binary packages amount to doing the compile stage on someone else's server, the whole install process goes exaclty the same way except that ratehr than compiling the binaries, you fetch them.

-- It brings everyone closer to the source tree. It's natural to grow up from being a novice user, to being a bit of a code hacker. There the code is, in front of your face, begging you to look at it--many people say this will scare people off, but nothing *requires* you to look at the code; and it's incredibly tempting for the curious. I think this leads to more developers, and is the main reason why *BSD has been able to keep pace with Linux despite having many fewer users.

-- The filesystem is a concrete, easy to understand organization for the packages. I can visualize where things are and how they relate to one another. With other package managers, like RPM or DEB, the dependencies seem complicated and abstract. When there is a failure, I haven't got a clue what to do (well I do now, but I didn't used to). AT least with compiling when there is a failure I can kind of see that it is a file in this package that lives over here, and that is causing my problem. I may not know what to do, but I know where the problem "lives". This makes me a little more motivated ot try and fix it, possibly by trying to install that other package some different way or something. In theory deb is the same, but it just doesn't *feel* the same.

In my opinion the only package management approaches that anyone should seriously consider are the Debian approach (apt/dpkg) and the *BSD appraoch (ports, plus their package management tools that back it up). Both of these allow all kinds of fun stuff like upgrading your system live without rebooting; synchronizing on a daily basis with the most current version; and have intricate and strong concepts of dependencies between packages.

In theory, they are functionally equivalent--or close enough--but I prefer the filesystem based implementation that has source code at its heart. It not only seems more Unix-like to me, it seems more open.

The big counter-argument to all of this is that source is scary to average users, many of whom don't understand the filesystem at all. I figure this is no argument at all, because you can bury the compilation under a pretty GUI just as easily as any other dependency system. And if your user can't figure out a filesystem, they won't be installing stuff using *any* package manager: it'll be pre-installed, or nothing, for them.

Just my $0.02