Please create an account to participate in the Slashdot moderation system


Forgot your password?
User Journal

Journal Journal: HOWTO: Thousands of open files on Linux 1

A project I worked on recently required splitting up more than ten million files, each with thousands of data points, into one file per type of data point and each with more than ten million entries. This seems straight-forward until you realize that by default, few kinds of systems allow having hundreds or thousands of files open simultaneously.

In my case I wrote a Python script to do the job and ended up with about 12,000 output files open at once, while attending to each of the more than ten million input files one at a time.

Fortunately, Debian GNU/Linux is a system that can be instructed to allow having that many open files, although you might get a different impression from the bash(1) manual page, which has the following to say for the "ulimit -n" command:

-n The maximum number of open file descriptors (most systems do not allow this value to be set)

The key here is that by "systems", the manual page really refers to "kinds of systems", not individual GNU/Linux systems. But even on GNU/Linux, it's a bit involved to actually get a non-root shell with an increased number of allowed open files. Here is one way to do it:

Let's begin with seeing how many files we are currently allowed (your number may vary, of course):

vinsci@debian $ ulimit -n

Ok, now increase it. First, become root for example using su(1), sudo(8) or sux(1). Use sux(1) if you need an X application to open a large number of files:

vinsci@debian $ su

As root, we are by default allowed the same number of simultaneously open files:

root@debian # ulimit -n

Now increase the number of allowed open files for this shell to 20,500 and check that it happened:

root@debian # ulimit -n 20500
root@debian # ulimit -n

This increased limit only applies to the current shell and any child processes it has. We're still root, of course, but to get a user shell with this limit we can now simply su(1) again, this time to the desired user account and the new shell will inherit the newly set limit:

root@debian # su - vinsci

... and there it is:

vinsci@debian $ ulimit -n

At this point you can then run the program(s) that requires more open files than normally allowed.

Keep in mind, though, that by exiting the current shell we become root again, so it would not be a good idea to hand over a shell set up this way to an untrusted user.

vinsci@debian $ exit
root@debian # exit

Afterwards, back in the original shell, nothing has changed so the old limit still applies:

vinsci@debian $ ulimit -n

Do you have a more elegant way?

In any problem, if you find yourself doing an infinite amount of work, the answer may be obtained by inspection.