Perl Domination in CGI Programming?

at0m asks: "Perl seems to be the language that most people use for CGI programming. But is there a good reason for it? Sure, it's easier to use Perl than a lower level language, but programs would be more efficient if C/C++ were used. Programmers don't sacrifice this efficiency when programming applications, so why is it the standard to use a higher level language for CGI, when one wouldn't use one for other programs? I've been using Perl for all my CGI projects, but for my next one I'm contemplating using C++ to make it more efficient. did I make the correct assumption?" Most CGI apps prefer development speed over the complexity inherent in C/C++, which is why Perl and other scripting languages are used. Your thoughts?

Perl is already heavyweight

Perl people take pains to point out that Perl is not a CGI language. It is a language. It is weak in areas such as GUI development, but for networking, database work, or many other things it is a good, solid application development language.

It is no longer, IMHO, the best web development language - if all you are doing is web development, PHP will probably serve you better. The reason people are using Perl, is because the same skill set can be applied in so many places. Since starting work at COLT (EU telco/ISP) I have used Perl for:

1. A few one liners to sort out a messy archive of documents left on a machine

2. Automated admin of Security Dynamics ACE server (a one time password system)

3. Writing an idiot-proof menu driven program for updating ipfilter rules on Solaris boxen

4. Writing an interface so that Veritas Netbackup logs are written to a database.

5. Writing a NOCOL client to alert if an Oracle database goes down for some reason.

Perl is simply a good language for doing these kinds of tasks and many others. Those familiar with Netcraft may be interested to know that their survey software, which polls 99.x% of every web server on the web _every month_ is written in Perl (Highly modified version of LWP). Not a lightweight app.

Goldman Sachs use Perl extensively to manage their mission critical databases.

Perl may be a good CGI language, but it is more intersting, and provides greater benefits when used as an application language.

Re:Why not C/++? overhead

While I imagine equally optimized code would find
this claim perfectly true, Perl gives you a lot
of optimization for free, optimization that's
hard to do in C. Things like hashing and
good string support are not inherent in C, and
many developers would just use C-strings as-is
and linear associative arrays, and in the case
of strings need to write quite complicated things
to do good text manipulation, things that very
often are slower than Perl's internal string
implementation (and often buggy too). So...

Yes, Perl has added compile time and interpretive
penalties, but often the better algorithms that
you end up writing in Perl will end up faster
than in C, and so there are many times Perl will
turn out much faster unless you're really taking
a lot of time to optimize your C.

Only somewhat

The -w line turns on optional run-time warnings. That is not a change in syntax, it is like adding a lot of debugging code all over the place. It does not change the syntax

Using strict does change the syntax - it turns off the really flexible "Guess What I Mean" features. Yes, Perl has a lot of those. Yes, they are dangerous. They are fine in short scripts, but not in long ones.

However when 2000 line C programs turn into 75 line Perl programs, maintainability is assisted. Overly verbose syntax, and overly low-level syntax, both are the cause of much grief. There is a trade-off, and for what I do Perl gets it right.

And yes, explore some odd corners. Looping over a /g match is not something that is widely known, but if you have to take parsing to the next level it is utterly invaluable!

Cheers,
Ben

Re:Why not C/++

Precisely. I used C/C++ for about eight years before giving in to Perl for certain things.

My advice (which with $1.50 will get you a cup of coffee)

• Use C/C++ when you need fine grained control of memory and/or hardware, or if you need to perform numerical computations. These features make it nice for writing operating systems, device drivers, games, etc. But these capabilities become liabilities when you do not really need them.
• Use Perl to manipulate text. After all, by its design, it excells in text manipulation - which what is happening in >99% of web applications. It has the added advantage of more rapid development, but I would still use it for web stuff instead of C/C++ even if it had to be compiled. Believe me, it is much nicer to use Perl's regular expressions capabilities than to do it in C/C++/lex/yacc (which I've used for many previous projects that needed sophisticated string handling - before I finally learned Perl).
• I have mixed feelings about Java. It's somewhere in the middle. You lose the fine control of memory & hardware that C/C++ has but don't gain much in the way of higher level language features, except for automatic garbage collection and (the claim of) greater portability. It's sort of a spayed/neutered C++, and Perl is probably more portable anyway. Java will probably suceed on the strength of its libraries.

Perl is king for a reason

Although many people point to the ease of writing Perl as the primary reason it is on top for CGI programming, there are many other reasons. I would imagine that topping this list is the fact that more people know and use Perl regularly than most other languages, since it is such a flexible and portable language; when people start writing CGI scripts, Perl is a natural for them.

Perl has tons of freely available libraries and modules that encapsulate almost every bit of functionality that you could ever want. Anything that takes more than a few lines of code has probably been done and encapsulated before. You just need to get it (CPAN [cpan.org]).

Perl is a very expressive language, more so than most other languages I've seen or used. Because of this, people get very attached to Perl and what it can do.

Perl has better support for regular expressions, parsing, and string manipulation than any other language or tool in existance. This becomes extrememly important when converting data on the fly, from text files, other formats (such as XML), from databases, or from any other thing you can think of. Grabbing raw data and parameters from the URLs and from POSTed scripts is easier in Perl than in most other languages, due to this support.

What people usually bring up as downside to using Perl as a CGI language are not specific to Perl, they're specific to CGI. "Perl is a huge executable," they say, "and it is expensive to run a Perl script." But most of the overhead comes from the webserver forking a process, not from Perl itself. Yes, Perl scripts can become bogged down with tons of modules and libraries, but so can any language that has the capability to use libraries. As far as being a huge executable, take a look at the binaries produced by some of the Microsoft Web solutions (ahem, VB, ahem) and we'll talk about huge executables. The Tcl binary is about the same size as the Perl binary, and many complex C programs end up being very large also, once you incorporate regex parsing routines, CGI libraries, and image processing libraries.

darren

Use the best tool for the job

Most of the work in a CGI script consists of reading in template HTML files, substituting a few variables in them, and writing out the results. More complex work may involve a few more sophisticated text substitutions, a bit of database access, and perhaps traversing a few internal data structures as well. In all these cases, the work is being done inside library functions rather than in your code: when you call printf() or look something up in a hashtable, the performance is controlled by the implementation of that library function, rather than any code that you wrote yourself.

As your calculations become more sophisticated, the balance shifts. If you are writing a CGI for generating fractal images or raytracing a 3d model, Perl would indeed be a stupid option (and in fact this is when you might even find C to be too highlevel, and prefer to start twiddling those bits around in asm). But if you profile something like the comments.pl that generated this page, I suspect you will find that only a tiny percentage of the time is being spent executing the loops and conditionals written by Mr. Malda, and the vast majority inside the I/O routines, string handling, hash lookups, and regexp functions, that were written (in C) by Mr. Wall :-)

Also, higher level languages decrease development speed and make things easier to tweak. Since websites tend to be in continual development, and new features are required on internet time, this is a major win. In conventional application software it may be a useful tradeoff to spend an extra week coding in order to double your execution speed, but if you know you are going to have to change this program a fortnight from now, that suddenly starts to look much less sensible :-)

Your real question

It seems that your real question pokes at something deeper. Mainly, why do web-folk use an interpreted language for their applications when it would be faster to use a compiled language like C.

I think i can shed some light on this. There are several reasons which I'll outline below.

• Rapid Development - A typical software project might last for months before hitting the shelves. Most web projects have to be spec'ed, built and online in a matter of weeks. Interpreted languages (like Perl and Python) make it easier to develop things in a hurry because you can make changes without re-compiling.
• The performance bottleneck is bandwidth, not performance. Usually, it's the speed of someone's modem, or the crowded internet backbones that slow down a web-page's performance. Using a faster language isn't going to help that, so typically web-folk go for the easiest solution. The easiest solution is to use an interpreted language. The reason is listed in my first bullet point.
• Much of CGI scripting consists of text-prodcessing. A high percentage of CGI programming can be summed by saying "Fetch. Parse. Print". Since Perl, Tcl/Tk, and Python have built-in, powerful pattern-matching and text-processing routines, these specific languages lend themselves nicely to web-app design. This might be less true as a whole than 2 years ago, but I know personally that I'm still doing an awful lot of $x=~s/blah/blahblahblah/g; in my web programs. Can't do that any easier than in perl/python/tcl.

In the end, it comes down to whether or not you'll get a noticable benefit from using something that runs faster. I think that for 99% of web-apps, you don't get enough of a speed increase by switching to C/C++ so people just don't even bother testing them out.

Perl is good, Perl is bad

I've had to maintain Perl-written CGI and whenever I have to do so, I usually wish that Larry Wall had been hit by a truck before he could have come up with the idea for Perl...

Not that I have anything against Larry, but his TMTOWTDI (There's More Than One Way To Do It) philosophy has made it way too easy for people to write really awful spaghetti code in Perl. Much more easily than in any other language. In Perl, precisely because of TMTOWTDI, you can have four scripts that do exactly the same thing and look completely different! ARGH!

And that is the "danger" of using Perl for CGI, and this is the situation I think the "web programming" industry is in - everyone and their grandmother can pick up a "Perl for Dummies" at the local bookstore and, without any concept of "proper programming practices" (i.e. don't write spaghetti code, try to comment at least what's not immediately obvious, stay structured so you can make changes easier) can start banging out CGI code and since Perl will take just about anything, nobody knows the crap that's running until it comes time for that burger-flipper to leave this job and move on to the next.

I've been given Perl CGIs to try to figure out why they're breaking that are 100's of K in size and 1000's of lines of Perl code and took 10's of megs of core and way too much time to execute. When they were written, the author literally cut-and-pasted from any source that looked vaguely like it would do something like what they wanted their script to do and just left all the extra cruft in and kept just appending to the existing code any time they modified the scripts with thousands of lines of code doing calulations the results of which were just being overwritten by later lines of code. Perl being Perl didn't care, but when it came time for someone else to figure out what the damned thing was doing - forget about it.

Yes, I occasionally write stuff in Perl, but I don't use any of the funky streamlined grammatical constructs that make the code horrendous to try to read through if you go back and have to maintain it, which also means that I don't usually gain a whole lot by writing my code in Perl except that I can be sloppy and it doesn't care, which IMHO is A Bad Thing. If possible, I will rewrite the thing in C or as a Java servlet, which just makes it much easier to maintain since there's ONLY ONE way to do it in those languages and you have to do it THAT WAY; when you see a statement in C or Java, you can figure out what it means and it can only mean one thing. Not like Perl... *shudder*

Personally I think Perl should be considered a "non-rewriteable" language; kind of the programming equivalent to a CD-R. Once the program is out there and has been running unmaintained for a while, don't even think about modifying it! If you need to do something else, write a new program because it'll probably be faster than trying to figure out what you did with the old program, and will almost certainly be faster than trying to find out what someone else has done in their Perl program.

In small shops where a piece of code might have to be banged out for something quick-and-dirty and never looked at again, do it with Perl. If you're going to try to write something that will need to be maintained and changed and worked on by many people, you either have to make sure those people are going to be VERY CAREFUL and VERY RESTRAINED in their programmin, or just don't even do it with Perl - it will quickly become unmaintainable.

-=-=-=-=-