Logfiles Made Interesting with glTail

Fudgie writes "My boss claimed it was pretty much impossible to create an entertaining way to visualize server traffic and events in a short time frame, so of course I had to prove him wrong. A weekend of neglecting my family produced a small ruby program which connects to your servers via SSH, grabs and parses data from Apaches access log and Ruby on Rails production log, and displays your traffic and statistics in real-time using a simple OpenGL interface (tested under Linux and Mac OS/X). It's a bit hard to explain over text, so please have a look at fudgie.org for an example movie, and more information."

Oh dear...

[DamonHD]

...I'm afraid that's the nearest I've seen to a simulated pissing contest ever! B^>

Rgds

Damon

Re:

[Nimey]

I think those are blobs of spooge. Excitement from having such a large request, possibly.

Oh great...

[GodlikeDoglike]

...we just made his log screen look like a bukkake flick.

Just took a look at the video

[Centurix]

And it looks like lots of things taking a wee. Once the site is slashdotted, it'll be a veritable golden shower...

Nice work though.

engineering management 101

[Anonymous Coward]

tell the engineer it can't be done

Re:

[H310iSe]

Just want to give props, very nice you made my morning. Now to convert this to a heads up display for my helmet and I'm 1 step closer to becoming the motorcycle hacker I always dreamed I could be. And 1 step closer to earning a darwin award...

Re:

[mattgreen]

It really is. Kudos to the original poster, I chuckled quite a bit.

Visitorville

[Anonymous Coward]

The most entertaining way I ever saw to view logs was Visitorville [visitorville.com]-its kind of like SimCity meets web logging.

Not "Fudgie", glTail

[gumpish]

It's pretty obvious that fudgie.org is just the name of the site and glTail is the name of the program.

Re:

[JamesRose]

On the main page it says glTail, and when you click the link to read it with comments it says Fudgie, so actually, it seems both are there.

Re:

[Bert64]

Also, "slash" is also British slang for "urinate".

Re:

[Aladrin]

And American slang for 'fan-fiction with male-male relationships.' What's your point?

Re:

[nacturation]

Also, "slash" is also British slang for "urinate".

And backslash is what happens if you urinate onto a parabolic surface?
 

Re:

[LiquidCoooled]

Actually, I think backslash occurs when you try pissing into the wind.

Wow !

[cheros]

Obligatory jokes about 'taking the piss' aside, that is brilliant. It's the ultimate 'machine that does ping' (to name an old sketch) to keep management amused, but also provides real data. I bet that screen will go ballistic when you get Slashdotted (also a good way to visualise DDoS, maybe?).

I was about to say that it's a sort of etherape on steroids, but I've just realised your visualisation could benefit etherape instead (if you don't know etherape, look it up. No tools identifies a virus infection quicker).

Class, I'm impressed.

Re:

[bughunter]

I bet that screen will go ballistic when you get Slashdotted

Look closer. It already is ballistic.

Re:

[Poromenos1]

Man, capitalize names. I got all sorts of things in my mind when I read about your EtheRape program...

just a ploy to visualize the slashdot effect

[molo]

Notice in the movie that one of the sites being monitored is fudgie.org, which is what is linked to here. This looks like a ploy to visualize the slashdot effect. :) Wonder what that must look like. Might tax the renderer pretty hard. I guess that is one way to get load testing done!

-molo

Re:just a ploy to visualize the slashdot effect

[Fudgie]

Still running at 30 fps with ~25 requests / second.

Re:

[Fweeky]

I just ran it through 10,000 Apache requests. After a minute and a half or so it stopped spewing dots from most of the graphs other than the "Content" ones, which spewed for about 8 minutes. In all those logs (about 60 seconds of activity) took 6 minutes 22 seconds CPU time on a 1.66GHz Core Duo Mac Mini.

Most of that time seems to have been spent drawing dots at maximum speed spewing out of the "Content" lines; maybe they need to increase speed in response to higher request rates so it's not waiting for t

Re:

[Fudgie]

Not sure why it stopped for you, I've had it running throughout a slashdotting without any problems at all. Peaked at 3500 req/min and still spewed dots from all the correct places at 30 fps.

Re:

[Fweeky]

Well, each stream seems to have a maximum rate it can spew out dots; if you exceed that, they back up. If you can spew out 1,000 dots from each stream in a minute but you've got 10,000 to actually spew through it, it's going to take 10 minutes doing it.

Re:

[Fudgie]

You're correct, and I will be adressing this in the next version. It's currently limited to 1000/FPS per second.

User agents and OS

[Wabbit Wabbit]

This would be very cool indeed.

I guess we could download the source and do it ourselves!

I don't know why so many comments were hating on this tool. As a big fan of "visualization" (Tufte books, etc.) I find Fudgie easy to understand and useful. The possibilities here are amazing.

Kudos to you, Fudgie (er...that sounds kinda bad)

Re:just a ploy to visualize the slashdot effect

[Fudgie]

http://www.fudgie.org/slashdotted.jpg [fudgie.org] for how that looks.

That's no moon

[rjamestaylor]

Serious prostrate problems at Fundie.org, it appears... I'm looking forward to plugging this in to sysstat for some over-utilized servers I manage....

Re:

[foobsr]

Great work (How I love that I may contribute a positive remark ;)

CC.

Here's what it looks like when you're not ./-ed

[chipster]

http://chip.cuccio.us/gl_tail.png [cuccio.us]

Perhaps the parser doesn't like my Apache logs?

2437 frames in 5.000 seconds = 487.400 FPS
Elements[0], Activities[0]
2550 frames in 5.001 seconds = 509.898 FPS
Elements[0], Activities[0]
1182 frames in 5.002 seconds = 236.305 FPS
Elements[0], Activities[0]
987 frames in 5.001 seconds = 397.321 FPS
Elements[0], Activities[0]
2534 frames in 5.003 seconds = 506.496 FPS
Elements[0], Activities[0]
2506 frames in 5.000 seconds = 501.200 FPS
Elements[0], Activities[0]
2505 frames in 5.0

Re:

[chipster]

I figured it out.

My apache config has the "HostNameLookup" feature enabled for the logs.

The ruby script's apache log regex parser only allowed for IP's in the logs. I changed it from [\d.] to [a-z0-9.] (line 87).

Bingo.

PS: THis is a pretty neat script.

Re:

[Fudgie]

Ah. I turned that off in '00 and forgot all about it. Sorry. :-)

Re:

[chipster]

No worries! Ideally, hostname lookups introduce extra load and traffic anyway :) The a-z0-9 should capture IP's and hostnames.

Nice work.

Re:

[eclectro]

Wow, it's like slashdot hurls chunks.

Re:

[o2sd]

Wow, it's like slashdot hurls chunks.

I say hurl. If slashdot blows chunks and fudgie comes back, shes yours. If it spews and fudgie runs, it was never meant to be.

Re:

[nacturation]

Very nice. One suggestion: rather than have each side's dots fall off at the bottom of the opposite side, how about matching up serving requests with the originating referral so that the dots go to the corresponding spot on the right? Also, if you're not familiar with Flight Patterns [ucla.edu] it's along the same lines. Borrowing from that, it'd be quite interesting to show a 2D map arranged in a hub and spoke model with the center being the site(s) and the spokes representing the top 10 (or 20... configurable) re

Re:

[Fudgie]

Interesting idea. Shouldn't be too hard to try something like that, I already have some code in there doing something similar meant for incoming emails, uploads and other data going into the servers/sites. Try adding :type => 5 to the URL activities for an example. -- Erlend

doom

[rucs_hack]

didn't someone once do a version of doom that displayed network activity?

I recall seeing screenshots, but that was years ago.

Re:doom

[xappax]

perhaps you mean this: http://www.cs.unm.edu/~dlchao/flake/doom/ [unm.edu]

Re:

[rucs_hack]

ah yes, that was it, not for networks then.

I loved this line:

[Anonymous Coward]

I loved this line:

"Certain processes are vital to the computer's operation and should not be killed. For example, after I took the screenshot of myself being attacked by csh, csh was shot by friendly fire from behind, possibly by tcsh or xv, and my session was abruptly terminated."

Re:

[lahi]

I can see why that approach didn't take off. (The idea seems to be 8 years old.)

However, the line you quote is quite satisfying: csh certainly deserves to be shot. Of course, so do users of csh. This also applies to tcsh of course.

-Lasse

Re:

[Seumas]

My sick great-grandmother uses csh, you insensitive clod!

Re:

[lahi]

Pity. Perhaps if she stopped, she'd get well.

Thanks for calling me an insensitive clod, btw.

-Lasse

Re:

[aled]

It's a shame nobody did a newer version with Quake.

Oh, Sweeeetness!

[avirrey]

You gotta add an 'Asteroids' ship on the screen that lets you shoot down connections!

"Oh, look! Bob just logged on... let's get 'em!"

...

"IT support. How can I help you?"

"Hi, this is Bob..."

--
X's and O's for all my foes.

Re:

[NFN_NLN]

We're finally catching up to movies now... you know the cheesy and disconnected from reality sequence where some hackers enters a system by navigating a 3D maze... and the firewall is a monster you have to literally kill. The movie Masterminds comes to mind.

Re:

[quanticle]

And maybe after that you can add a tool to allow you to kill "rabbits" with "flu shots" ;-)

Wow

[its_me_ken_lai]

Man this is cool. Very cool.

Re:

[Volatar]

I highly agree, I am definatly going to check this out.

Re:

[symbolic]

Agreed. I saw something similar a few years ago, but this seems a bit more refined. I think there's actually a lot you can do when combining a graphics rendering engine with something like network activity. All it takes is a little creativity, a little time, and a boss who says it can't be done.

Compiz for syslogs - ohmygawd !

[udippel]

Luckily, I saw the movie before the meltdown of the server. It always pays to be on time. ;)

For those unlucky and late, actually, you missed a competition of peeing coloured snowflakes from the right versus doing the same from the left.
Only, the sources on the left are much better at aiming.
Plus, you have some 'Login ...' scrolling top to bottom; like the cast of a movie.

Heads up, Fudgie, it is truely the most amazing display of log files ever creeping across my eyes.
Keep the good work up, and please post again when you have something actually useful for the sysadmin.

I declare you 'King of Log Candy' !

Ob quote

[Provocateur]

All I see now is blonde, brunette, redhead.

Postfix?

[JShadow21]

I'd enjoy a postfix version

Re:

[Fudgie]

Shouldn't be too hard. I'll cook one up this evening.

Re:

[account_deleted]

Comment removed based on user account deletion

Re:

[Fudgie]

At work I show about 30 logfiles, divided across 10 different servers running at 50+ FPS on an old Centrino laptop with a GeForce 5xxx mobile.

Re:

[MarkRose]

I have it monitoring Apache. At around 1000 requests per minute, I get about 10 fps on my 1750 MHz Duron. It's CPU bound, not GPU bound.

Re:

[JShadow21]

Awesome, thank you sir

Re:

[Fudgie]

A basic Postfix parser cooked up and introduced in v0.02. Also includes a simple IIS parser. More refined parsing of postfix will come. :-)

Re:

[Fudgie]

Using the libopengl-ruby version provided has caused nothing but trouble, so I've changed to the gem version.

gem install ruby-opengl

GNU GPL

[wikinerd]

#!/usr/bin/env ruby # gl_tail.rb v0.01 - OpenGL visualization of your server traffic # Copyright 2007 Erlend Simonsen # # Licensed under the GPLv2

Hey, this is not the correct way to apply the GNU GPL licence. I don't know whether you had very little time available or just don't care, but the correct way is to explain exactly what licence (full title) the program is under and enable the user to find the licence (provide a copy of it and explain that the author of the licence is FSF, giving their address). We nerds of course understand completely what you mean, but other people may have no idea what you are talking about. To learn how to apply GPL

Re:

[makomk]

I wonder if the author has spent time doing Linux kernel development? Variants of that statement are quite common on files in it, for some reason.

Re:

[kamochan]

No, you are not the only one...

Sorry, but the boss won this bet

[nurb432]

its still NOT entertaining.. Its more bizzare then anything else.

Re:

[aftk2]

I'm afraid I'd have to agree. The first thing I thought was: "Hmm..how about trying to make logfiles more readable/understandable instead?" I'm impressed by the technical acuity though.

Re:

[Fudgie]

It's both harder for me to track a scrolling display of text moving in erratic bursts, and processing the information in each line than it is to take a quick glance at a screen and see if there are many small dots or few large ones.

syslog, not ssh+tail -f

[allenw]

Why use ssh + tail -f when one can send the output to a centralized syslog server? There isn't any need to setup an account, keys, etc. when you can have the individual servers consolidate the data for you.

seconded

[Cheesey]

Remote syslog also means that your servers are more secure: (a) because it is harder for crackers to falsify remote logs as they need to compromise two machines, not just one; and (b) because your visualisation program doesn't need access to SSH keys for all of the machines it monitors, so a compromise on the visualisation computer doesn't automatically mean that all of the servers can also be compromised. However, you could presumably adapt this tool to use syslog quite easily.

Re:

[piranha(jpl)]

I'm sorry; I know your comment is old, but: no. No, no. No no no no no.

syslog is insecure; messages are unauthenticated. Don't believe me? Use the logger(1) utility to forge a message from any daemon on your system, as an unprivileged user. Send a UDP packet to an open syslog daemon to forge a message to look as if it came from any daemon on the originating host. Forge that UDP packet as if it came from any system in the world; there's no two-way handshake to verify the path to the sender is legitim

Re:

[MarkRose]

You could, like me, be using a shared host where you have access to the server logs, but not to the server configuration files. This is a fantastic way to monitor performance remotely.

Re:

[allenw]

That is a very good point. I'm used to dealing with scales beyond a single node ;) where you have access to such things.

In any case, I'm considering borrowing the idea and using it to 'watch' blocks on HDFS [apache.org]. I think it would be interesting to have a visual of blocks/files getting read/written/replicated. It might show patterns that we're otherwise not seeing.

Running glTail on Windows

[Mazin07]

If you want to run glTail on Windows:

1. Use the One-click Ruby installer from rubyforge (not Cygwin ruby)
2. Make sure to `gem install net-ssh`
3. Change "require 'glut'" to "require 'glut_prev'" to enable legacy GLUT ruby bindings

Took me a while to figure this out.

Re:

[0racle]

Thank you.

Re:

[0racle]

Actually, now it runs but I get nothing but a blank screen on both Windows and OS X.

Re:

[Fudgie]

Probably one of these:

• The parser you've chosen is unable to parse the log due to a wrong logfile format
• You've failed to enter the path to the logfile correctly
• net-ssh is failing to let you know about some problem with logging in

Enable session_options[:verbose] = :debug and see what that prints out.

Audialization

[Aleksej]

fastfinge> I used to have a program that would play a musical note every time someone hit a port. so for each port it would have a different note
fastfinge> i put it in the dmz
fastfinge> much musical entertainment
fastfinge> I should find the source for that thing again. i could change midi intruments depending on the type of packet.
fastfinge> or maybe create length and timbre data from the source IP?
2006-09-20

Booooring.

[Angstroem]

We did something similar like 10 years ago, hooking the log-file to the sound server where each port hat its individual sound and the frequency of connects directly related to the respective sound's volume.

Was rather interesting as you actually could *hear* all those Windows trojans and worms trying to dig their way into your (Linux) system.

Google called

[Anonymous Coward]

They heard about your cool project and want to subject you to a series of tedious interviews, ultimately not offering you a job because you didn't go to stanford.

Not impressed

[Anonymous Coward]

A weekend

It *really* shows that this was hacked together over a weekend. I've spent 15 minutes trying to get it to run, and all I see are Ruby warnings about about obsolete code, and failed dependencies. I've installed about a dozen packages to try to satisfy this beast's dependency hunger, but to no avail. Behold:

$ ls /usr/lib/ruby/1.8/net/ssh
connection lenient-host-key-verifier.rb service userauth
errors.rb null-host-key-verifier.rb session.rb util
host-key-verifier.rb proxy transp

Re:

[Fudgie]

Try and install the gem version of net-ssh or change the require_gem to plain old require so you use the packaged net-ssh instead? I've got net/ssh in

/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh

and

/var/lib/gems/1.8/gems/net-ssh-1.1.2/lib/net/ssh

depending on which Ubuntu version I'm running.

It's not hard, and quite a few have been able to get it running on Linux, OS X and Windows. FreeBSD is still a no-go.

Two words...

[John Whorfin]

Movie OS

Move of the slashdotting...

[Fudgie]

Is available at the site.

Re:Looks promising

[Fudgie]

Anything put into a logfile could be parsed and shown. I've tried with emails, shoutcast listeners and server logins, but they're not as interesting to show in the movie as I don't have the kind of traffic to make it useful.

Re:

[MarkRose]

This would be really handy for MySQL queries. My shared MySQL server runs 10 to 200 queries per second for me alone. Finding a good way to represent the data could be interesting.

Re:Looks promising

[DudeTheMath]

So...how many hours of unpaid overtime did your boss get out of you?

I like getting paid for my awesome work. Kudos, though.

Re:

[fmobus]

I believe this sort of tool is useful for realtime monitoring of net resources utilization. It can assist you giving graphic clues when something goes out of the usual parameters, like DDoS, slashdotments (sp?), router failure, etc. Depending on information being monitored and how it is displayed, it could also be used for long-term decision like buying more hardware or switching software because the current setup is not handling the load.

One nice, but more local example is the "duck" activity monitor (a

Re:

[merreborn]

A place I used to work is now trying to develop something like this: visualizations where you can tell trouble is brewing in a glance

If you just install any of the standard RRDTool frontends out there, e.g. cacti, or my personal favorite, munin (far easier to install/extend/use than cacti), and check them regularly, it's not hard to tell when something's wrong. Traffic and usage patterns are pretty consistent from week to week on the boxes I've administered. After a month of checking graphs in munin daily

Re:

[zero_offset]

Even more to the point, the article says the boss challenged him to create a visualization tool that was entertaining... this is sort of interesting, but entertaining? Or did Fudgie misspeak?

Re:

[slyborg]

Uh, wtf are you talking about?

And replace it with what?

[SanityInAnarchy]

RDP? VNC? RSH???

Re:

[MarkRose]

WTF? OMG? BBQ???

Re:

[DrSkwid]

Those mechanisms are for RPC, you can choose whatever method you like when your whole IP stack is SSL encrypted, like mine [bell-labs.com].

I'm constantly surprised at what people will plod along with!

Re:

[SanityInAnarchy]

you can choose whatever method you like when your whole IP stack is SSL encrypted, like mine.

That doesn't make it secure. SSH also has an authentication protocol, and it's per-user. If yours is per-IP-stack, you already lost -- both because we already have that (in the form of VPNs and ipsec) and because it's not secure (anyone who can connect to the server can authenticate).

So once again: What method of RPC would you use instead of SSH? Telnet? Last I checked, it won't do RSA authentication -- it's reall

Re:

[DrSkwid]

9p with centralised separately encrypted authentication, thank you for asking.

Re:

[Fudgie]

Sure it can. You'd just need to send the sudo-command line, and send the password if you got a password prompt in return. Or you could just let other users read the access log for a while, so see how it looks before you decide if this is something you'd like to try.

Re:Wait, what... they're not interesting?

[Fudgie]

A lot of my time at work is spent looking at logfiles from webservers, applications servers, and databases looking for things about to break down, but after I introduced this I just need to glance at a screen to instantly see if some server has stopped answering, is taking too long to answer, or is generating way more exceptions than normal. I also add an event (the login text bouncing down the screen in the movie) on each money generating activity, which always amazes marketing people when they walk by.

Re:

[mboverload]

Very cool.

This is the first time I have felt I needed to say anything on Slashdot in a while.

Well done, sir.

Re:

[zero_offset]

Almost half a decade? Holy shit, that's nearly one-twentieth of a century!

Re:

[Fudgie]

Grab the divx version of the movie, then.

Re:

[Fudgie]

Ah, finally someone else with this error. Been bugging me, and the person that reported it didn't mention running 64bit. That could be it. -- Erlend

Re:

[Fudgie]

Solution is to install the latest ruby-opengl from gems, and replace all

GLUT.BitmapCharacter(GLUT::BITMAP_8_BY_13,c)

with

GLUT.BitmapCharacterX(c)

The next version will have an auto-fallback to this function if the exception is raised.