Frater 219's Journal: Firewallin'

My workplace -- an internationally reputed research institution with about 1500 employees and 2500 Internet-connected computers (and a /16 network prefix) -- now has a firewall. Finally.

Oh, we had a firewall of sorts before. What we had was a default-allow packet filter, which we populated manually with rules blocking access from addresses which had portscanned us, and to machines we had discovered were insecure. See, a few years back, the head sysadmin at the time had asked to install a firewall -- but some of the scientists were concerned that such a thing would get in the way of innovative uses of the network. But he managed to get not a firewall, but a "filter" -- an early Netscreen firewall appliance configured in this default-allow mode.

Maintaining this system was very labor-intensive. Our intrusion detection system (IDS) was based on Snort and email -- meaning that every ten minutes, it would email us a chunk of logs. When we could, we'd keep the live logs in an xterm in the corner of the screen -- and put in firewall blocks against any remote node that portscanned, probed, or tried to Nimda us. Besides being a lot of work, this was also error-prone: we often found that we had accidentally blocked some legitimate traffic, since an automated set of FTP jobs can look a lot like a scan or a DoS.

And so it went for a few years. Then, last year, my boss (who is remarkably un-PHB-like for a guy who likes Windows XP) convinced the scientists' IT oversight group that we needed a real, default-deny firewall. The project was dumped in my lap as medium-range: not so time-critical that I should quit doing Linux systems support to implement it, but also in need of long-range planning and consensus gathering before we went forward with it. (He's trying to turn me into a manager. Really, he is.)

(I should note: Our IT department is very slow moving. We are not the sort of department that can push out a complete transformation of institutional computer use in a day -- or even a week. We like to think we are conscientious, methodical, and modestly refrain from shoving technologies on an unprepared user base, but the fact of the matter is that we are slow. The week I started working here, two and a half years ago, we started talking about replacing the aging and unreliable mail servers. We started building the new mail system a year later, and finished migrating users to it a year after that.)

Before we could put up a default-deny firewall, we had to establish clearly what services needed to be allowed through it. In our institution, anyone on staff can request an IP address and add a new computer to the network -- with whatever OS and services they want to run. It's not our job to tell them no -- it's our job to make their stuff work the way they want it. So I needed to extend this philosophy to the realm of firewall access rules. The result was a database-driven Web application which let people request firewall openings, and let us approve their requests and translate them into firewall rules understandable by our spiffy new Netscreen-500 firewall.

We gave our users a month to register their existing services. On this past Monday night, the network administrator and I switched our Internet link over to filter through the new firewall.

It's actually gone rather well. We had a few glitches -- our dial-up server is outside the firewall, but the RADIUS server it authenticates against is inside; we'd forgotten to give it a pass rule, and didn't notice until the next morning when we got the user complaints. A few people failed to register their services, or didn't realize that accepting raw X11 sessions from Norway involves allowing certain ports through the firewall.

And now I and the other network security team members find ourselves in a very different job. Instead of watching IDS logs like hawks, and scrambling to block the latest source of attacks, we're now building up information about what our previously inscrutable user base actually wants to expose on the network. With the new firewall, we now know for certain that no incoming SYN is going to a system whose operator doesn't want an incoming SYN. We have a valid list of exposed services, so we can run vulnerability scanners like Nessus with impunity.

And the worst complaint from the users? One paranoid is griping that we don't block pings.