Upgrading A Headless Server?

Paul Bristow writes "I've been using a Headless Linux server at home for years and years, but I always find myself putting off doing a distro upgrade. It always means digging out the old graphics card, keyboard and monitor, dusting them off and plugging them in, just so I can have a console to upgrade on. Are there any linux distributions that will allow remote upgrades? You know, log in as root viua ssh, webmin, whatever, run upgrade prog, remotely reboot, off we go. ;-) I know I'm not the only one running a headless server, and I *know* how stable it is, but occasionally RedHat (in my case) come out with a nice new feature that I might want to upgrade for. Also, with just one server it might just be possible, but how do you upgrade a headless server farm?"

Nik posted this relevant article back in March about the PC Weasel, which sounds like a good way to do this, at the cost of a small hardware investment per box. Are there better ones?

A KVM

[tf23]

Buy a KVM switch. I've got a Belkin Omni 4-port. Now when I need to, I switch over to the server. No more fuddling with cables try to reach around to the back of the server box and all that. It's already plugged into my current keyboard/mouse/monitor.

Try Debian

[compwizrd]

(insertyoureditorhere,soidon'tstartaflamewaraboute ditors) /etc/apt/sources.list, change to what version you want
apt-get update
apt-get upgrade

done.

you want debian :P

apt-get update && apt-get -u -y upgrade

[kapella]

The distro you're looking for is Debian. The APT tools provide automated upgrading and installing without downtime or rebooting.

My modified Slackware can do this.

[Skapare]

I haven't actually done it headless (as in no video card), yet, but I have completely re-installed without using the keyboard, mouse, video, or any serial port. The way I did this was the way I layed out the system in the first place. Partition 1 has a 63meg self-contained rescue/maintenance system. The remainder of the drive is partitioned for the primary OS. Scripts included in the rescue/maintenance system can do the partitioning, reboot to see the new partitions, format the new partitions, and load the new OS file tree from over the network.

Unfortunately, this isn't helpful for your existing Redhat system. But it shows some ideas for things that can be done.

Debian can do it out of the box

[peterthomas]

Debian will do this straight out of the box

One "apt-get dist-upgrade" and you're away!

Easy enough under *BSD

[Echo|Fox]

You can do it exactly like you said in your example in FreeBSD (I'm sure Open and Net are the same). Just SSH in, su to root, CVSup source, make world, build an updated kernel, and reboot. The last time I upgraded one of my webservers I did it (sorta) this way even though that machine does have a little mono monitor. I'm not aware of any Linux distro's that you can upgrade like this, but any BSD users with headless servers may be interested...

Serial console?

[Alan Shutko]

Seems like a null-modem and a laptop (or another box) acting as a temporary serial console is the trick. I could have sworn RH had support for upgrades over a serial console, but I can't seem to find any info. Anyone know more?

Reboot = no

[fliplap]

I'm not sure but i think what he's looking for is something that doesn't require a reboot, since most machines won't boot without a keyboard/monitor. I really think what you're looking for here is debian, granted you won't be able to upgrade the kernel without rebooting, you'll still have a quick and easy way to upgrade the entire distrobution(-kernel) without rebooting or attaching a keyboard/monitor. Also with debian there's no need to download all the packages yourself, apt does that for you: apt-get update apt-get upgrade

Re:Reboot = no

[compwizrd]

set video to none, in your bios. most of the time, it'll run without a video card in it then.

Debian & serial-console

[Rural]

Debian has great support for over-the-network upgrades, since it doesn't require (nor have...) many graphical upgrade or installation programs. For the paranoid, who want to see what happens to the machine after a reboot, the standard Serial Console support of the kernel and LILO are great. For a quick reference on setting these up, there's a file in the kernel DOCUMENTATION directory. BIOS changes are not possible without some special hardware, but everything else works. This is also a must if all of your expansion slots are used...

I have done this with Debian - but Why?

[percival]

Debian can do this, as others have said, and I have actually DONE this with Debian (slink to potato) over ssh. You must remember to have _at least_ two ssh sessions open as root, I kept three just in case. The sshd upgrade was what scared me, but it handed off just fine. Of course, I had other problems...

I guess you could also create a Kickstart disk for Redhat and have it automatically install the newer Redhat.

I also have to ask: Why do you want to upgrade your headless box? It works, right?

-Percival

Re:Easy enough under *BSD

[lw54]

This is one of those things where I wonder why Linux doesn't "get it". :-)

We routinely upgrade our remote headless FreeBSD boxen without any problems whatsoever.

fezbox

[pastymeat]

I don't know what distro you're using, but since there's a lot of debain-related info out there, there's also fezbox [fezbox.com], a tool that lets you use the "kickstart" option of redhat.

Essentially, once you've used fezbox to create a kickstart install script, all you have to do is put the cd and a floppy disk with the config files in the pc, reboot it, and it just gets done. Again, I don't know if a PC will boot with no video/keyboard/etc, but there might be some way to disable this in the BIOS. worth a try...

btw, cheap linux cdz from cheapbytes.com, linuxmall.com, in case ur like me and don't have broadband :-(

Re:Reboot = no

[scruffyMark]

Is that just with x86 hardware? I know Apple hardware will boot without a KB just fine, but I don't really know about any of the other processor families out there.

You can use RPM to do a LIVE upgrade ...

[BitMan]

Have done this several times myself.

Just run rpm -Fhv *.rpm in your newer distro's RPM directory (e.g., /mnt/cdrom/RedHat/RPMS) and it will only upgrade those componenets that are install.

Heck, sometimes you don't even need to reboot! (although expect a few library issues until you do)

-- Bryan "TheBS" Smith

Re:Reboot = no

[wesmills]

What you describe is exactly how my Linux (redhat) machine is set up. It has a video card, but no monitor and no keyboard (not even a switchbox) attached. The BIOS is set to "Halt on Critical Only (No KB)," and if yours doesn't have that setting, many BIOSes have a "Halt on No Errors" setting. It doesn't seem to care about video, and the only complaint I ever get from Linux are a couple messages in the bootup log saying "keyboard: Too many NACKs -- noisy kbd cable?"

---

Off with your head

[smatthew]

I've got the same situation at my place. My main linux server is hidden behind the TV so that it's Ricochet wireless modem can get a good signal at the window (I live in a basement).

As long as your computer can boot by itself, i.e. no booting problems and it brings up the network you're golden.

Your RPMs are easy to upgrade - just 'rpm -Uvh pkgname'. No need to reboot, but I do say damn the uptime and reboot if you've seriously just changed every piece of code in the box ;-}

Kernel can be a bit trickier. Telnet in, and patch it first thing (Oh wait, you've got a stock redhat kernel? You could rpm -Uvh it, but i've got a certain fondness for going to ftp.kernel.org and grabbing the newest stable kernel(2.2.16 I believe). Then you can just patch it and recompile whenever you want - no icky redhatness to deal with.) After it's patched - either login through an X terminal and 'make xconfig'. If you're using telnet 'make menuconfig'.

I'm hoping you know what to do after that, but if you don't - drop by the linux document project. They've got a mirror at www.linuxhq.com. There's a Kernel-Upgrading HOWTO which pretty much says everything i've said - and more

Re:Serial console?

[technos]

Redhat used to support serial terminals out-of-the-box. I know it still does on Sparc, because I just used it. Slap a nullmodem on the first serial port, set minicom for 9600 8N1. VT100 or ANSI, ANSI preferred. SuSe should support it as well. (Have seen it done, never actually done it however.)

Re:A KVM

[heliocentric]

I think you are missing the point - there is a lot to be said about a computer that has few devices in/attached to it. It sounds like this person is looking to maintain this stability and reduce the hassel.

Since you did bring it up and I'm sure there are others out there who are thinking the same thing I'll toss in my two cents on the monitor/keyboar/mouse switches: get something powered. For example the black box stuff we use at work is great and it can handle sun (read 13w3) and PC (read SVGA) interchangly on the same box (you can program the switch with what type is plugged into it and what it needs to output). I thought I'd get a similar effect when I got a monitor/keyboard/mouse switch that was not powered - it works fine on systems where I used a 14" monitor, but for every video card/17" or greater monitor combo I tried I got some signal degradation (read fuzzyness) most likely due to the length of signal cable (and signal drop at the box itself). I tried loading the cable with ferites as well. If anyone has suggestions with my problem speak away - but if this is the case that the orig. questioner posed I don't see this as a cheap/benefitial option since he seems quite happy with his lack of head, sorry, headless system.

Re:Try Debian

[fsck!]

i don't care how great debian's toolset is, there are still some things you just don't do without reliable console access. i run debian on all of my servers, and i still insist on driving to the site before making any major changes. i expect most people do the same.

please don't degrade debian by trolling about how great apt is.

Re:Try Debian

[Just Your Average Li]

doesn't have to be debian. With RPM systems, you can rpm -Fvh ftp://mirrorsite/redhat6.x and i believe there's an install dependencies thing. (But redhat changes the package names sometimes and splits packages so that doesn't always work exactly right)

Debian GNU/Linux

[rbf]

I'd go with (and do) Debian GNU/Linux. Besides the fact I'm addicted to it, it is the most stable and can be easily upgraded (minus kernel) without rebooting.

Here is what you will want to do once you've got Debian GNU/Linux installed:
edit /etc/apt/sources.list with your favorite editor to make it point to whichever release you want.
apt-get clean (to purge already downloaded and probably outdated files)
apt-get update (to update the packages list)
apt-get -f upgrade (to upgrade most packages, skipping things like major libs - the -f tells apt it should try to fix dependancies)
apt-get -f dist-upgrade (to do a full system-wide upgrade of all packages)
logout, login. done! (unless you upgraded the kernel, then you'll have to reboot).

Enjoy!

*BSD Console

[marius]

{Free,Net,Open}BSD all support serial port consoles during the boot phase, something that Linux has been lacking. Just plug in a serial cable to one of your favorite /dev/cuaa* ports and there you go.

Re:You can use RPM to do a LIVE upgrade ...

[Paul Bristow]

I'll give this a try. Thanks.

Re:I have done this with Debian - but Why?

[Paul Bristow]

Answer to your question: I now find myself trying to maintain several remote servers as well (different countries even) and some of them need newer versions of RedHat (think 810 chipset nightmares). Plus some of the new admin features in RH6.2 are desirable.

Re:Off with your head

[Paul Bristow]

I'll give this a try. Strangely enough, I'm completely happy with remote kernel tweaking. It's the distro upgrade that makes my brain hurt. And no, I don't have a stock RH kernel. Ever. :-)

Thanks for the suggestions

[Paul Bristow]

I'll try the rpm methods of upgrading the redhat distro. But, for the longer term, I'll take a look at debian (I know I left that test partition on one of these drives...). Then maybe I'll post a HOWTO.

Re:Off with your head

[pete-classic]

No need to reboot, but I do say damn the uptime and reboot if you've seriously just changed every piece of code in the box ;-}

Ahh, grasshopper. It is time to reveal the secret of uptime.

init 1
init 4