Efficient HTML Organization and Distribution on Webservers?

rasjani asks: "I recently started working as sysadmin. First major thing i want to rearrange is page distribution. Currently we are using rsync over ssh to copy the stuff to production servers and no backlog of any kind. I would like to implement CVS (or the like) into this scheme, so that if the NOC notices that something is broken in the web, they can do a rollback from CVS if the webmasters or editors aren't around to fix the problem. So people, do you have any thoughts how to implement this? Has anyone done something similar and willing to share their experiences? What gotchas might I stumble upon? Should I still use rsync/ssh for file distribution and add the check-in for a few scripts or should I just make a cron job in production servers to poll CVS for updated material?"

GNOME uses CVS

[GenCuster]

I am not sure exactly how it works, but I know for a fact www.gnome.org and news.gnome.org are done from cvs.

Why not email the webmasters and get all the help you need.

Re:GNOME uses CVS

[Phork]

i think debian.org is also done in CVS, but im not sure, but from what i hacve seen, it appears to be a common practice in colaberative open source projects. and as for how it works, im guessing you set you www-root to the cvs repository, or have the webserver do a checkin every 5 minutes.

Re:GNOME uses CVS

[JabberWokky]

i think debian.org is also done in CVS

All the kde.org websites are done in open CVS - you can browse the setup and see how it's done. CVS as web repository is very very common all over the place.

On the client side, anybody know of any GUI CVS tools that Delphi coders and Photoshop on Mac Graphic Artists can use? Intelligent in their own fields, but leery of any CLI tools.

--
Evan

Re:GNOME uses CVS

[lynnroth]

Check out cvsgui.org [cvsgui.org]. They have a Win32 client as well as Mac and Linux.

I've used the Win32 client and it works great.

Re:GNOME uses CVS

[update()]

All the kde.org websites are done in open CVS - you can browse the setup and see how it's done.

Link [kde.org]

Setting up CVS over SSH

[coyote-san]

This might be more detailed than the information you need right now, but here goes.

To set up CVS over SSH, use the following environment variables:

CVS_RSH=/usr/bin/ssh
CVSROOT=:ext:user@cvsserver:/usr/local/cvsroot

and set up CVS to use RSA Authentication (/etc/ssh/sshd_config)

RhostsAuthentication no
RhostsRSAAuthentication yes
RSAAuthentication yes
PasswordAuthentication no

plus similar for SSH2. This requires that each user and each host have its public SSH key on the CVS server, in their home directory and /etc/ssh/ssh_known_hosts respectively. In practice, you might find it an acceptable tradeoff between security and convenience to turn off the RhostsRSAAuthentication flag.

Finally, there's some work on implementing SSL/TLS directly into the CVS server, to eliminate the need to provide local user accounts on the server. This should dramatically increase the security of the repositories since it allows them to be turned into closed systems without user shell access. In the most likely scenario, CVS will be able to function much like SSH - you can operate in anonymous mode, or you can require PKI authentication of either or both parties.

overkill

[mckwant]

We thought about doing something like this, but found that CVS was overkill for our purposes. We didn't get past the stage where we determined that we didn't actually NEED every old version of our files, just something backed up so we could fall back if we needed to.

So we set up a staging server to which the developers have access, and only the sysadmins have access to the functions to move over files to production. This allows us to keep development separate from production, in addition to providing two sets of backups (one prod, one dev).

The next stage of this is to set up a box with lots of HD space, so we can keep "hot" backups of the html back through the days. We figured out that the backups for the various web servers would come to something like 9GB/wk. The boxes we eventually went with are attached to NAS boxes with 240GB each (mirrored, so functionally 120GB, which lets us keep roughly 3 months of backups on a hard drive, as opposed to tape.

Note that you could easily set up a *nix box with 300GB of space relatively cheaply. The processor and memory requirements are going to be practically nil, so we figured that it would be about $1200 for the box. This is beige boxed, which my boss threw out since we couldn't get HW support, so our system cost significantly more.

When we need to restore older files, we can just load up the old tar.gz, copy the files as needed, and we're outta there. No worries about tape drive screwups, and so on.

Hope this helps.

DaemonNews Article

[LiNT_]

Check here:Using CVS to manage a website [daemonnews.org]

Pater...

[gmhowell]

Is that you?

Doesn't this belong on slashcode.com?

What You Need Is...

[corky6921]

...a content management system (CMS).

I work in the CMS group at a large tech company. (Key word: large.) We use ATG [atg.com] and Documentum [documentum.com] to form ours, but there are many others depending on your needs... Interwoven, CVS, etc.

Here are the major features you should have in this system:

• Easy management of content by non-technical people (yes, this wasn't one of your original goals, but if you're going to work on a content management system, do it right and enable forms-based publishing so not everyone publishing to your site has to know HTML/scripting.)
• Version control, which CVS covers. Look at Documentum for more of this type of stuff.
• Instant publishing so people can push content NOW and dated publishing so that people can push press releases out on Friday at 8AM without actually having to touch the system on Friday.
• A large database on the backend that can handle all of these features, and at least two frontend servers: one for staging and one for released content.

This may sound like major overkill, but trust me, it's not. Put it this way: if you implement a solution using CVS (command-line tools) and rsync, you've just created a barrier to entry for publishing on your site. You want the marketing people to be able to push their cute little Flash/PPT/PDF presentations out NOW without having to log into a command-line system, and you want those same marketing people to do that without having to know anything besides Flash/PPT/PDF. You want publishing on your site to be easy and straightforward so that you, the sysadmin, can focus on the backend stuff without having to deal with marketing whining that they can't seem to get their new PDF on the site.

Spend the extra money and go with a content management system from the companies that do this for a living, and then you can rest easy and do the things you really want to do in your job while letting the website content manage itself.

Use RPMs

[turnerjh]

Use RPMs. Store your doctree in CVS and have a script/makefile that will do exports and create a CVS image from it. Do the same with code so that static and dynamic code both get proper, versioned treatment. This makes it very easy to revert to known good configurations, etc. The downside is you have to have root access on the web server. I can speak from practice in saying this works quite well, especially if you have staging/QA environments to test on. Simply roll the RPM, test it, and if it passes, push the same RPM live.

The Simple Solution

[md17]

We have multiple instances (dev & prod) running on the same server using a NetworkAppliances [netapp.com] througn nfs. Two nice things about this are:
1) We can also mount the netapp on windows.
2) It automatically takes disk snapshots, which are very easy to access. (just cd .snapshot/hourly.0 or something) it keeps a few hourly,daily,weekly, etc.


I'm not trying to be a salesman, but we love our netapp.