Netscape Dumps Critical File, Breaks RSS 0.9 Feeds

An anonymous reader writes "In the standard definition of RSS 0.91, there are a couple of lines referring to 'DOCTYPE' and referencing a 'dtd' spec hosted on Netscape's website. According to an article on DeviceForge.com quite a few RSS feeds around the web probably stopped working properly over the past few weeks because Netscape recently stopped hosting the critical rss-0.91.dtd file. Probably someone over at netscape.com simply thought he was cleaning up some insignificant cruft." Some explanation has been offered by a Netscape employee.

Ack

[Cygfrydd (957180)]

I would've seen this post sooner, but my RSS feed was broken... something about a 404?

Why would this break RSS readers?

[eurleif (613257)]

I don't see how this would break RSS readers. DTDs pretty much never get read except by validators. Normal SGML and XML parsers just treat the DTD URL as an opaque string, not as something that can be retrieved.

Re:Why would this break RSS readers?

[acroyear (5882)]

Actually the DTD is loaded up by pretty much every proper XML library even if validation is "off".

The DTD contains more than just the element definitions and hierarchy. Its also used to define entities (&...;) that are non-standard to XML but may be expected in the file. HTML has tons of pre-defined entities but XML only has the core 4. All others are defined in DTDs and loaded on the fly as part of the processing.

There are ways to turn it off at the lowest levels, but higher-level abstractions/libraries might not give access to that. For example, with JAXP + SAX you can turn off DTD loading, but Jakarta Commons Digester doesn't give a setting where you can trigger that, so Digester tries to load the dtd, and even with validation off you can't change that. My only recourse is to take the DTD lines out of the various config files. (Reason: My JBoss server is deployed in private networks where the server can't reach the internet).

Re:

[Ozan (176854)]

I always thought about having abstraction and redundancy introduced in a way that critical resources like the DTD get a real URN like urn:netscape.com:rss:0.91 and then parsers fetch the resource by looking up the URL in a special service that translates a URN to one or multiple URLs. With multiple ones, if the first one doesn't work another one can be tried.

Basicly this system would do to URNs and URLs what DNS does to domain names and IP numbers.

Re:

[acroyear (5882)]

well, as long as they're talking about an XML "2.0", maybe you should write to a committee member and submit the idea?

Re:

[acroyear (5882)]

which "hosts file"?

we have licensed customers with this (in fact, they're the problem - internally we don't see any problem with having outbound internet access for our servers and trust the firewalls we put up); i can't go telling their IT departments (who are sometimes completely inept) that we need to go mucking around with OS configuration files, especially in Windows.

Re:

[tomhudson (43916)]

True, but if all they're doing is grabbing a dtd from there, who will notice ? It's offline right now anyway ...

My.Netscape is undergoing changes!

We are currently working to launch an updated My.Netscape complete with all the functionality you've enjoyed in the past built upon an entirely new technical foundation. This will come with additional functionality and a much overdue face-lift. Unfortunately, such an endeavor requires us to take the current My.Netscape off line for the time being. We're sorry

Re:

[acroyear (5882)]

In general, you're right. The various standard languages built in XML do basically require and expect that you're able to look at files on the internet. XML wasn't designed for many of the private-network uses its being used for today, and neither were many of the tools that reference public DTDs.

However, with most parsers there is a way to set up a mapping where it can look on the local box for a file matching a specific URL. JBoss, Hibernate, and Struts all do that for DTD files they have in their resp

Re:

[KrisWithAK (32865)]

You are right. I wish I would have seen this article earlier so that I could have posted sooner -- and others to get to see the "solution"!

Ever since I started developing on a laptop during my commute, I discovered that XML-based programs like J2EE servers would simply stop working. I experienced the same thing at work where, by default, your desktop applications (namely Eclipse) do not have access to the internet, and the servers will never have access to the "Internet".

The proper thing to do is for your

Re:

[acroyear (5882)]

Caching it requires having access to get it in the first place, doesn't it? I can't ask them to open their system up to the 'net just to grab some file every time they restart their servers, can I?

See my comment elsewhere in this thread - there's some "magic" that Java can do to load a local file as a resource given an external URL but I've never seen a decent example of how to set that up for my own apps.

Re:

[Mithrandir (3459)]

What you need to implement is org.xml.sax.EntityResolver. There's several methods that need to be implemented that are the different ways the SAX parser could query for stuff. Basically it will give you the Public ID and/or System ID and ask you to return a stream to what that resolves to. Then, in your code, all you do is run a hashmap that maps a given ID to a local resource (eg file or database BLOB) and then do your own stream opening/processing from there. I attempted to post some example code but see

Re:

[TheThiefMaster (992038)]

A lot of rss readers can't parse a custom dtd, they assume that rss is pretty much fixed, and ignore the dtd line completely.

Re:Then they're broken!

[r00t (33219)]

Fetching the spec is idiotic.

First of all, that's a needless operation. It can take time; don't forget the DNS lookup and all.

Second of all, it's not as if you could handle any random DTD. Software doesn't work that way. (this is one of the reasons why XML itself is a mostly-lame idea) If the XML doesn't match expectations, you can't convert it to your own internal representation. You probably have a C struct that you need to fill in. Even in some wild interpreted language like perl, you just won't have any use for unexpected data structures and you damn well need the expected data structures.

Re:

[Bogtha (906264)]

That's just wrong, you should try software authored by folk who know what they're doing.

It's called a non-validating processor and it's totally compliant with the XML 1.0 specification [w3.org].

Re:

[Thuktun (221615)]

Of course they retrieve it - unless they already have a local or cached copy. How else would they be able to parse a document marked up using a custom DTD?

A DTD is used when the parser wants to validate the document at parse time. It isn't needed for parsing. Having a DTD present won't necessarily allow your XML application to interpret random XML you didn't expect.

Ugh

[SuperBanana (662181)]

According to an article on DeviceForge.com quite a few RSS feeds around the probably web stopped working properly over the past few weeks because Netscape recently stopped hosting the critical rss-0.91.dtd file.

STOP, Grammar time. Ooooh whoooaaa oh oh...

Probably someone over at netscape.com simply thought he was cleaning up some insignificant cruft."

Or Netscape got tired of people using their bandwidth. Regardless of the reasons: if you reference a file on someone's site, it's hardly their fault if they move/change/delete it, and it breaks your stuff.

Re:

[fatphil (181876)]

But DTD's were designed to be precisely that. Likewise class paths in java.
Unnecessary hard coding of something that's not necessarily non-ephemeral.
I never liked the idea, I'm glad to see that some of my worries are well-founded.

The point...

[Junta (36770)]

Is to have a common component shared among many documents without replication.

Class paths is java are the perfect example to say how it *should* work. Java CLASSPATHs in every application/installation I have seen are site-local, all paths accessible without going over the internet to another site to get classes.

To be similar, an RSS site should copy this DTD to their local server, or to a server with which they have a concrete understanding of the relationship. Either a commercial agreement with a peer or at least using a server from an organization who explicitly defines the purpose of hosting to be a common place to promote it as a standard.

Did netscape promise itself to be an organization sharing that DTD explicitly, or did site developers get in the practice because 'it just always was there'?

DTDs are different

[pikine (771084)]

A DTD (document type definition) is a file that describes how an SGML document is structured. In this case, the DTD that went missing defines RSS 0.91, which is used by Navigator 4 for "channel" subscription.

It is expected that DTDs are hotlinked. For example, if you ever look at html source of a web page, you would see:

<!DOCTYPE ...>

on the top, and the hotlink goes to somewhere on w3.org. That is because W3 is the authority body that defines the html.

Since Netscape is the authority body that defines RSS 0.91, it is a bit strange how they stopped hosting the definition.

In any case, the missing definition won't affect software that processes RSS feeds. It only affects software that checks whether a SGML document is structured properly according to that missing DTD.

The main interest to this article seems to be the speculation how a deprecated web 1.0 company could end up hiring a clueless webmaster who deletes important files without recognizing its importance.

Re:

[aiken_d (127097)]

Well, you're doing pretty good spotting grammar problems, but you're pretty ignorant when it comes to the content of the story.

RSS is a spec that was created and promoted by Netscape. For proper validation, this remote (Netscape-hosted) file is needed and was supplied by Netscape to facilitate implementation of RSS. Removing it breaks a standard that they promoted. That is, they encouraged people to use this file that they hosted as part of the RSS spec. To reiterate, they told people to use this file a

Obligatory Lamport quote on distibuted systems

[Programmer_Errant (1004370)]

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

SPOF?

[basketcase (114777)]

Or maybe some smart person at Netscape decided to teach some people a lesson about using a 3rd party as a single point of failure?

Doesn't AOL own Netscape?

[gravesb (967413)]

And if so, why would anyone rely on AOL to make something on the web work?

HAHA

[eMbry00s (952989)]

Suck that, Web 2.0!

Host it yourself!

[nascarguy27 (984493)]

This is the precise reason why I host everything myself including my own series of tubes, dubbed the Internets. I host not only every file that my site uses, but I also have a program that regularly crawls the entire Internet and compresses it onto my own distributed system. That way I can browse the Internet by myself without worrying if someone else's system will fail. Although I do need to replace systems every now and then. But that's not a problem, b/c the distributed system has 3-5 copies of the Internet, each copy in a different place. Wait, isn't their some other company that does that? I can't quite place the name.

Seriously though, relying on some other system so your site will work is a recipe for disaster. It's similar to relying on someone to take you to work everyday. After a while, you get used to that fact that someone else is driving you that you don't even think about it. Then your driver gets deleted somehow. And you're stuck with no way to work.

Re:

[ScrewMaster (602015)]

Then your driver gets deleted somehow. And you're stuck with no way to work.

It's even worse if your driver gets somehow deleted while you're on the road. Happens to my XP box all the time.

This is a major problem,

[Morky (577776)]

This could seriously affect both of the guys using Netscape.

Netscape Says No RSS 0.91 For You

[gastropod (105661)]

From April 2001, "Netscape removed the RSS 0.91 DTD from their website. This means that all RSS feeds which depend on the RSS 0.91 (many, MANY news sites) cannot be used with a validating parser." [slashdot.org]

It seems as though it just took them 5+ years to follow up on the threat? Primary links are broken, but of course the lively /. discussion (which, um, I haven't read) remains.

Sorry about that

[christopherfinke (608750)]

my.netscape.com is undergoing a redesign, and when we announced the redesign [netscape.com] about 10 days ago, the DNS entry for my.netscape.com was changed to point to the new server where My Netscape will be living. This had the effect of making anything under the old my.netscape.com unavailable, since the only thing public on the new server is a splash page. (Nobody on the team was especially aware of this DTD file since all of the old Netscape employees were let go last year around the time Netscape.com was redeveloped; anybody working at Netscape now was hired since then.)

Now, why this file was living under my.netscape.com is anybody's guess, but we'll have it restored ASAP. I only wish that someone had brought it to our attention so that I didn't have to find out about it from Slashdot.

Christopher Finke
Netscape Developer

Re:Sorry about that

[mmurphy000 (556983)]

What's the official way to let you know about this sort of thing? I'm not trolling -- the better you can inform folk like us about how to interact with you, the more likely it is you'll get a response when you need it. For example, a quick scan of the Help and FAQ pages linked to off of the Netscape home page shows no mention of how to contact folk like you.

Re:Sorry about that

[christopherfinke (608750)]

What's the official way to let you know about this sort of thing?

You're correct that contact information appears to be MIA in the Netscape Help pages; I'll make sure to remedy that ASAP.

For something as serious as this, a user could have checked the profile of one of the Netscape Anchors or developers, where many of them list their screennames or websites, and subsequently, their e-mail addresses. (At least, I know I do [netscape.com].) Alternatively, any Netscape.com member could use Netscape sitemail to contact any of the staff members. Obviously, these are unacceptable for normal circumstances, but I wouldn't call this situation a normal circumstance.

Re:

[christopherfinke (608750)]

Sorry, I meant that it was unacceptable that a member would have to take all of those steps to actually contact someone, not that it was unacceptable for people to contact the emplpoyees.

You gotta be kidding me...

[Sleepy (4551)]

This blast is not squarely aimed at you, but you triggered it. Treat this in the spirit it is meant please (if I didn't give a crap at all, I wouldn't comment. Show this to your insulated bosses who don't know the first thing about community and transparency. Kudos to you BTW for showing initiative and acting on a Slashdot post. Honestly, I would not have given the "new Netscape" that much credit.).

>I only wish that someone had brought it to our attention so that I didn't have to find out about it from Slashdot.

This rankles.

Have you EVER tried contacting Netscape from the outside world? Seriously, I can count the number of times:

*) When my.netscape.com locked out Konqueror (1998?)
*) When my.netscape.com WITHDREW the ability to embed RSS feed on your "my" page -- actually this was PRE-RSS if I recall. Way before it was commonplace, you could embed Slashdot and Linux Today feeds. Then they killed it, presumably because they got enough users or some pointy haired reason. 1999.
*) When my.netscape.com adopted a shitty policy of DELETING all your mail if you don't login for 30 days. This did not seem to be publicised by an actual email. They don't seem to delete the mailbox itself, which violates RFCs I'm sure and basically insinuates the mailbox is active. I lost tons of mail from 1996-2003 (yeah yeah backups. Some things I didn't think I would need later). ?? Happened in 2003. Note that mailboxes were only 5MB still, so I quickly bailed for a 100MB Yahoo account.
*) The 2001 deletion of Netscape Developer. This lost a ton of Netscape copyrighted Javascript documentation.

Just TRY contacting Netscape from their page. The best you can do is use the WRONG FORM to submit to some contracter who won't forward it. Or, oh yeah - there's a 900 number for by the minute Support.

Back when it mattered, there was no 'Google Guy' for Netscape, who would act as an unofficial liason. After Jamie Z left, no one internally tried to fill the shoes of a community facing employee.

While I'll be eternally grateful for Netscape's open sourcing of their browser. What a different world it is now. Too bad that step is something the current management would never have allowed (that's the perception). I can't think of a more opaque Internet company than today's Netscape. I'm sure there are people who disagree or wish it could be changed (you're here..) but that and a $1 gets you a cup of black coffee. Show this to your boss - there are suggestions here :-)

Re:You gotta be kidding me...

[christopherfinke (608750)]

You make several good points that I want to respond to more fully, but I've got to run out, so I'll have to do that later. In the meantime, I'll put this out there: my e-mail address is [e-mail] [mailto]; my screenname and other contact information is available at my website [efinke.com]. Anyone who wishes to do so can contact me regarding issues with any of the Netscape websites or the Netscape browser; if I can't solve your problem, I can definitely get you connected with the right person.

Re:

[christopherfinke (608750)]

I can't speak for the pre-2006 Netscape, but as far as the current Netscape organization is concerned, I have to disagree with you regarding transparency.

* The former GM, Jason Calacanis [calacanis.com], blogged extensively about Netscape and encouraged other employees to do the same. He also called out other industry big-wigs [calacanis.com] for "not having time" to blog about their product.
* Many [fabienne.us] Netscape [neothoughts.com] Anchors [robotskirts.com], Navigators [themulife.com], and developers [efinke.com] have an active blog [sampletheweb.com] where they write about Netscape and/or are available to discuss it.
* There i

Re:Sorry about that

[martyb (196687)]

(Nobody on the team was especially aware of this DTD file since all of the old Netscape employees were let go last year around the time Netscape.com was redeveloped; anybody working at Netscape now was hired since then.)

Now, why this file was living under my.netscape.com is anybody's guess, but we'll have it restored ASAP. I only wish that someone had brought it to our attention so that I didn't have to find out about it from Slashdot.

Ummm, maybe I'm mising something here, but I would think that your web log would show a spike in 404 errors for this file, right? In my experience, it is helpful to assume that I do not know what I don't know, and to put procedures in place to help make those omissions stick out. So, a scan of your log files not only for this file, but for any others that also have a high number of 404's (especially from a multitude of referers) would be worth investigating.

BTW, best of luck on the redesign!

Re:

[VGPowerlord (621254)]

They removed every file, causing a spike in 404s for all of them.

Re:

[gjuk (940514)]

1) Isn't it great that the guy comes on and is open and helpful? There are plenty of organisations could learn from this

2) I found it amusing that the /. summary states "quite a few RSS feeds around the web probably stopped working properly" - what; so perhaps none stopped working? Would be great to see a list of ones which actually did.... anyont?

Re:Sorry about that

[dubl-u (51156)]

Hi, Christopher. First off, full marks for stepping up and explaining things honestly. You have done more good for Netscape than a dozen PR people. I'm sure you'll take a lot of crap from my fellow Slashdotters, but don't let it throw you. Listen to and acknowledge their legitimate complaints and you'll do fine.

I only wish that someone had brought it to our attention so that I didn't have to find out about it from Slashdot.

If you are looking to learn a lesson from this, how about this one: URLs are forever!

Whenever I make a change to a live server, my biggest concern is to not break existing usage. If I ever change an URL, I make sure to redirect old usage to new usage that's just as good. And if I'm ever not sure something is used, I generally look back at least three months in the logs. Especially if you've inherited a pile full of mystery, good analytical tools for your server logs are vital. Trying to run even a modestly-sized site without them is like running a large store without tracking your inventory: your life will become a series of unfortunate surprises.

Re:Sorry about that

[christopherfinke (608750)]

URLs are forever!

Indeed, words to live by. I wouldn't pin this mistake on one person not checking the right logfile though; in a company as large as AOL, when an entire 150-person workforce is laid off and a new (much smaller) team is brought in to manage the old properties, things sometimes get lost in the shuffle. The entire my.netscape.com service happened to be one of those things. I'm sure that this incident will act as a reminder to never let this type of thing happen again.

And BTW, it appears that the DTD file will be restored early tomorrow morning at the latest.

Huh?

[Digital Dharma (673185)]

Netscape? What's a Netscape?

Seriously bad programming

[owlstead (636356)]

If I would create a reader that was dependent on version 0.91 of the distribution, it sure as hell would include the DTD in local storage. It makes no sense to create a reader that can also use, say, version 0.92 since you would not know what had changed (and there is no such thing as inheritence between versions of a DTD afaik). Actually, as other readers noted, it would be terribly stupid to make your web-server or client rely on a third party computer for which you cannot guarantee the uptime.

These URL's are mainly there for their Uniqueness, not so much as for a place of quaranteed storage. Of course, they are also a nice place to look for the actual definition, but after that you would need a local repository. This is the first thing an XML library should support, and the first thing a moderately intelligent programmer should look at. I get *very* annoyed if this kind of basic rules are ignored. And I've even seen them ignored by people pointing to the XML digital signature definitions, where security and reliability should be the first requirements in the design.

Also, what would happen if w3c.org or netscape.com go the way of the Gopher? If they go bust? It's a quickly changing world out there.

Re:

[voidptr (609)]

The DTD is spelled out in the incomming feed file though. Even if I had a local copy in my application, the RSS feed I just grabbed from Joe's Random Blog probably references the netscape copy, and there historically hasn't been a good mechanism to redirect that to a local DTD store.

The URLs there were initially intended to be used as unique namespaces since each domain had someone that could guarantee uniqueness inside that domain, not really as actual retrievable URLs, although they were usually valid and

Logs?

[hhawk (26580)]

not trying to be a troll here.. but.. one would think that that file would have been accessed quite often and that would have shown up in the logs...

If I was a new hire at some old company where everyone else had been let go, I'd at least check out the logs and see what is being used? and then if some file is being hit 1,000's of times a day.. maybe ask a few questions..

A rewording...

[Shaltenn (1031884)]

Once upon a midnight dreary, while I websurfed, weak and weary, Over many a strange and spurious bookmark of 'free news galore', While I clicked my fav'rite feed, suddenly there came a warning, And my heart was filled with mourning, mourning for my dear amour. "'Tis not possible," I muttered, "give me back my free news source!" Quoth the server, "404".

Re:

[tomhudson (43916)]

Welcome to the new Quantum Web Computing Experience. Blink again, and there's a 50/50 chance it'll be back to normal - or your computer will disappear.

Incomplete Quote

[multipart/mixed (163409)]

I believe the quote you're looking for is:

Damn you, Bill Gates! Damn you all to DRM!

Re:

[XMyth (266414)]

Because of abstraction (the lower level XML library is what would retrieve the DTD, but it'd only receive the raw XML data to be parsed/validated) the request for the DTD almost certainly doesn't contain the referrer. Even if it COULD contain the referrer, there's no real reason why it should IMHO.

Re:

[Schraegstrichpunkt (931443)]

Exactly. The problem is that you're supposed to have an XML catalog that lists the location of the DTDs on the local filesystem (at least, that's how 4suite works). If you don't set that up, a validating parser can either fail or fetch the DTD from the URL specified in the document (personally, I prefer that it just fail so that I can fix the problem, but it's not the default).

Re:

[DavidTC (10147)]

It's not any different, except the w3c is run by intelligent people, and Netscape, apparently, is not.

I've always thought the full paths were a bit stupid too, and they should have some sort of shortcut standard, one that says "Use w3c's HTML4.0 standard", and the web browser knows how to contruct a path to find w3c standards. That way, when "Use netscape's RSS0.91" standard stopped working, web browsers could have a trivial update, or their config could even be changed manually, to tell them where to find