Brief Analysis On Reverse Engineering Software

An unnamed correspondent writes: " An article on PlanetIT.com discusses a court ruling that establishes the reverse-engineering of hardware and software as legal, under the "fair use" umbrella. What ramifications does this have in the industry? Can I reverse-engineer MS Word and write a word processor that can read and save .DOC files?" The article also asks the eternal burning question "Is the DMCA contradicting itself?" Though the court cases this piece deals with aren't new, the issues they deal with aren't going away, and it turns out that the Bleem and DeCSS cases may have more influence on other reverse engineering cases than anyone anticipated. Will sense chase out absurdity?

Re:Adobe's tight hold on PDF?

No. Adobe is an ethical company. Ever since the first version of Postscript (now is 3rd) all the documentation are public. You can download pdf files of Postscript Language Ref Manual and PDF Ref Manual and other good documentation for free (those books cost several dozen bucks if you buy the hardcopy version from Addison-Wesley, and they are exactly the print version of the pdfs). People have been generating postscript and pdf on-the-fly. You can output straight ps/pdf code (more involved but very flexible and powerful, especially .ps because it's not just a format but a full-blown lisp-like programming language). There are higher level free library such as pdflib with has C/C++/Java/Perl/Python binding. I have many half-a-page perl script that generate high quality graphical reports on the fly (I worked on genomics laboratory with robotics producing tons of data daily). The nice thing about PDF is that everybody can view them and always print true. It's also convenient to create hyperlinked PDF documents using pdflib.

What effect does this really have on software?

(Sorry for the repost formatting better)

Does this mean we can legally see BeOS work on Macs and Be has no real reason to bitch anymore about lawsuits from Apple?

What does this mean for open source stuff as well? Is it now legal to take the TNT drivers from windows and reverse engineer those and publish your results to make optimized drivers for other OSes (same goes with any driver that isn't already supported). I know this happens a lot already through things like bus sniffers, but now that we can just disassemble things for research, that could make a lot of things much much easier. Especially since those complex while loops that check certain hardware statuses will now become much easier to decode by looking at the source code instead of accesses.

What will this mean for internet game servers as well? Will things like the UO clients and server ports become entirely legal now so there is no grey area or possible lawsuits?

What about something like disassembling the various parts of QNX and rewriting it to make a free alternative to QNX that's improved and optimized for specific platforms. You could greatly improve things like diskaccess in it by removing the microkernel-ish features and all the message passing it does.

DeCSS Reverse Engineering? No proof

I urge everyone who thinks that DeCSS was reverse engineering to actually read materials such as the transcript of Johansen's testimony [eff.org]. There is simply no evidence that DeCSS was the product of legitimate reverse engineering. Not just once but twice anonymous information was contributed to crack the problem in a form that does not resemble what one would get from treating the system as a black box. Johansen testified: "Yes, I believe the CSS authentication had been posted anonymously in Assembler language on the Internet, and Derek Fawcus had picked that up and rewritten it in C language and posted it on his website." Note the word "Assembler". Johansen also testified that he was given further information from a complete stranger on IRC. On the Livid-dev mailing list on Saturday, October 02, 1999 Eric Smith had posted: "The specific issue WRT the CSS code is that the x86 code was apparently simply ripped out of a working commerical implementation (which was presumably copyrighted)" to which Derek Fawcus had replied "Well I guess it might have been, but I don't _know_ that." (Fawcus went on to explain how he had "worked to understand the algorithm underlying the x86 code.") Why the developers didn't run away as fast as they could once there were questions is something I cannot understand. Didn't anyone learn from previous examples such as Compaq's reverse engineering of the IBM PC BIOS? Compaq set up their reverse engineering effort so that at every stage they could prove the source of information using engineers whom they could assert did not have prior exposure to IBM IP.

We have forgotten the IBM PC

The IBM PC was reverse engineered. Barely within the boundary of legality. But legal, nonetheless. If this had not happened, would we now live in the so-called "information age"? Probaly not. Reverse engineering and mass-producing the IBM PC clone put real computing power in the hands of almost anyone who really wanted it. Why should we now make illicit the same proletarian and, dare I say, egalitarian mechanism that put us where we are today?

Re:What's the difference between bleem! and DeCSS?

Yes but there's a danger here, and that is that if someone can reverse-engineer the PlayStation, that means it's possible to determine how the machine works without shelling out big bucks to Sony. This means that their entire business model, which consists of losing money on the consoles while sticking it to developers in licensing fees just so they can find out what makes the damned thing tick, collapses and they'd be forced to develop a MORE secure, MORE proprietary console in order to keep going, or risk their console turning into an open commodity like the PC. Given the fact that the PlayStation 2 is their first step in their master plan to conquer the world of broadcast and online media, I don't think they're going to let that happen. So, Sony will continue to treat reverse engineering of their hardware as a serious crime, regardles of what the law actually says.

Re:MS Word format

One thing about the Word spec is that it depends on the "OLE Structured Stream format", which is an undocumented spec, but is of course built into Windows and is an extention to MacOS. That's where the embedded stuff like Spreadsheets and even some 'built-in' stuff like line art come in.

The thing to realize about Microsoft releasing the Word spec is that they very carefully wanted to give 3rd Party vendors enough information to create DOC files that Word could open, *but not* information to open any DOC file that Word created. So what you see is a subset of information that Word is committed to support.

And as another sidenote on the Word issue, I imagine that Microsoft themselves has a few employees dedicated full time to 'reverse-engineering' the Word format when they plan a new release. Even MS has had interoperability problems (for example, Word 97 before the service pack).
--

Re:God says it's ok

Or (I'll save the Brits the trouble and make this joke for them) God's not American.

Reverse Engineering and Ethics

This section ties back to the Slashdot Ethics [slashdot.org] discussion a few days back, etc.:

In general, there are two legitimate arguments for using reverse engineering: to integrate a system smoothly with other systems and to ensure that the system does not do damage to the environments in which it operates, says Frank Prince, senior analyst for infrastructure security and management at Forrester Research, a Cambridge, Mass., technology consulting firm.

On the flip side, companies often make another argument for using reverse engineering -- an illegitimate argument,Prince believes.

"They think if they can figure out how it does what it does, they may be able to make a better or cheaper or faster version and make some money without incurring all of the costs of the initial development," he says. "How you feel about any of these arguments depends on which side of them you stand on and what you stand to gain or lose. Once again, we end up in a battle over profiting from effort and taking responsibility for your creations."

As I mentioned before:

"The problem is that the common rules for ethics are flawed. There are weaknesses in the common rules for ethics because while they promote various virtues, they also promote weakness in the face of unethical behavior by others.

This is a problem, and opens a can of worms."

The chaos of the digital age leaves us grasping at straws. People are not following any specific set of rules beyond momentary personal convenience, which ultimately has lead to police states, burned out enviroments, fished-out oceans, etc etc etc.

In Reverse engineering, the same potential is there, but we are seeing it at another level, in another arena. It is a symptom of a larger situation.

Bottom Line, we need to get our shit together and work this out before it gets truely screwed up in the legal system (I know it already is screwed up, but it is not thoroughly entrenched yet)

Re: Doesn't really matter

Reverse engineering does not mean black box methods. Black box methods are one technique of reverse engineering. They have the advantage of looking a little better when lawsuits come around, and the disadvantage of being much harder.

Compaq used clean room techniques because of what it was they were doing. When writing low level code, and it has to be bug compatible with something else, you're going to recreate the exact code that was in the original, because there's only so many ways to do something. Now copyright isn't like patents in what it protects. Patents protect the idea, even if you rediscover it. Copyright only protect a specific implementation. For a patent, it doesn't matter how you came up with the idea. On the other hand, for copyright, it only matters how you got the idea. If I get an idea for a poem, and come up with something that's nearly identical to some obscure Robert Frost piece, the courts would start with the presumption that since it's so similar, it must be a copy, but if I could prove that I had never read the poem, nor anything that referred to the poem, or had any knowledge that the poem existed, then my version would be mine. Probably this would make it even more of a pain for the next person to spontaneously come up with this poem.

With a patent, on the other hand, if I have a blinding flash in my algebra class and write down some patented algorithm, it's still infringing. It doesn't matter that I never heard of it.

Coming back to DeCSS, if the algorithm is obtained by disassembling the assembly, and then that algorithm is published in a natural language description, then the same author does an implementation in C, it's probably okay. The C isn't going to directly lift anything from the assembly except some of the tables. Given the principle that data cannot be copyrighted (you can't copyright the fact that your study shows that 53% of tech workers want to kill their boss), you could argue that the tables are also not a creative work.

The only issue of anonymous information is whether any of that may have come from people who have signed NDAs. Trade secrets are like patents (covering ideas) but with copyright style rules for when they apply (if you rediscover it, someone's trade secret doesn't apply).

Reverse engineering in non-US jurisdictions

Two years ago an Australian court ruled reverse engineering to be lawful (Slashdot story, October 1999) [slashdot.org]. Other jurisdictions outside the US have given similar positive decisions.

MS Word format

I'd just like to point out that Microsoft have released the full-specs for Word .doc files and an open source .doc reader is available, it's called wvware [wvware.com].

Vested interests will ensure this doesn't happen

Due to the fact that so many large corporations have so much invested in their IP, and make so much from it, and the fact that these corporations are so important to the US economy, and hence to the political future of those who matter, it is certain that reverse engineering will remain either illegal or inconveniently legally difficult.

rr

Well, that clears that up, then.

I'm confused. Possibly so is the author of this article. He seems to imply that UCITA is a pending piece of federal legislation, rather than state legislation. As it is, UCITA appears to be dead and buried in most states (hooray!).

He draws a line between the Reimerdes and Connectix cases by quoting that Reimerdes "didn't have a right to the DVD". Did he steal it? More confusion.

Anyway, it seems the 9th Circuit gets overturned all the time, so I wouldn't get too hopeful about this being a positive sign.

Have you forgotten already?

"Can I reverse-engineer MS Word and write a word processor that can read and save .DOC files?"

No, because Auntie EULA forbids it, and she'll get uncle Bill and uncle Steve to watch over you if you even try.

Re:What's the difference between bleem! and DeCSS?

The actual Playstation consoles are - and always have been - sold at a loss.

That is "common knoweldge" yes, but I'm starting to question if that's true in this day and age. Sony is a massive company, they build a lot of things and obtain others in huge quantities.

To say that they're selling it at a loss is a huge leap of faith. I'd like to see proof of this.

Re:God says it's ok

Nope, copyright does...

Re:We have forgotten the IBM PC

How was the IBM PC reverse engineered?

IBM sold technical specs, bus pinouts, BIOS ROM assembler source code and other data regarding the first IBM PC, sold in 1980. This was widely available information. I had an IBM binder with all of these items until it was lost in a move a couple years ago.

IBM was following in the footsteps of Apple ][, which went so far as to publish the schematics of all of their Apple and Apple ][ computers. It was an OPEN architecture.

IBM knew that the only way to make the machine catch on was to get any hobbyist with a breadboard to make cool new cards to fit inside.

Now, Compaq rewrote the BIOS from scratch, taking only the interrupt table and register content "API" for compatibility. Since they had an open book with the original source code, that's not "reverse engineering," that's "re-engineering."

Franklin copied Apple's ROMs verbatim, and were toasted in court for copyright violations.

From a patent perspective...

Let's take a look at the telephone.

Before Mr. Edison made his new and improved telephone, there was an older version (which, I'm sure, had a patent on it). In order for him to make his own telephone, he either would have to work from scratch, get the details from the company/patent, or get a license. Patents exist, of course, to give the original creator money in compensation for his/her efforts. If a creator does not let others gain access to the technology, a monopoly is essentially created. Should patents be ignored under such circumstances (so we could have a better telephone for example)?

Whether or not this situation is historically accurate, is it right to do this? (unless there's some stupid patent on the whole concept which happens too much these days anyways)

Reverse engineering an issue for .NET

(Others may want to correct any huge mistakes I make in my analysis...)

My understanding is that all .NET-capable languages are compiled to MSIL ("Microsoft Intermediate Language") which is _not_ machine-specific, nor is it as low-level even as Java's bytecodes; it is quite human-readable (calls like System.Console.Print etc. are in the clear; variable and memory accessor functions however are done via registers) and readily reversible. This MSIL can then be just-in-time compiled to machine-specific code by the MSIL VM.

As of Visual Studio Beta 1, MS were non-committal on how they were going to handle reverse-engineering issues, since it's a whole heap easier to do that now than ever (well, ever since VB 2, anyway, which also wrote out its code in the clear).

Presumably they will have to come up with a strategy that lets the developer target x86 at build time, rather than MSIL...?

Cheers
Alastair

What is being done to protect our rights?

I see alot of Anti-DMCA and related things talk on Slashdot, but its not clear what the next step beyond talk is. What is being done to repeal the DMCA? Are there technology-savvy lawyers out fighting battles for us, and if not, are any reading this message?

Also, how enforceable has the DMCA been? There are lots of activities that could fall under its protection that have not yet been shut down. For example, those in the emulation world have just figured out how to break CPS2 encryption used in many modern Capcom arcade games. It seems that this would be illegal under the DMCA. How many times has it been put to use against reverse engineers / hackers?

Captain_Frisk

God says it's ok

Physicists, chemists and biologists have been reverse-engineering the world for centuries, and yet God didn't file a lawsuit. I take it as an endorsement of reverse-engineering by God himself.

What's the difference between bleem! and DeCSS?

From PlanetIT article:

"The Reimerdes case dealt with somebody who didn't have a right to the DVD but was cracking through it to get the code, whereas the Connectix case dealt with a situation where a company was legally entitled to be using the code and reverse-engineering it for purposes of interoperability."

Keith Kupferschmid, intellectual property counsel at the Software and Information Industry Association

I may be missing something here, but I don't see the distinction. PlayStation is a proprietary platform. PlayStation games were built to run on that platform. bleem! was written to allow people who had purchased a license to a PlayStation game to play it on some platform other than PlayStation.

DVD players are a proprietary platform (because of the "decryption" code they contain). DVD's are built (encoded) to be played on that platform. DeCSS was written to allow people who had purchased a license to a DVD movie to play it on some platform other than a commercial DVD player.

Am I missing something here?

--Kaos

Reverse Engineering -- A Unique Opportunity.

RE presents a unique opportunity: A facility for determining a natural expiration on copyright.

The duration of copyright is arbitrary. Some have suggested that the duration of copyright for software be shortened, but it would still be arbitrary.

Unlike music, literature, and other copyrighted works, software has a distinquishing property. It is possible to create a program that works exactly as the original without copying the original.

Thus, the time that it takes to reverse engineer a piece of software establishes a natural duration for the copyright on the original. This is not to say that we should revoke copyright on the original once a RE has occured. It simply says that RE renders the copyright on the original somewhat moot. If the RE product is distributed gratis, the money value of the original copyright is eliminated.

To a certain extent, this is already the way things are. So, if RE is legal (so long as it doesn't involve actual reuse of copyrighted code) I have no problem with it. Under such circumstances, the Free Software community takes on the job of establishing the natural duration of copyright in a free market.

Of course now I will probably hear from some people who don't believe in a natural right to IP; but I do, and so do a lot of other people. It is unlikely we will ever agree on that issue, but perhaps we can agree to RE as a standard for limiting copyright.

Reverse Engineering file formats

I reverse engineered quite a few MS file formats (see my out-of-print book Undocumented Windows File Formats) and never had any hassles from MS regarding the reverse engineering.

In fact, MS tried to hire me to provide them with the specs for one of their file formats. Apparently the author of the code never documented the file format. MS had released specs for it, but they were completely wrong.

After being told by several friends that MS was notorious for delaying payment with contractors, I asked for half the money up-front. They refused and I never did the work.

But I digress. I reverse engineered a number of file formats that were "proprietary" Microsoft files. If they're going to go after anyone for it, surely they would have gone after me since I was publishing them left and right in magazines and my book.

I've figured ever since then that MS must have known that the whole thing about reverse engineering in their licenses must be unenforceable.

You can also look at all the work Andrew Schulman and Matt Pietrek did reverse engineering Windows code and the PE file format and neither of them ever got hassled either, as far as I know.

Pete Davis