MN Bill Would Require Use of Open Data Formats 176
Andy Updegrove writes "A bill has been introduced in Minnesota that would require all Executive branch agencies to 'use open standards in situations where the other requirements of a project do not make it technically impossible to do this.' The text of the bill is focused specifically on 'open data formats.' While the amendment does not refer to open source software, the definition of 'open standards' that it contains would be conducive to open source implementations of open standards. The fact that such a bill has been introduced is significant in a number of respects. First, the debate over open formats will now be ongoing in two U.S. states rather than one. Second, if the bill is successful, the Minnesota CIO will be required to enforce a law requiring the use of open formats, rather than be forced to justify his or her authority to do so. Third, the size of the market share that can be won (or lost) depending upon a vendor's compliance with open standards will increase. And finally, if two states successfully adopt and implement open data format policies, other states will be more inclined to follow."
I hope it passes (Score:5, Insightful)
Re:I hope it passes (Score:1)
Lock-in isn't the point (Score:5, Insightful)
If I buy some paper from the Bienfang company and write a report on it, that report will still exist and be readable possibly for hundreds of years after Bienfang goes out of business. If Microsoft stops making a word processor or (god forbid) goes out of business, the situation may be different.
"So what," you say, "just reverse engineer it." But what if, in the intervening years, Microsoft has successfully lobbied for laws that make that a criminal offense? We're talking about future-proofing data here; whether it's implausible is not really the point. The point is that using a closed format introduced risk.
Another, more likely scenario: Microsoft subtly changes its format, or changes the way that newer versions of its software interprets the older format files. The government is forced to upgrade because Microsoft stops supporting the older version of the program, but the newer version does weird things to all those old records when it opens them.
There are various reasons to choose true open formats and standards beyond ideological ones.
Re:Lock-in isn't the point (Score:5, Insightful)
Re:Lock-in isn't the point (Score:3, Insightful)
Re:Lock-in isn't the point (Score:2)
OO and Semi-Permanent Objects (Score:5, Interesting)
In the olden days, people concentrated quite heavily on open formats. Many programmers saw the data as separate from the program. In this environment, one would expect multiple programs to be dinking with the contents of a file. In such a world, maintaining and adhering to published open formats was the key to success.
One of the ideals of OO revolution was that object would own the data. Taken to extremes that means that one object should own the data through its entire existence. Early ideas on the problem of persistence was that OO would just save the internal state of the program to the disk, rather than going through the complex task of converting the data to an open format (risking the potential that other programs would be tempted to modify the data). It seems to me that OO ushered in proliferation of proprietary formats. It definitely provided an excuse for creating proprietary formats.
It seems to me that open data formats is contrary to the ideals of object oriented programming. However, when dealing with data that last longer than the computer, it seems naive that one object will be able to own the data.
Re:OO and Semi-Permanent Objects (Score:3, Insightful)
One of the ideals of OO revolution was that object would own the data. Taken to extremes that means that one object should own the data through its entire existence [...] It seems to me that OO ushered in proliferation of proprietary formats.
You, sir, are obviously not a programmer.
The object-oriented paradigm is just a programming idea for how a program should be "broken down." An OO programmer would look at a problem and think of how it can be broken down into more logical, programmable parts - lik
Re:OO and Semi-Permanent Objects (Score:5, Insightful)
That sounds pretty much like the same insults that were slung in my face umpteen years ago when I was trying to argue for using standardized formats for data in several projects.
Anyway, I find it sad that so few "programmers" actually bother learning anything about the history of what they are doing.
In "legacy programming" people pretty much saw data as separate from the program. In "legacy" programming you would generally go about a project in two steps. You would design the data format in one step and the program in another. In a project you would write and publish both documentation for the format and program.
The problem with this "legacy" design methodology was that you have to keep your published data formats in sync with all programs that access the data. Let's say I had a data store where the lastname was 24 characters. In release 1.17, I change the lastname to 32 characters. If someone else had written a program that was directly accessing and manipulating this data; we would have a crisis.
The bold statement of OO design was that the object owned the data. The object would have complete control of all data from start to finish. You would only publish the interface and not the data format (you would still write both documents, the latter would not be published). This ideal worked well in interface design, but fell flat when dealing with long lived objects saved to disk. Have you ever heard the phrase the "problem of persistence," or are you as ignorant as you are arrogant?
The problem of persistence was this horrible challenge for pure OO design. No matter how you went about it, data needs to be stored, replicated and transferred outside of the control of a single program. Anyway, many of the first OO programmers took the bold statement that the object owned the data to its logical conclusion. They ended up writing programs that saved data in terse, obfuscated, proprietary formats.
I know this happened because I was there with my little hexadecimal editor looking at the data.
OOP was never able to achieve the goal of a single object controlling long lived data. The OO world gradually dropped its bold statement. This is the way history works.
This long post is relevant to today's article because many OO languages still have objectStore methods that save the internal state of the object to disk. I suspect that any OO program using these object store methods are in violation of the MN law. Even though I dislike programs that store their data in bizarre formats, I don't like seeing them legislated out of existence.
Sun and other companies use XML for object persistence. These files are more readable, but still hard to work with.
Please note that I was talking about publication. Publishing a document is different from writing a document. If you hold that the the data should only be accessed through the interface, then you would not publish the file format, now, would you?
I laughed when I read that line. The first OO adopters generally saw themselves as architects. The idea of "breaking things down into procedures" was part of the old "legacy" way of thinking. The OO architects build things up from objects. A person doing OOP isn't a computer programmer. When you think in OOP you are architecting systems, not programming computers.
The really funny thing is that OOP is systematically becoming more like the traditional programming paradigms that it was supposed to transcend.
Re:OO and Semi-Permanent Objects (Score:2)
Indeed they do, but things are perhaps getting better. Ruby [ruby-lang.org], for instance, is switching to use YAML [yaml.org] for this, a standard that is open, human-readable and terse, the two latter much more so than for instance XML. YAML has features specifically designed to allow object persistance for any language as well as persistance of any data.
It works nicely without cr
Re:OO and Semi-Permanent Objects (Score:2)
Not only that, he's calling the kettle black, and I doubt he's a programmer himself. A real programmer has experience outside of OO, and is aware that there's a valid argument that there are other models than OO. One doesn't have to agree with the arguments, but no one calling themsel
Re:OO and Semi-Permanent Objects (Score:2)
posturing (Score:2)
Still, if enough states and countries do this
Re:I hope it passes (Score:2)
Either way, with the best efforts of the Star Office, Open Office, Word Perfect, and many, many other teams of people focused on figuring out the standard, it is still impossible to get
Blue-state phenomenon? (Score:4, Insightful)
Re:Blue-state phenomenon? (Score:5, Interesting)
Re:Blue-state phenomenon? (Score:1)
Re:Blue-state phenomenon? (Score:2)
Couldn't that be said for most states during the last election?
Re:Blue-state phenomenon? (Score:1)
Re:Blue-state phenomenon? (Score:2, Informative)
True, it was closer than say 1984, but that was to be expected in a nation so equally divided by so many things (War, economic issues). Make no mistake though, MN is still very much a liberal state.
The important thing to note also that MN has a very strong tradtion of 3rd parties, and that may have figured into it as well.
http://uselectionatlas.org/USPRESIDENT/state.php?y ear=2004&fips=2
partisanship (Score:2, Interesting)
Both "major" parties have a vested interest in protecting the status quo. Nearly all states went to that sort of "all or nothing" EC allocation in the early mid-19th century, you know. It's not like partisanship is a new phenomenon. Geo. Washington warned us it.
We need a system that reduces the dependency on partisanship. I don't have anything against partisanship, per se, but I do not like some of its effects. Making the system nonrewarding to those effects would help diminish those aspects. I belie
Re:Blue-state phenomenon? (Score:1)
Re:Blue-state phenomenon? (Score:5, Insightful)
Don't get too excited. If it wasn't for Governor Mitt Romney (R) ODF in Massachussets would be dead, and it's primary opponent was state senator Marc Pacheco (D).
Re:Blue-state phenomenon? (Score:2)
Re:Blue-state phenomenon? (Score:2)
Re:Blue-state phenomenon? (Score:2)
Like... ummm... Macaroni ?
Are the standards ready? (Score:3, Insightful)
Having said that, I think it's high time our public offices stop feeding monopolistic practices by continuing with document format requirements that more or less pre-determine software choice.
Well, (Score:5, Insightful)
Re:Well, (Score:2)
Re:Are the standards ready? (Score:1)
This didn't happen with
Re:Are the standards ready? (Score:4, Informative)
The MA situation doesn't involve legislation, but is an executive order from the IT department. The IT department is responsible for implementing the switch, and there's no reason it couldn't abandon the project if it turned out to be unworkable. They're also perfectly able to make exceptions for cases where they can't get it to work or simply don't feel like dealing (if someone had an extremely complex Word macro that they use a lot, and the ITD couldn't figure out how to do it in ODF, they could just shrug and let it go), because it's just a policy, not a law.
With respect to the maturity of ODF, it was developed by a group of organizations which, between them, are likely to have all of the needs that anyone has. For example, the Society for Biblical Literature was an active member of the technical committee. This may be a bit surprising, until you realize that they've got at least one document (a translation of the bible) in every known living language, documents in many dead languages, and things like illuminated hand-written manuscripts. Additionally, ODF was designed to include the concepts in Microsoft Office formats (based on existing converters and on inspection of the interface presented to the user).
Re:Are the standards ready? (Score:4, Informative)
Moreover, I suspect they may have more technical insight than most - LTR/RTL, printed and script, heavy diacritical use, cuneiform, IPA and other transliteration schemes, etc. are technical hurdles they've been dealing with for quite some time now in both printed and electronic format. They have even been freely distributing a Hebrew font [sbl-site.org] for years.
Just wanted to clear that up, lest people think they are a group of bible thumpers or modern monks (e-monks?).
Re:Are the standards ready? (Score:2)
And it is a bit unusual for a group of humanities academics to get involved in technology, unfortuna
Re:Are the standards ready? (Score:1)
The format itself is a zip file with some xml and image files, etc. It's very easy to use.
I'm unrelated to the OOo project or ODF, but as my .sig says I write a converter that d
Re:Are the standards ready? (Score:2)
Re:Are the standards ready? (Score:2)
That is a fantastic idea, and I completely agree with it. Basically what you're suggesting, it sounds like, is that we have document format requirements that actually guarantee any software provider the legal right to implement support for the format chosen. I totally support that.
Yes, that would be a great cure for the abuse of mon
Re:Are the standards ready? (Score:3, Informative)
I'm far from any real expert on the format, but at my job, I have done some fairly non-trivial conversions of technical documentation (in DITA XML, if anyone cares) into ODF, and while what I'm doing is fairly rough (it's an in-house use sort of thing) the format does seem to support all the basic concepts of a word processing document... page layouts, running headers/f
Re:Are the standards ready? (Score:2)
It's not ready. I'm a koffice user and I keep seeing problems - not big ones, but problems - where the standard is ambiguous and it ends up being koffice changing to what o
More openness is only a good thing. (Score:4, Insightful)
Wow, my state legislature is working. (Score:1)
Re:Wow, my state legislature is working. (Score:1)
Hopefully Pawlenty will bumb it up a bite more if it passes.
http://www.minnesotashubert.org/ [minnesotashubert.org]
Open Standards / Open Source (Score:4, Interesting)
But this isn't about Open Source, it's about Open Standards, two orthogonal issues. Of course, Open Source is preferable, but it's not required to have Open Standards. Microsoft could add ODF support to its next version of MS Office (which they'll of course try to resist for as long as possible, as it'll kill their market lock-in), and it would be viable as a software supplier, but it'd have to compete on ease of use, price, robustness, etc.. It'd have to compete on its merits for once, instead of being the mandatory choice because of the current platform lock-in (even though OpenOffice.org does an excellent job interoperating with MS Office files).
Re:Open Standards / Open Source (Score:4, Insightful)
MS will incorporate ODF, and will do it very badly.
Same as with the web. You'll have to have an OpenOffice.org ODF, and MS Office ODF, and the two will not meet without quirks in OpenOffice.org
It's going to be ugly. MS's resistance is just their first bit of opposition.
Re:Open Standards / Open Source (Score:2)
Re:Open Standards / Open Source (Score:1, Interesting)
Computer users would not put up with image format XYZ only working in program XYZ and image format ABC only working in program ABC, etc. They choose open formats. The programs using open formats then compete on functionality not lock-in.
An Open Standard also helps with data longevit
Re:Open Standards / Open Source (Score:2)
Minnesota is dead on target (Score:5, Interesting)
Royalties (Score:3, Informative)
Re:Royalties (Score:2, Informative)
Actually, despite a lot of non-committal grunts, that's not announced yet one way or the other even for current versions of MOOX and its current licenses. Obviously MS knows the licenses are going to be scrutinized carefully so the odds of any gotchas being easy to spot are low. I'd be really careful about the wording in the license anyway. I wouldn't be surprised in the least if MOOX were dependent
Last time I checked.. (Score:4, Funny)
Re:Last time I checked.. (Score:1)
Yeah, Microsoft, again.
Only between characters 32 and 127. (Score:2)
The actions taken as a result of characters from 0 through 31 are specific to the implementation and circumstance. (An embedded motherboard
Re:Unicode (Score:2)
(Unicode has gone through a few revisions, which I am also a bit concerned with. Once you have a formal specification la
Re:Unicode (Score:2)
Having said that, between this thread and a previous one on Unicode in which I debated the merits of the vatious bit-lengths, I'm now satisfied that Unicode i
Re:Last time I checked.. (Score:2)
You went for the humor mod, and there's nothing wrong with that, but once you unzip an ODF file, you actually have ASCII text. (Well, strictly, something like utf8-encoded XML files, but notepad/wordpad handles them decently... smart-quotes notwithstanding, but they're evil anyway. ;)
Not a big story yet (Score:5, Insightful)
It is good that such an idea is starting to bubble up, but it has yet to pass into law ANYWHERE at this point. The political power, wealth and proven tendency to resort to outright illegal measures of the Foe is going to make this a long difficult struggle.
How government procurement works (Score:1, Informative)
where the other requirements of a project do not make it technically impossible to do this
The thing is, 90% of government purchasing is steered by very, very careful tailoring of the claimed "requirements". The way government purchasing works, what this bill is very likely to do is just make it so all procurement requests are written up in such a way that it is "impossible" for a format to meet
Re:How government procurement works (Score:4, Insightful)
So, what happens, is you specify all your requirements, and in reality, only a few vendors meet those requirements, but a dozen or so "cheaper" vendors who meet the specs on paper end up winning the contract, and then we're stuck with crap.
This has lead to requirements being VERY specific, so that you don't end up with something that barely does what it claims to.
NOT! (Score:3, Insightful)
Put simply... (Score:4, Insightful)
MECC (Score:4, Informative)
Government assisted-funding software could again have a positive impact as MECC did. OOo, Mozilla, etc. SE Linux even...
Re:MECC (Score:3, Informative)
In Theory (Score:2)
Sadly enough, it puts the burden on WordPerfect to support ODF. Which isn't coming real soon. http://www.linux-watch.com/news/NS7758948461.html [linux-watch.com] I think Wordperfect is the preferred word processor in the
Great... until the feds step in (Score:2)
This will be great, until, just as they're trying to do with food labelling standards, the federal government makes a law with a new standard that specifically invalidates any state laws that are more restrictive.
But I guess you can't blame the lobbyists... would you rather bribe..err..payoff..err, I mean, lobby a handful of people in one town, or have to spread your efforts ac
Old story (Score:1)
There is nothing GNU about these.
It would nice to be more than just formats (Score:5, Interesting)
Re:It would nice to be more than just formats (Score:2)
Duluth, Open Format Epicenter!! (Score:1)
It wont pass - They will try to Tack Gay Marrage B (Score:1)
This is not OpenOffice.org v. Microsoft (Score:2)
This is not about some OpenSource community effort versus Microsoft.
This is about a Sun/IBM alliance versus Microsoft. IBM and Sun are both quite capable in terms of political efforts. I'd put IBM way above Microsoft, for that matter. Sun's StarOffice for smaller organizations, IBM's Workplace for enterprise class, and OpenOffice.org to fill in various gaps.
Fortunately, this is not David versus Goliath. This is more like clash of the titans, and in terms of wooing
Obligatory pessimists view (Score:3, Insightful)
OTOH, might be a way to eventually bleed MS dry, eh?
(Might take 1000 years...must take the long view)
Impossible ? (Score:3, Insightful)
Re:Impossible ? (Score:2)
Re:Impossible ? (Score:1)
so now we'll see very carefully tailored... (Score:2)
I recall, back in the days when IBM was the enemy, being told by someone that his boss handed him a copy of the specs for a 360/370 clone and a copy of Principles of Operation and told to find some difference betweeen the two. I imagine that difference was written into a requirements specification so that only the IBM iron would satisfy it.
Minnesota (Score:3, Funny)
No Anti-Trust Trial? (Score:1)
Anyone living in Minnesota (Score:2)
Wont work (Score:1)
Re:Wont work (Score:2)
I haven't tried it lately, but quite frequently those "ancient documents" open fine in OO.org, and DON'T in recent versions of Word. Not supported.
I think I still have an ancient mac with system 6 and Word on floppy, probably v2x-v4x, V5 was too big for floppy based usage IIRC.
I'll have to give it a go.
You actually provided the PERFECT example of the reason for the law.
Re:Wont work (Score:2)
Most gvt[sic] agencies are required by law to keep all the old records. Many of wich[sic] were created during the days of windows 3.1 and 3.2. For some odd reason documents created back then (this is all assuming they were made with microsoft office) can not be oppened[sic] with openoffice or any other present day software excluding microsoft office.
What an interesting assertion. It contradicts both my personal experience and the general consensus among IT workers. I have inherited many old .doc files o
Has ANYONE actually thought this through??? (Score:1)
Does Microsoft Word file formats qualify as an "open format?"
Does Adobe Photoshop file qualify as an "open format?"
Does Quark Express document file qualify as an "open format?"
Does an Oracle database count as an "open format?"
As far as I can see, all this proposal would do is to:
(1) Stop the state government from using a TREMENDOUS amount of useful software.
(2) Incur insane compliance costs when trying to get employees not to save in the
Re:Has ANYONE actually thought this through??? (Score:1)
You haven't lived until you've tried to explain to a cranky 80 year old what acrobat reader is, nevermind open office.
The real problem is that the open standards aren't useful for people and the standards that are used aren't open.
Re:Has ANYONE actually thought this through??? (Score:4, Informative)
"Open standards" means specifications for the encoding and transfer of computer data that:
(1) is free for all to implement and use in perpetuity, with no royalty or fee;
(2) has no restrictions on the use of data stored in the format;
(3) has no restrictions on the creation of software that stores, transmits, receives, or accesses data codified in such way;
(4) has a specification available for all to read, in a human-readable format, written in commonly accepted technical language;
(5) is documented, so that anyone can write software that can read and interpret the complete semantics of any data file stored in the data format;
(6) if it allows extensions, ensures that all extensions of the data format are themselves documented and have the other characteristics of an open data format;
(7) allows any file written in that format to be identified as adhering or not adhering to the format;
(8) if it includes any use of encryption, provides that the encryption algorithm is usable on a royalty-free, nondiscriminatory manner in perpetuity, and is documented so that anyone in possession of the appropriate encryption key or keys is able to write software to unencrypt the data.
Too broad, I think (Score:2)
If you look at this solely in terms of word processing, yeah it makes sense.
But consider a database product like say Oracle. Since they don't document the file format, they're in violation of this bill. Yet it's relatively trivial to extract data from an Oracle database and use it elsewhere. And if your app uses ANSI SQL you can move relatively easily between Oracle and other database servers. Hence the historical definition of Open Systems, a
Re:Too broad, I think (Score:2)
Re:Too broad, I think (Score:2)
Again, the point was... Ideally what you want are Open Systems. That is, the ability to interoperate, retrieve data, etc. in a useful manner. It's more complicated than just the file store specification.
Sadly, I live in Minnesota, so I get to suffer in the pocketbook from this idea.
headline: "MN bill makes WA Bill Waah!" (Score:2)
I mean we don't need open standards -- we need we need competition, er, innovation, er, securit... uhm... you know what I mean!
Probably a good idea, and probably a bad bill (Score:4, Interesting)
What might be surprising is that I hold this view inspite of being a Microsoft Employee. While I certainly want as many people as possible using MS software, I want them doing so because it's the best choice for their situation. I'd like to think that Word can deliver more value to its users than the ability to open Word files, so if govt agencies want to mandate that documents be created, shared, stored, etc in published, royalty-free formats, that's fine with me. Government agencies are a large customer of ours, so hopefully government action around requiring open file formats will push us to make our tools best-of-breed for dealing with those formats, or may even push us to open some of our own.
I don't use WMA for my music files, even though I could just email the guy that designed it if I have a problem. Just because I can today, doesn't mean I can 5 years from now. And there won't ever be a supported WMA player for something like OpenBSD, which might otherwise be a perfectly good audio appliance.
Now here's where I explain the title. As much as I am for the idealistic POV that open formats should be used where possible, I also beleive that govt is amazingly effective at turning a good idea into a bad law. (See also: 99% of current US laws). Another comment suggested that this story is more about Sun/IBM fighting MS via legislation, as opposed to some ideological position that is being done for the best interests of the people. If that's true, it's unfortuneate that our govt is continuing to do these sorts of things that are allegedly in the best intersts of "the people" but no one can explain exactly how, and ultimately, other businesses or politicians seem to derive the most benefit.
Re:Quick question: (Score:2)
If I were going to speculate, I'd suggest that ODF isn't the "right" format. I'd think that MS would be more likely to "open" the Word XML document format in Office 12 than to adopt someone elses standard.
One thing that can suck about an open format is that it's hard to understand when it's frozen. People really hat
Re:Quick question: (Score:2)
For example, it is quite important to have italics in the right places. Let's assume that the first version doesn't support italics and there is an "extension" to that version that does. I would offer that this generates the possiblity of having a reader which will not render something in italics when it
Govt is ofter the first (Score:2, Insightful)
Whether building dams in Nevada or sending Americans to the moon, it is the one entity that can face risk of failure and not flinch. In part this is because it represents our collective courage to attempt the impossible.
For this re
Brilliant! (Score:2)
I wholeheartedly approve. For years, one of my biggest and most complicated tasks has been the transfer of data from source A to target B. Usually A is in a format that is undocumented and inconsistent. Reverse-engineering can be a nightmare when formats are badly designed (although that explains why they are often considered propietary) and used randomly. You would be surprised by the vast number of programmers who neglect to start a file with an identification tag and format version number.
I would happi
Open Systems more important than Open Source (Score:2)
It's a depressingly common situation to find oneself in: Reverse-engineering a protocol or format by playing with inputs and looking at outputs because that's easier than decoding (hah) the source that generates it.
Hm. I wonder (Score:2)
Unlike the other ITU-T G.X voice standards, G.729 requires licensing fees to the patentholders.
Better make sure its free of patents (Score:2)
Re:I live in MN, stuff open source crap (Score:3, Insightful)
As for "de-facto standards" - even de-facto standards are supposed to be documented and open to competitors. Standards exist for more than one entity to adhere to. Otherwise it's a "de-facto monopoly".
I live in MN - What is the bill numbers? (Score:1)
Thanks
Re:I live in MN, stuff open source crap (Score:1)
So, you're saying you prefer the original, bloated office application that doesn't work properly... but costs a lot more?
--
Get out of my state, Red.
Re:I live in MN, stuff open source crap (Score:3, Interesting)
This is certainly a Good Thing.