decoration decoration
Stories

GROKLAW
When you want to know more...
decoration
For layout only
Home
Archives
Site Map
Search
About Groklaw
Awards
Legal Research
Timelines
ApplevSamsung
ApplevSamsung p.2
ArchiveExplorer
Autozone
Bilski
Cases
Cast: Lawyers
Comes v. MS
Contracts/Documents
Courts
DRM
Gordon v MS
GPL
Grokdoc
HTML How To
IPI v RH
IV v. Google
Legal Docs
Lodsys
MS Litigations
MSvB&N
News Picks
Novell v. MS
Novell-MS Deal
ODF/OOXML
OOXML Appeals
OraclevGoogle
Patents
ProjectMonterey
Psystar
Quote Database
Red Hat v SCO
Salus Book
SCEA v Hotz
SCO Appeals
SCO Bankruptcy
SCO Financials
SCO Overview
SCO v IBM
SCO v Novell
SCO:Soup2Nuts
SCOsource
Sean Daly
Software Patents
Switch to Linux
Transcripts
Unix Books
Your contributions keep Groklaw going.
To donate to Groklaw 2.0:

Groklaw Gear

Click here to send an email to the editor of this weblog.


To read comments to this article, go here
The Future Is Open: What OpenDocument Is And Why You Should Care ~ by Daniel Carrera
Sunday, January 30 2005 @ 05:00 AM EST

I asked Daniel Carrera, an OpenOffice.org volunteer, if he'd please explain to us the OpenDocument format. How does a format get chosen? And is OpenDocument on the list when governments like the State of Massachusetts make up such lists of acceptable formats for governmental use? If not, what can be done to change that? He graciously agreed. Because we are all concerned about proprietary formats and standards, and more and more governments are adopting policies requiring open standards, it's a very important subject.

Daniel calls OpenDocument "our best chance to fight vendor lock-in associated with proprietary formats." It's currently being considered by the EU Commission as a candidate for an official format. Note that you can also download this article in various formats, including, naturally, OpenDocument, from links at the end of the article. In some browsers, you may need to Save As, instead of just clicking on the links.

Daniel Carrera is a mathematics PhD student at the University of Maryland, as well as an active OpenOffice.org volunteer. He serves in the OOo community council as the "community representative". Daniel also facilitates the OOoAuthors project, where most English documentation is developed. He will be speaking at the OOoRegiCon in San Diego on Feb 9, the day before and leading into the " Desktop Summit 2005", a conference about the present and future of open source and desktop Linux technologies being held in San Diego, CA February 9-11. He is speaking opening day on the subject, "INREACH, Building an Active Community."

The complete list of speakers is here, and I must say, it sounds like fun. Mitch Kapor and Doc Searls are two of the keynote speakers at the Desktop conference. Sun's Simon Phipps kicks off OOoRegicon, and I've heard he is an excellent speaker. You can find out more here. It's limited seating, and the page says the price goes up after February 5.

*******************************

The Future is Open: What OpenDocument Is and Why You Should Care,
~ by Daniel Carrera


What Is OpenDocument?

Executive summary

Quiz: Complete the sentence,"OpenDocument is ..."

(a) An open, XML-based file format.
(b) An open standard, supported by the OASIS and ISO standards groups.
(c)The default file format for the upcoming OpenOffice.org 2.0 and KOffice 1.4.
(d) A top prospect for an official format for the European Commission.
(e) Our best chance to fight vendor lock-in associated with proprietary formats.
(f) All of the above.

The correct answer is (f) All of the above.

Timeline

This article links to reports written over a period of several months. To make things easier to follow (in particular, the name of the format), I include an approximate timeline.

  1. OpenOffice.org creates an open file format called "OpenOffice.org 1.0 format."

  2. The European Union commissions Valoris to report on open file formats.

  3. The OpenOffice.org 1.0 format is submitted for OASIS standardization. KDE and Corel join the OASIS Technical Committee and expand the format to cover a wider range of applications.

  4. The new OASIS format is called "Open Office XML". OpenOffice.org and KOffice both commit to making the format their primary/native format.

  5. The Valoris report is published. Microsoft and Sun respond to it. The European Union TAC makes recommendations.

  6. The format is submitted for ISO standardization and changes its name to the OpenDocument format.

In order to avoid promoting an outdated name, I will always refer to the format as OpenDocument.

The European Union and the Valoris report

The key player here is the European Union's Telematics between Administrations Committee (TAC). The TAC hired the Valoris consulting group to explore the possibility of using Open Standards to create a more competitive marketplace.

The Valoris report was well researched and very thorough. They recommended the adoption of an open XML format backed by a recognized open standards group.

The Valoris report is a very detailed and most compelling document describing how open XML technologies are impacting government information systems and requirements policies.

It's difficult to summarize a 78-page document. I encourage you to read it yourself [PDF]. But over all, it is well-researched and rather encouraging of open source, OpenDocument, and EU hopes for an open marketplace.

The Valoris group closely monitored the development of the OpenDocument format, and by November 2004 every one of the concerns and issues they raised were directly addressed by OASIS TC. As a case in how governments and organizations go about setting information system requirements policy, the Valoris effort provides an unparalleled insight.

Selection criteria

For the report, Valoris compiled an exhaustive list of existing file formats. These were compared against a series of requirements. Only two formats performed well, and these were selected for further analysis.

Being familiar with these criteria is important for us. It teaches us what a government requires before it will adopt an open XML format. I divide these into three groups:

Neutrality
  1. Open: At this stage of analysis Valoris only looked for openness in the sense that a public, royalty-free specification is available. For example, PDF and MS XML both met this requirement.

  2. Non-binary: Binary formats get in the way of neutrality. A format that depends on Windows components will be hard to support on GNU/Linux and Mac OS.

  3. Cross-platform: An obvious requirement for neutrality.

Technical merit
  1. Preserve format fidelity: This refers to both presentation and structure. For many applications, format fidelity is an absolute imperative.

  2. Modifiable: This excludes formats like PDF.

  3. Support current word processor features: An open format is no use if it can't represent your data. This list included Unicode support, bi-directional (Hebrew), and scripting, among others. Abiword and KOffice failed this requirement.

  4. Support emerging requirements: Digital signatures, access rights, version control, etc. Almost every format failed this requirement.

Widely adopted

This sounds odd. Why is this important? Shouldn't we pick the best format regardless?

The answer is simple: Have you heard of the ODA? I didn't think so.

In the 80's Europe made an attempt to define an open standard format called Open Document Architecture (ODA). It received a lot of support from European institutions including the European Commission. It was also an ECMA standard and an ISO standard.

ODA failed miserably.

ODA had no backing from the industry. The format was complex, and the companies preferred to support more pragmatic standards such as SGML and RTF.

Conclusion: Industry support matters. This is why the W3C never accepts a standard unless it has at least one working implementation.

Note: "widely adopted" does not mean "dominant". It means that there is enough industry adoption to sustain the format. For example, OpenDocument meets this requirement. Unlike the ODA effort, OpenDocument enjoys the support of two office suites as well as major players like Sun, IBM, HP, Novell and even Adobe.

Finalists

No format met all the criteria perfectly well. But two formats stood out above the rest:

  • OpenDocument - main failing: lacked some support for emerging requirements.
  • Microsoft XML - main failing: medium fidelity, unsure about cross-platform status.
OpenDocument vs MS XML

The only apparent advantage of MS XML over the OpenDocument format was the presence of custom-designed schemas.On the other hand, OpenDocument had several advantages:

  • More open: No legal constraints, and support from OASIS.
  • Reuse of existing open standards when possible (SVG, Dublin Core, MathML, etc).
  • Higher format fidelity.
  • Friendly to XSLT and other XML-based tools.

Reactions To The Report

Microsoft Responds to the Valoris Report

Microsoft's response [PDF] essentially comes down to trying to convince the reader that custom designed schemas are really really important. This isn't surprising since it seems to be the only card they could play.

Sun Responds to the Valoris Report

Sun's response [PDF] expresses several concerns about Microsoft's approach to XML. I recommend reading the full response for details, but here are a few:

  • MS XML doesn't cover all of MS Office (for example, it doesn't cover PowerPoint).

  • MS XML doesn't support some of the advanced MS Office features.

  • MS XML can contain binary objects that depend on MS Office and Windows (e.g. OLE and VBA) and those lack complete documentation.

  • MS did not commit to make future changes to MS XML available to the public, only the current one.

The EU TAC Makes Recommendations

After getting responses, the EU TAC wrote their final recommendations. Those too are very encouraging. They stood firm on the importance of file formats:

"...the public sector should avoid any format that does not safeguard equal opportunities to market actors to implement format-processing applications, especially where this might impose product selection on the side of citizens or businesses..."

Their response to Microsoft's argument on custom-defined schemas was interesting. Instead of arguing whether the feature is really valuable, they chose to challenge OASIS to add custom-defined schemas to the format to match Microsoft's argument. The TAC also had some suggestions for Microsoft, finding that the MS XML and MS XML Reference License were simply not open enough.

Recommendations for OASIS:

  • Add custom-defined schemas to the format.
  • Submit the format to ISO for a more official seal of approval.

Recommendations for Microsoft:

  • A public commitment to keeping future versions of MS XML public.
  • Submit the format to an international standards body of their choice.
  • Remove non-XML components from the format.

Recommendations for other industry players:

  • Participate in the OpenDocument standardization process to encourage wider industry consensus around the format.
  • Include filters to support both OASIS OpenDocument and MS XML.
  • Provide tools to help the public sector migrate its documents to XML formats.

Recommendations for the general public:

  • Provide your documents in multiple formats, or alternatively, in an open format with industry consensus and adoption.

Microsoft responds to the TAC recommendations

Microsoft's response [PDF] essentially comes down to:

  • They agree to publish future versions of MS XML under non-discriminatory terms.

  • They say they will "vigorously" work on documenting the non-XML elements of the format, though they disagree that they should exclude all non-XML elements. They claim that OpenDocument also has non-XML elements (such as images) but fail to point out that in OpenDocument those are in a separate directory and in MS XML they are embedded throughout the XML tags.

  • They also say that supporting various formats is important, but fail to say that they'll support OpenDocument.

  • They insist that their licensing scheme is just fine, and that "royalty-free licensing programs have a role to play alongside formal standards". Translation: No, they won't submit it to a standards body.

Sun responds to the TAC recommendations

Sun's response [PDF] reiterates the importance of formats that are truly open, in every sense. They like the idea of ISO support and are confident that the format will become an ISO standard.

IBM responds to the TAC recommendations

IBM's response [PDF] was also positive. There was a tiny bit that concerned me:

... we believe [that standard] technologies should be published without restriction (other than reasonable royalties for essential patents)...

Well, I don't really know much about this, so I'll let other people decide if the stuff in brackets is significant. But other than that, IBM informs the EU that:

  • They will join the the OASIS Technical Committee.
  • They already offer products (Workplace) that conform with OpenDocument.

This last bit isn't surprising, since Workplace is based on OpenOffice.org code, but it's a good reminder.

Where We Are, Where We Need To Go

The OpenDocument format is arguably the single most important step for the FOSS movement after software patents. It is unique in that:

  • It cuts right to the core of the fight: Vendor lock-in.

  • This is a battle we can win.

This Is a Battle We Can Win

The OpenDocument format is in a unique position because it has a real chance of succeeding:

  • Technical merit: With the inclusion of custom schemas, OpenDocument can meet and exceed every technical hurdle that can be thrown at it.

  • Adoption: OpenOffice.org, KOffice, IBM Workplace, StarOffice. OpenDocument is not a dream. It is a real format with enough support to present a real alternative.

    OpenOffice.org is cross platform, easy to download and install, and is preparing to ship version 2.0. Sun, IBM, HP, Red Hat, Novell,and even Adobe were able to announce full support of the EU open XML standards requirements based on the inclusion of OpenOffice.org components in their products and services. That both OpenOffice.org and KOffice are open source with open component frameworks enables the entire marketplace of vendors to easily comply if they choose to do so.

  • Open standard: OpenDocument is more than an open format, it is a open standard . That is, it is backed by standards groups, ISO and OASIS. It is not controlled by any company or product. Not by Sun or IBM. Not by OpenOffice.org or KDE. Is Microsoft willing to match that?

  • Pushing OpenDocument as a Standard : The European Union is pushing for a policy of open standards. So is the state of Massachusetts, USA. The time is right to push for OpenDocument as the standard for government documents. People are listening.

What OpenDocument Needs To Succeed

The Valoris report makes it clear that any open format that is to succeed as an official standard for government documents needs four things:

  • Technical merit: Most open formats that were evaluated did not make it to the final round because they lacked important features. Any technical shortcoming, no matter how theoretical, gives Microsoft a lobbying opportunity.

  • Adoption: No adoption, no success. Learn from ODA.

  • Open standard: Why is this so important? Why not just any open format like the old OpenOffice.org 1.0 format or KOffice? Because Microsoft would argue unfair treatment. Yes, I know, ha ha. But still, Microsoft could effectively lobby against any format controlled by one application supplier.

    This is why OpenOffice.org and KOffice are both switching to OpenDocument as their primary format (in versions 2.0 and 1.4 respectively).

  • The Fight for Open Standards, OpenDocument As a Standard: The fight for open standards will not be easy. It will take will power from politicians, and the public who elects them, to come through. Open XML technologies are the future of collaborative computing, but the proprietary efforts to compromise their implementations, or lock them up with insidious patents and restrictive licenses, continues.

What You Can Do To Help

Keep your focus on the four items above. Let's go through each and see what we need.

Technical merit

Done

If you find a significant failing in the format, join the OASIS TC and participate. Or, you can follow the example of XML expert David Wheeler. Even though David is not a member of OASIS, he routinely participates in discussions and contributes to the specification through the public forum. The OASIS TC also maintains liaisons with other open XML efforts such as the OASIS UBL and the W3C XForms groups.

Then there is always the option of working through one of the many OpenOffice.org project groups to directly impact the specification. Examples of this include the Bibliography and DocBook groups who labored long and hard to prepare the specification to perfect the demands of the Library of Congress' very complex "MODS" schema.

Adoption
  • Write filters for Abiword, Gnumeric, Sodipodi and Inkscape.

  • Promote applications which support OpenDocument.

  • Help at OpenOffice.org or KOffice.

Open standard

Done

Pushing for OpenDocument
  • Help convince the EU and MA to support OpenDocument.

  • Write letters, lobby.

  • Inform people. Feel free to use this article to do so.

  • Become informed yourself.

    • Read the Valoris report and the documents listed here.

    • If you live near San Diego, California, go to the OpenOffice.org RegiCon on Feb 9 and hear Gary Edwards from OASIS talk about OpenDocument:

      The Shot Heard Round the World: How OASIS Open Document Changes Everything.

      Description of the talk: "Now two years in the making, the OASIS Open Document XML file format specification is making its mark. This session will discuss how the OASIS TC made dramatic enhancements to the file format specification to meet the next generation information system requirements of the European Union. Enhancements such as XForms, SVG, and SMiL were needed to meet the "custom-defined schema" requirement. Submission of the specification to ISO (the International Standards Organization) was another challenge met by the TC. This discussion will also cover the dynamics of Microsoft's arguments to the EU that MSXML file formats and licenses are "open enough". The EU held their ground though. The requirements for " Open Standards based Open XML file formats" are now in the EU purchase cycle. A fact which quickly led to the capitulation of Microsoft in mid November of 2004, when they agreed to comply with the requirements in future releases, as well as to work with Sun to provide OASIS/ISO file format filters as plugins for current MS Office applications. Since IBM's WorkPlace desktop environment is based on OpenOffice.org components, IBM, Adobe, and over a 150 other IBM affiliated information technology vendors were also able to announce compliance with the EU requirements in November. When open standards policies find their way into the purchase cycle, the marketplace becomes competitive in every way imaginable. More importantly though, open standards based policies enable computational consumers take ownership of both their information, and the information processes they depend on. And that changes everything."

    • Attend the above talk and provide a report for Groklaw.

Acknowledgments

Gary Edwards, from the OASIS TC, reviewed this article for technical accuracy. Jean Hollis Weber edited the final draft. Any errors or omissions are entirely my own.

Resources

Read the Valoris report, along with TAC recommendations and all responses.

This article is licensed under the same Creative Commons License as PJ's articles. Daniel Carrera can be reached at dcarrera at openoffice.org

You can download this article in OpenDocument format as well as OpenOffice.org 1.0 and PDF.


  View Printable Version


Groklaw © Copyright 2003-2013 Pamela Jones.
All trademarks and copyrights on this page are owned by their respective owners.
Comments are owned by the individual posters.

PJ's articles are licensed under a Creative Commons License. ( Details )