If you're not worn out yet and missed my blog post from earlier today, check out this quote from Adam Farquhar, previously quoted in Microsoft's Ecma press release as an XML Reference Schema supporter:

Early in November, Microsoft announced a project to digitize 100,000 rare and out-of-print books from the British Library collection. [For more information, see the NewsBreak at http://www.infotoday.com/newsbreaks/nb051121-2.shtml.] Farquhar says that that effort is not directly related to the Open XML announcement: “Some people think we are adopting Microsoft formats as our standard for digital preservation. This is not right; we are striving to make sure that content we receive in MS formats will be preserved.” He continued: “What format will we deliver? We deliver a lot of articles and in many formats. We deliver content in PDF, Office Open, ODF, TIFF — whatever format the customer wants.”

Well. That's different than what Microsoft represented, isn't it? Updegrove:

In short, the British Museum is not recommending the XML Reference Schema as an exclusive solution, but giving a sigh of relief that if Microsoft has decided to offer some sort of methodology to permit long-term preservation.

Unfortunately, it takes time for news such as this to make its way into the marketplace, if it ever becomes widely disseminated at all. In Massachusetts, the flow of high-quality, timely information is particularly important, because the legislature will decide in the not too distant future whether the ITD's ODF policy will live or die (sadly, the same may apply to the ITD's chief, State CIO Peter Quinn, if his opponents have anything to say about it).

How soon could a decision be made by the Legislature? My most recent information is that passage of the bill in question is unlikely before the end of the year, but this should not be regarded as conclusive. Of note in this regard is the fact that another public discussion will be held on December 6, this time convened by Senator Hart, and (this time) including advocates on both sides of the issue

The bottom line is that OpenDocument has already been approved as a standard and is already supported by several products. The Microsoft effort, by their own estimate, will take a year. Some analysts are already questioning whether Microsoft's format will ever be really open:

... Gary Barnett, a research director at analyst firm Ovum, said on Tuesday he doubted that the move would result in the format becoming "truly open". "It's a tactical move by Microsoft to give its proprietary document formats a glimmer of openness," Barnett claimed. He added that Microsoft is only entitled to describe its file formats as open if it "gives up control of its formats to a standards body that is accessible".

If Microsoft maintains control over its XML-based file format it will be able to arbitrarily change the standard when it wants, enabling it to keep ahead of any competitors that wish to implement the standard, according to Barnett.

Mark Taylor, executive director of the Open Source Consortium, agreed that Microsoft's move was not as open as it might first appear. He said that Microsoft appears to want to extend its "Office monopoly into the XML age. If the intention is to really play nicely with others in the open standards game, then why patent applications in this area?"

Microsoft must be puzzled. Why is it almost no one trusts them, even when they do something that on its face looks good? Here they've announced a patent covenant for their XML, and we're all out here with our magnifying glasses looking for gotchas. But let's face it. There's a lot of water under this bridge. The burnt child dreads the fire and all that. We remember Netscape and Java and everything, all those tricky Microsoft gotchas. It's hardly irrational to look for more. *Not* looking for them would be irrational, given the history. In fact, let's review a little Microsoft history.

We remember when Microsoft joined the World Wide Web Consortium (W3C), an international consortium which develops Web standards and guidelines, to build consensus around Web technologies such as for HTML. Despite being a member, Microsoft developed their own proprietary HTML extensions anyway, their own little tweaks and twists, on the specification, which degrades the display of a good deal of the web by other browsers, simply because Microsoft won't follow the standard themselves, or maybe worse. They hold a dominant position in terms of numbers of users. You know, that old monopoly thing. So it's a meaningful negative effect. So just joining a standards group doesn't necessarily mean Microsoft will play fair. Remember this little incident in 2001?

Last week, people who tried to visit MSN.com with a non-Microsoft browser found themselves locked out. Although Microsoft's own Internet Explorer easily accessed the popular site, other browsers--such as Opera, Mozilla, Amaya and some versions of Netscape--received error messages and recommended that people "upgrade" to Internet Explorer.

The purpose of standards is so that everyone can speak the same "language," so to speak. Remember when that airplane crashed, because the air traffic controller and the pilot didn't speak the same language? The Web is international. It's just a lot of computers all over the world trying to interact. Without an agreement on how to meaningfully and seamlessly do that, the value of it goes down, which means that by doing what it did with HTML, Microsoft devalued the Web for everyone else, while enhancing its own market position. Just a few little extensions, and gotcha.

Think that's ancient history? W3C has a validation service, where you can go this very day and check your HTML for errors. Here's the W3C's validation report on msdn.microsoft.com: "Failed validation, 68 errors....This page is not valid HTML 4.0 Transitional". There are more than 68 errors, but some are duplicative. Is it because Microsoft doesn't know how to write HTML? They joined W3C. They know how. They don't care.

Here's another example, the report on another Microsoft page, one from their XML Development Center ("The language of information interchange"), with Ray Ozzie's name on it, on their RSS specification "Simple Sharing Extensions for RSS and OPML," at http://msdn.microsoft.com/xml/rss/sse/: "Failed validation, 97 errors. . . . This page is not valid HTML 4.0 Transitional." Well, W3C should know.

By the way Microsoft's copyrights in this specification were licensed with fanfare under a liberal Creative Commons license, which drew praise from some, but skepticism from others.

So, ironically enough, Microsoft presents information on its RSS "sharing extensions" using HTML that can't pass W3C's HTML validation. That worries some people, as do the "extensions" to RSS.

Why does it matter? What happens if your HTML doesn't validate properly? I'll let W3C explain:

What is Markup Validation?

Most pages on the World Wide Web are written in computer languages (such as HTML) that allow Web authors to structure text, add multimedia content, and specify what appearance, or style, the result should have.

As for every language, these have their own grammar, vocabulary and syntax, and every document written with these computer languages are supposed to follow these rules. The (X)HTML languages, for all versions up to XHTML 1.1, are using machine-readable grammars called DTDs, a mechanism inherited from SGML.

However, just as texts in a natural language can include spelling or grammar errors, documents using Markup languages may (for various reasons) not be following these rules. The process of verifying whether a document actually follows the rules for the language(s) it uses is called validation, and the tool used for that is a validator. A document that passes this process with success is called valid.

With these concepts in mind, we can define "markup validation" as the process of checking a Web document against the grammar (generally a DTD) it claims to be using....

Is validity the same thing as conformance?

No, they are different concepts.

Markup languages are defined in technical specifications, which generally include a formal grammar. A document is valid when it is correctly written in accordance to the formal grammar, whereas conformance relates to the specification itself. The two might be equivalent, but in most cases, some conformance requirements can not be expressed in the grammar, making validity only a part of the conformance....

Why should I validate my HTML pages?

One of the important maxims of computer programming is: Be conservative in what you produce; be liberal in what you accept.

Browsers follow the second half of this maxim by accepting Web pages and trying to display them even if they're not legal HTML. Usually this means that the browser will try to make educated guesses about what you probably meant. The problem is that different browsers (or even different versions of the same browser) will make different guesses about the same illegal construct; worse, if your HTML is really pathological, the browser could get hopelessly confused and produce a mangled mess, or even crash.

That is, of course, what happens to browsers that compete with Microsoft when they try to display pages written in Microsoft's proprietarily-twisted HTML. Users who don't know that it is Microsoft's HTML causing the problem are likely to think it's their browswer's fault, and switch to IE, imagining it's "better". You think it is deliberate?

Or remember what happened to WordPerfect? You can read about it in the current Novell v. Microsoft antitrust litigation. Here's Novell's complaint, and here's just a small part of what they are alleging:

55. A top Microsoft executive wrote that Microsoft should "smile" at Novell, falsely signifying Microsoft's willingness to help the two companies' common customers integrate their various products, while actuaIly "pulling the trigger" and killing Novell. Indeed, Microsoft's Chairman and CEO, Bill Gates, instructed his executives to develop plans to retaliate against Novell for its cooperation with the government authorities investigating Microsoft. As explained below, Microsoft fulfilled these instructions by withholding technical information about the ever-changing functions of Windows, including the integrated browsing functions in Windows 95, and by excluding Novell's office productivity applications from the major channels of distribution and other potential platforms....

71. During the development of Windows 95, Microsoft's executives schemed to integrate the browsing functions into Windows 95 in a manner designed to cause the maximum possible damage to competitors. ... For instance, Microsoft intentionally made the use of any browsing technology other than Microsoft's browser a "jolting experience" for its own Windows customers, solely to create the false impression that other browsers were not effective. ...

72. As a result of Microsoft's integration of the browsing functions into Windows, ISVs needed documentation of the browsing extensions to design their applications to perform the most basic file management functions. Microsoft initially documented the browsing extensions in the beta releases of Windows 95 and otherwise appeared to cooperate with ISVs in developing applications for release with Windows 95....

73. Microsoft "evangelized" the benefits of using the browsing extensions. In the early stages of developing WordPerfect for Windows 95, Novell thus devoted significant resources to ensuring compatibility with and otherwise exploiting the benefits of Windows' integrated browsing functions. Further, as encouraged by Microsoft, Novell expended additional resources to expand upon the extensions, providing still greater functionality for its own customers and potentially for other ISVs and their customers. ....

74. In an e-mail dated October 3, 1994, however, Bill Gates ordered his top executives to retract the documentation of the browsing extensions, but only until Microsoft's own developers of the Office suite of applications had sufficient time to work with the hidden extensions to build an insurmountable advantage over competitors such as WordPerfect. Gates further explained that without this advantage, Office could not compete with the major ISVs.

75. In public test versions of Windows 95 released a few months before the final product shipped to consumers, ripped out these programming interfaces without warning to Novell. After Microsoft withdrew the documentation of the browsing extensions, Novell was suddenly unable to provide basic file management functions in WordPerfect; in many instances, a user literally could not open a document he previously created and saved. Indeed, WordPerfect could no longer use the functions that Novell had innovated atop the extensions, while Microsoft Word could still take advantage of such innovations.

76. When Novell asked Microsoft why it removed the Explorer interfaces and browsing extensions, Microsoft claimed that it did not have the time and resources to complete their development. But in fact, the Explorer interfaces and browsing extensions had been complete and functional before Microsoft removed them. ...

77. Thereafter, when Microsoft released Windows 95 and Office 95, at virtually the same time, Microsoft suddenly reversed course and documented the programming interfaces. Doing so voided the alternatives that Microsoft previously forced Novell to expend an entire year developing and, at the precise moment when WordPerfect needed to enter the market, forced Novell to spend additional time designing basic functions of WordPerfect all over again. . . .

83. In addition to withholding technical information, Microsoft created and controlled new "industry" standards and established unjustified certification requirements to delay the release of Novell's applications and to impair their performance for Novell's customers.

84. First, as discussed above, Microsoft excluded from the markets the "OpenDoc" technology for sharing information among applications, by using its monopoly power to force a different standard upon the industry. . . .

85. Microsoft responded to this competitive threat by preventing CIL from making OpenDoc compatible with Windows 95. For example, Microsoft routinely required all ISVs to execute nondisclosure agreements as a condition of receiving the information they needed to develop their applications. These agreements, however, contained terms that uniquely targeted ISVs who were members of CIL, by preventing their employees who worked on OpenDoc from receiving Windows 95 betas or specifications, which effectively prevented CIL from initially developing OpenDoc for Windows 95. In addition, Microsoft required ISVs working with a Windows 95 beta to agree that they would not work on OpenDoc for two years. While Microsoft eventually dropped this requirement, its impact had immediate anticompetitive effects on OpenDoc's development.

86. Further, Microsoft unilaterally announced that OLE would be incorporated directly into Windows, instead of existing independently of the operating system as a technology to be adopted or rejected by ISVs, depending on their assessments of its technical merit. Microsoft then required OLE-compatibility as a condition of Microsoft's certification of an application's compatibility with Windows 95. This certification requirement was a significant barrier to entry into the applications markets, because Microsoft represented to the industry that any application lacking the certification could not be trusted to run on Windows 95. By exploiting this barrier to entry, Microsoft forced ISVs to make their applications OLE-compatible. Furthermore, Microsoft ensured that only applications using its tools, and not those of its competitors, would reach customers. This anticompetitive behavior by Microsoft is similar to the behavior described in the Government Suit with respect to Microsoft's efforts to force ISVs to use Microsoft's implementation of Java. "Specifically, in the First Wave agreements that it signed with dozens of ISVs in 1997 and 1998, Microsoft conditioned early Windows 98 and Windows NT betas, other technical information, and the right to use certain Microsoft seals of approval on the agreement of those ISVs to use Microsoft's version of the Windows [Java virtual machines] as the 'default.'" Findings of Fact ¶401.

87. There was no valid technical or business reason for requiring OLE- compatibility as a condition of the Windows 95 certification; OpenDoc was even more capable of providing the same linking and embedding functions, and in the absence of the certification requirement and other anticompetitive acts, OpenDoc and OLE would have continued to compete on their technical merits. Indeed, Microsoft initially announced that applications using OpenDoc would be deemed OLE-compatible, and would receive Microsoft's certification for Windows 95, because OpenDoc was a "superset" of OLE, meaning it provided every function of OLE, and more. Later, after Novell, other ISVs and CIL were far advanced in their efforts to develop and use OpenDoc, Microsoft announced that applications using OpenDoc would not receive automatic certification, and might not receive certification at all.

88. Seeing that Microsoft's anticompetitive acts would ensure the demise of OpenDoc, ISVs were left with no choice but to adopt Microsoft's proprietary OLE protocol as the de facto industry standard for linking and embedding. Even after making OLE the industry standard, however, Microsoft still withheld specifications and final, debugged versions of OLE until after Microsoft released its competing applications. Microsoft's anticompetitive acts concerning OLE further increased the "time-to-market" lead that Microsoft's office productivity applications unlawfully achieved over Novell's applications.

89. Second, Microsoft required office productivity applications seeking Windows 95 certification to be compatible with the very different Windows NT, which is an operating system for larger and more powerful computers that are used as "servers" to link numerous PCs (and peripherals) across an organization into a network. There was no justification for this requirement. Further, Windows 95 and Windows NT were so dissimilar that an application running on one system could not run on the other without substantial modification. Novell expended significant development resources to make its applications compatible with Windows NT, resulting in further delay in the release of Novell's applications for Windows 95.

90. Third, Microsoft unilaterally made the proprietary Rich Text Format ("RTF") of Microsoft Word the standard file format for text-based documents in applications developed for Windows. Upon capturing the standard, Microsoft strategically withheld the specification to injure competitors, including Novell.

So, as you can see, any announcement from Microsoft about standards comes in a context. Do you see why some are saying that there must be no proprietary extenstions in Microsoft's XML?

Or let's go back in time a bit and read the Caldera Statement of Facts in the DR DOS case, the section called FUD Drip Feed. That case settled prior to a court ruling, so keep that in mind as you read, but it was settled with money from Microsoft going to Caldera. A lot of money. You can read in the document numerous allegations of tricks Microsoft employed to make sure its competition was always a dollar short and a day late, such as by not providing them betas to work with in a timely manner. And here's one of my favorite paragraphs:

32. The "death spiral" is somewhat of a term of art at Microsoft. On October 18, 1991, Mike Maples enquired of several executives: "I would like to ask you to invest half a day with me following Comdex. What I would like to brainstorm is how to push Excel over the top and Lotus out of business." To which Silverberg replied: "I'd be glad to help tilt lotus into the death spiral. I could do it friday afternoon but not saturday."... At a management conference in June 1992, one of the "6 Core Strategies to build share" included "Drive competitors into a death spiral," complete with objectives and tactics. ... Ironically, other discussion there focused on "Our Image" and how to overcome the industry perception of "Microslop," "Microshaft," and "Microsleaze."

Microsoft is not known for peaceful coexistence. So, you will have to forgive us if some of us are allergic to Microsoft promises of cooperation and openness. There is some history here.

Let's end with Tim Bray, who asks the obvious question ("Why Should There Be Two?") and has a very sensible suggestion:

The ideal outcome would be a common shared office-XML dialect for the basics—and it should be ODF (or a subset), since that’s been designed and debugged—then another extended vocabulary to support Microsoft features , whether they’re cool new whizzy features or mouldy old legacy features (XML Namespaces are designed to support exactly this kind of thing). That way, if you stayed with the basic stuff you’d never need to worry about software lock-in; the difference between portable and proprietary would be crystal-clear. And, for the basic stuff that everybody uses, there’d be only one set of tags.

This outcome is technically feasible. Who could possibly be against it?