Groklaw - Lessons on Data Preservation From the Audio Industry

	When you want to know more...

Home
Archives
Site Map
Search
About Groklaw
Awards
Legal Research
Timelines

ApplevSamsung
ApplevSamsung p.2
ArchiveExplorer
Autozone
Bilski
Cases
Cast: Lawyers
Comes v. MS
Contracts/Documents
Courts
DRM
Gordon v MS
GPL
Grokdoc
HTML How To
IPI v RH
IV v. Google
Legal Docs
Lodsys
MS Litigations
MSvB&N
News Picks
Novell v. MS
Novell-MS Deal
ODF/OOXML
OOXML Appeals
OraclevGoogle
Patents
ProjectMonterey
Psystar
Quote Database
Red Hat v SCO
Salus Book
SCEA v Hotz
SCO Appeals
SCO Bankruptcy
SCO Financials
SCO Overview
SCO v IBM
SCO v Novell
SCO:Soup2Nuts
SCOsource
Sean Daly
Software Patents
Switch to Linux
Transcripts
Unix Books

Gear

Groklaw Gear

You won't find me on Facebook

Donate

No Legal Advice

The information on Groklaw is not intended to constitute legal advice. While Mark is a lawyer and he has asked other lawyers and law students to contribute articles, all of these articles are offered to help educate, not to provide specific legal advice. They are not your lawyers.

Here's Groklaw's comments policy.

What's New

STORIES
No new stories

COMMENTS last 48 hrs
No new comments

Sponsors

Hosting:

On servers donated to ibiblio by AMD.

Webmaster

Lessons on Data Preservation From the Audio Industry

Friday, March 17 2006 @ 01:22 AM EST

One of Groklaw's resources is people. Our readers work in a broad variety of professions. I got an email from a guy who is an audio specialist, and when he read about the ODF story, about the need for longterm storage of documents, it immediately resonated with him, because part of his job for many years has been preserving audio. Here's a bit of that email:

In the past I have seen what happens as audio formats come and go, and how difficult it can sometimes be to get audio from older formats. The lesson that has been learned for both audio and video is that if you want be able to get your material back in the future you have to be able to maintain a viable playback system.
Many archives have stocks of older obsolete equipement (particularly for video formats) that are cannibalized to keep their main playback machines going. Digitization has not really been an option for much of this material as it is still not possible to fully digitize absolutely everything that is captured on audio tape or on film. It is getting closer though.
I see the same problem with the documents that get saved with the audio or video. How may of these were saved on old 5 1/4 inch floppy discs and filed with the audio tapes? It is pretty hard to find one of those in a PC nowadays.
The same could be said for software. You have some stuff in an old version of a word processor from the old Commodore computer. Not only can you not read the 5 1/4 disc, but even if you could you cannot read the proprietary binary format file with the information you want.
And that has only happened in my working life (and still have a good 20 years to go).
The risk in storing information in undocumented binary formats is that there really is no guarantee that a) the hardware will still be around; b) the software can be found -- a) because hardly anyone keeps old hardware, and b) because many licences do not allow you to run old versions of the software you have upgraded. On that point, I recall watching boxes of old 'obsolete' software being dumped some years ago.
So you can see that from my perspective as an audio specialist, storing things in formats where the hardware or software to access the material may not be available or may be uneconomic to maintain is a risky proposition.

I asked if he'd be willing to elaborate for Groklaw, and he graciously agreed.

************************************

Lessons Learned From the Audio Industry

~ by The Sound Man

My world is audio, and I have worked in it for 25 years. The audio industry's experience with audio formats may, I believe, serve as an object lesson for people who are undecided about privately documented storage formats.

From the invention of the phonograph in 1877 the audio industry has had publicly documented standards. As a result it has always been possible to play early domestic 78s on any model of gramophone since (although not all companies used quite the same speed or same groove pitch). Modern disc lathes use the same physical principles to create LPs and 45s. You can play virtually any 78, 45 or 33rpm discs on just about any turntable that has the correct speed settings and the right stylus for the size of the groove. Finding a turntable might be a problem though -- I have 78s that I inherited from my father which I cannot play, but it is easy to find information on how to play these discs.

The evolution of professional formats has been similar. It is possible, to take early tape recordings (made in the 40s and 50s) and play them back on today's equipment (with some electrical adjustments) and hear pretty much what was recorded back then. Reel to Reel tape recorders are getting harder to buy too, and I think only one or two companies make them now. They all used the same standards for speed and tape width.

As a result of easy access to past formats the CD market has been flooded with compilations of classic material recorded on both 78 and early tape.

The market for professional recorders is now full of computer based solutions, almost all of them are closed source applications. The first system I used was ProTools on the Macintosh IIfx (just like the one Douglas Adams owned). The inputs to the system were via balanced audio or AES (Audio Engineering Society) digital, both publicly documented AES standards. The file format used for audio was AIFF, another publicly documented standard.

Today I use SADiE on a PC running Windows. It uses the same AES standards for input and output as well as MADI, another public standard. It uses WAVE format for files on the PC platform, but it can read and write AIFF (even off Mac discs mounted on a PC), and some private formats too, all in the name of allowing people to exchange audio and use their tool of choice.

Even the most simple free software editors use these same formats. Open Source audio systems use the same standards. A person with a home studio can send me a WAV or AIFF file made on any system they choose for mastering on our SADiE system. It reads both formats. I can send either format to the record company rep (using a Mac) for approval. The same software is not needed.

There are now portable recorders based on computer technology. I can buy a compact flash card from anyone, record onto it in any recorder that uses that type of media, plug the card into any computer with a card reader, and import the audio files into virtually any audio application you care to name.

It is interesting that standards were often not kept a secret - they were submitted to standards bodies for everyone to benefit from.

Different companies have even agreed on standards to exchange edit decision list information. This allows you to complete work on one system, and import both the audio and the bulk of your editing work onto another.

When There Is No Happy Ending

Not all professional audio stories have a happy ending though.

DAT (Digital Audio Tape) was widely adopted by the professional community in the early 1990s, after failing as a domestic format (probably due to the Audio Home Recording Act of 1992).

They were very cheap compared with the professional machines in use at the time, and companies like Sony and Tascam turned their efforts to making 'professional' DAT units. Many of the original proprietary (one format per vendor) recording systems fell into disuse and were dropped, both by manufacturers and studios.

There were standards for the recording format and the tape, and it was possible (unless someone's machine was out of alignment) to interchange tapes and machines regardless of who made them.

Here begins the strife. Sony announced in November 2005 that they would stop shipping DAT machines at the end of that year. DAT is now obsolete. Cheaper and better ways had been found to do the job, ways that improve the flow of material through the use of open standards. Unlike records, where the format was around for a very long time and was adopted by all, the DAT was only adopted by a small section of the community, and now you cannot buy new machines.

In order to play back DATs from our archives, we purchased a few of the last machines. We have to maintain these as long as we still have DATs to transfer to some other format. It might be hard in the future to make a DAT machine, but at least there is documentation so it could be done. So here we have a recent, openly documented standard that is now defunct, and in a short time many folks will no longer be able to access the content on those tapes because they no longer possess the hardware to play them or have the ability to make the hardware to play them.

Strangely, DAT has another life for data backup, using the ISO DDS public standard.

I should note that most DAT audio players can output a signal that complies with an AES standard, and this can be plugged into most digital mixers or recorded back into virtually any piece of gear today, whether it be a high end professional (proprietary) hard disk recorder, or a Pentium 4 in someone's bedroom with a semi-pro sound card.

In my world standards have allowed professionals to be creative using the the tools they choose, to share material, and to pass work from person to person without much fuss. Manufacturers have worked together to allow this. The audio community has demanded it.

No one wants to be incompatible in our industry. Investments in existing equipment can run into millions of dollars. Who would dare use a non standard format? Such a machine would be of no use in our studio. It would not work with the equipment we already have.

I should note that there are exceptions where the product is in a small niche, or the functionality provided is unique. Sometimes competitors adopt the standard set by the first to market. These standards are usually made available to all. If the industry finds the new standards useful then everyone wins. One good example is the ADAT lightpipe optical format for transferring multiple channels of digital audio. Developed by Alesis, it turned out to be such a useful format that many other manufacturers used it as a secondary or even primary format on their hardware.

I can safely say that many audio professionals would rather own a system that supported publicly documented standards. Companies can come and go, software applications can change with the passage of time and fashion. But the work created on these systems must remain. It is a creative legacy for the future. The future livelihood of thousands of companies depends on continued easy access to material produced today and in the past. As mentioned already, there is a thriving market for re-releases of old recordings.

Thinking back to the DAT experience, I have audio files that were recorded 10 years ago on a Macintosh. They have been moved between systems and storage formats many times. Over the years I have been able to open the audio files in many different applications, some of which are now defunct. I can copy the files as many times as a want and send it to whoever I want.

What about the software we use to store information? These days most audio ends up on a hard disc or data tape of some sort. I like to store any documentation with the audio. I used to print it out and put it in the tape box. Anyone can access a printed page (I was reminded of this after seeing in London an exhibition of fragments of the Bible dating from the first few centuries after Christ's death. Anyone can still read them.)

If I use DLT I know I can get the audio back for two generations of new hardware. I can decide to use this format or choose another. The manufacturer tells me in advance what I need to know.

What About Our Documents?

But what about my documents? The word processor I use saves files in a binary format which is not publicly documented. It was not always the dominant software in this class (remember Wordperfect?). It may not be in the future. What will be the future, like the DAT example above, of files saved in these formats when they become obsolete? Someone could build a DAT player in 10 years time given time and money -- they just follow the standard, but what about an obsolete, privately documented file format?

I have on my hard disk archived files created in Lotus 123. I can open them today, but for how much longer? I could convert them of course, but what if they were filed away on a 5 ¼ floppy in our basement, long forgotten, and then suddenly were needed?

I've said that it can be impractical to recover audio from obsolete formats, even with public documentation of how to do so. In the future it may be impractical to recover documents as well. It is just a matter of time. Imagine if all the literature of antiquity was encrypted in some way.

Privately documented standards put my creative work at risk. They could stop me retrieving work from the past, they limit my creativity now, they limit my freedom to work or collaborate with others (or make it harder), and they put at risk the recovery of my work in the future. They lock me in to software and hardware that may not be appropriate next week, let alone in five years.

As well as helping industry, standards also benefit the market because they provide security to the end consumer. A good example of consumer security is the Compact Disc.

Red Book audio CDs all use the same format (developed by Philips and Sony). Any disc that meets the Red Book standard will play in any player that also meets the standard. I can buy a disc in London that was made in Europe, and play it back in my home country on a player made in China. It just works. Some copy protected discs will not play in some new gear. They do not meet the standard. I avoid these if I can.

As an aside, current copy protection stops people ripping CDs at faster than real time using a computer. However the protection schemes only address today's technology -- they do not stop people playing CDs in a normal player and re-recording them in real time to another format, which is just how it used to be done in the days of LPs and cassettes.

Every DVD player on the planet (as far as I am aware) will play Red Book compact discs, and so will nearly every PC with a CD-ROM drive. My Sony DVD player bought last year will play a CD I bought in 1985 (Dire Straits' Brothers in Arms). My new laptop's DVD writer will play it too.

I, as a consumer, can buy an audio CD secure in the knowledge that it won't be obsolete in the near future, and that my investment is protected.

In my professional world, open standards ensure that I am free to provide creative services to my clients, and collaborate with anyone I choose.

What are you free to do in your world? Today? Tomorrow? For how long?

Please think about it.

Note: Links to commercial sites in this article and references to specific products and manufacturers are for illustrative purposes only, and are not an indication of endorsement of these products or sites.

Lessons on Data Preservation From the Audio Industry | 184 comments | Create New Account

Comments belong to whoever posts them. Please notify us of inappropriate comments.

Corrections Here!

Authored by: Anonymous on Friday, March 17 2006 @ 01:31 AM EST

Not that there are ever very many.