Groklaw reported that SCO knew JFS was in UnitedLinux, quoting the whitepaper and providing a link, back in November of 2003 in an article we did debunking SCO's claims about the four main high-end things SCO was squawking about at the time, here, and I linked to that article twice in the last week or so in other articles. Today, it was also in part posted to the Yahoo! SCOX Message Board at 12:30 PM. By 2:32 PM, the whitepaper had disappeared from SCO's website.

So I happen to have the whitepaper, and since it is now among the "Disappearing Evidence," as I like to call it, I have taken a screenshot of that section, as well as the sections on I/O and hyperthreading, and the introduction, which tells us that all the participants in UnitedLinux used the same base software. As you will no doubt recall, SCO has maintained, in its Second Amended Complaint, beginning with paragraph 99, that IBM breached its contract by putting JFS from AIX into Linux.

Here's what we published in November of 2003, from the whitepaper, which is titled simply "UnitedLinux" and is dated November 7, 2002:

http://www.caldera.com/images/pdf/scolinux/UnitedLinux_whitepaper.pdf (p. 13-15) [update: now here instead]: "The Journaled File System (JFS) is a full 64-bit file system. All of the appropriate file system structure fields are 64-bits in size. This allows JFS to support both large files and partitions. JFS was developed by IBM under the GPL license and is ported from its AIX systems. JFS provides a log-based, byte-level file system that was developed for transaction-oriented, high performance systems. Scalable and robust, its advantage over non-journaled file systems is its quick restart capability. JFS can restore a file system to a consistent state in a matter of seconds or minutes.”)

There isn't much use in making the link clickable now. I understand you get a Document Not Found page. I also notice that the Groklaw page from November 2003 is hard to read, because I didn't know how to do HTML back then. Groklaw members taught me bit by bit. So, I'll put the information after the graphics.

UPDATE: Thanks to Groklaw's readers, here's an even earlier version [PDF] of the paper, dated May of 2002, which as you can see is still on the Internet. And here's a paper all about SCO's version of UnitedLinux, still on IBM's site, written by John Terpstra in November of 2002. Terpstra was also a Caldera employee until around the time Darl McBride came aboard in July of 2002. Here's what he wrote about JFS and NUMA and all the rest of the features in the kernel SCO used in its version of UnitedLinux:

Linux kernel

The kernel is based on linux-2.4.19 with enterprise features enabled. The facilities and capabilities of the kernel include:

* File systems: Ext2, Ext3, JFS, Reiserfs, Logical Volume Manager (LVM); note that the kernel had support for POSIX ACLs as well as Extended Attributes
* I/O: raw I/O and asynchronous I/O
* Execution: Next Generation POSIX Threading (NGPT is a derivative of pThreads), HyperThreading
* Memory: NUMA, Memory Extension Technology (MXT), Large Memory Support (64GB physical RAM)
* SAN support: iSCSI
* Modern hardware support: ACPI
* SNMP/CIM support
* Protocols: IPv6

The UnitedLinux base system includes support for High Availability (HA).

As you can see, efficiency probably requires SCO to sue itself for all the high-end features it either put into Linux or distributed all over the world in its UL distro. Under the GPL. And now, for the proof of this November 2002 disappeared paper that used to be on SCO's website:

****************************

******************************

Republishing the November 2003 Groklaw article, "Cross Your Heart and Hope to Die, SCO?" in a more readable form -- how Groklaw got so popular, considering my flaws is a true mystery:

In its Supplemental Responses to IBM's Second Interrogatories and Second Requests for Documents, SCO gave this answer:

Insofar as this interrogatory seeks information as to whether plaintiff has ever distributed the code in question or otherwise made it available to the public, SCO has never authorized, approved or knowingly released any part of the subject code that contains or may contain its confidential and proprietary information and/or trade secrets for inclusion in any Linux kernel or as part of any Linux distribution.

Cross your heart and hope to die, SCO? Or cross your fingers behind your back? Let's see what the evidence shows.

SCO has specifically mentioned the following four as being code at issue in this case: JFS, NUMA, RCU, and SMP, and while it is conceivable that the "subject code" they are talking about in this response to IBM's interrogatory is referring to some other code, it seems reasonable to look at the code they have mentioned publicly. Actually, it's more than reasonable. It's our only choice, until they tell us exactly what code they are complaining about with specificity. Is it true that they never "authorized, approved or knowingly released" any of this code for inclusion in any Linux kernel or as part of any Linux distribution?

Let's start with JFS. In the case of JFS, they not only distributed Linux with JFS, one of Caldera's employees, Christoph Hellwig, contributed code to JFS, as Groklaw reported on July 18. Here is a snip from that article:

Here is an email in which he tells an inquirer how to contribute to JFS, including this tidbit: "I've run native sysvfs tools under linux, but as now that I'm Linux sysvfs maintainer I'm looking into implementing free versions of it. . . . The JFS/Linux core team has setup a CVS commitinfo, but currently I'm the only one who receives it."

And here he encourages someone to donate to the main JFS repository at IBM and talks about his role:

I'm one of the main commiters to JFS outside IBM and I'm really happy to see more people involved :)

First I'd like to encourage you to contribute your userspace changes to the main JFS repository at IBM. For the 1.0.11 release I have added autoconf/automake support to easify portability and a bunch of portablity patches (mostly getting rid of linuxisms) is under way to the Core team.

He also posts to the freebsd list as freebsd-fs at freebsd.org. Here is the press release when SCO in 2002 released "SCO Linux Server 4.0 for the Itanium (R) Processor Family" and which mentions that the product is based on United Linux. This SCO page lists JFS as one of its features. . . .

They are complaining that IBM contributed JFS to Linux, but their own employee, from this evidence, was involved in helping out. On the day IBM announced JFS was being given to Linux, Hellwig is listed as making five contributions to the kernel.

And he is listed on this page of JFS contributors. Here is IBM's page on Who Is Using JFS? and it lists United Linux. So they not only released a distro with JFS in it under the GPL, their employee helped make it happen. Here is a page listing the Skunkworks team, and you will note that the first entry is Ronald Joe Record, a SCO employee (rr@sco.com), and JFS is listed for him. But in case you aren't yet convinced, here is a handy list for you of some other pages that mention JFS, collected by Rand McNatt and nw:

http://uk.sco.com/events/Partner_Briefings/March_2003/sco_os_update.ppt (mentions NUMA also)

http://caldera.com/developers/training/unitedlinux/Developer_web_2B-3.ppt

http://www.caldera.com/unitedlinux/info/unitedlinuxwhitepaper.pdf (JFS chapter on page 15-16.)

http://www.caldera.com/images/pdf/scooffice/SettingUpSCOofficeMailServeronSCOLinux.pdf (P. 8: "SCO Linux gives you a choice of four journaling filesystems, EXT3, Reiser, XFS and JFS.")

http://www.caldera.com/skunkware/COMPONENTS.html

http://www.caldera.com/images/pdf/scolinux/UnitedLinux_whitepaper.pdf (p. 13-15: "“The Journaled File System (JFS) is a full 64-bit file system. All of the appropriate file system structure fields are 64-bits in size. This allows JFS to support both large files and partitions. JFS was developed by IBM under the GPL license and is ported from its AIX systems. JFS provides a log-based, byte-level file system that was developed for transaction-oriented, high performance systems. Scalable and robust, its advantage over non-journaled file systems is its quick restart capability. JFS can restore a file system to a consistent state in a matter of seconds or minutes.”)

http://www.caldera.com/products/scolinuxserveripf/features.html ("Journaling file systems add a higher level of reliability and faster recovery time. JFS, ReiserFS, XFS and Ext3 journaling file systems are included with SCO Linux Server. Each of these file systems has been tested and optimised for the best performance and stability.")

Here is another one, which mentions both JFS and XFS:

SCO Linux Server 4.0 (Powered by UnitedLinux 1.0) . . . . The SCO Linux Server 4.0 has many advanced enterprise filesystems, including ext2, ext3, reiserfs, jfs, and xfs, which of these have support for ACLs on SCO Linux 4.0?

What about SMP? Hellwig worked on SMP also. Alex Roston has researched Hellwig and SMP and contributes this:

Christoph Hellwig is also a contributor to Linux's SMP systems. Here he proposes a patch that fixes the booting of an "SMP-compiled i386 kernel on a SMP-capable motherboard with" a CPU which is not SMP.

Linus replies to him with this note.

So Hellwig patches his patch.

Later, a SCO employee, Senior Programmer Torsten Duwe, (duwe@caldera.de) acknowledges Hellwig's code. He also discusses the issues around binding a process to a CPU. (Process affinity) This is obviously only an issue on machines with more than one CPU, otherwise the process is already bound to the single CPU on the system. Here he posts a piece of code by Nick Pollit from SGI which allows the 2.4 kernel to perform something called "process pinning." Later on in the same discussion he discusses a Red Hat patch for the SMP code.

On another occasion he discusses SMP support for Pentium-3 chips and suggests that given the number of different architectures Linux supports that SMP should be either per-architecture or a configuration option. I've found several other references to SMP and Christoph Hellwig, but only one other seems significant. In this exchange he proposes a change to Linux's gendisk handling.

Red Hat's Arjan van de Ven notes that the patch is SMP unsafe and Hellwig agrees that he will fix the SMP problem if the patch is accepted, and if you follow the thread, it goes on a little longer with a discussion of whether Hellwig's code was inspired by someone else, and Hellwig acknowleges that he's rewriting another programmer's contribution.

In addition to his email address hch at ns.caldera.de, Christoph Hellwig also has another email address which he frequently uses for his Linux work. This address is hch at infradead.org. Here we see him making a reply to this post which discusses an SMP machine with problems, with a suggestion for either clearing up or diagnosing the problem.

"In a thread called "Re: Longstanding networking / SMP issue?" Hellwig acknowledges some buggy code. In this post he makes a suggestion for someone having trouble with XFS on an SMP machine.

Looking at all this, we see that Hellwig didn't just post a patch because he was having a problem with SMP. He's performing a large number of programming tasks involving this issue. He's creating SMP code, forwarding SMP code written by others, rewriting code belonging to others that impinges on SMP, and acknowledging buggy SMP code. In addition, he's interacting with other people at SCO. Hellwig's own SMP patch is acknowleged by another SCO programmer and a former SCO employee, Tigran A. Aivazian, who is acknowleged as a contributor to the Linux SMP HOWTO, although whether Mr. Aivazian was involved in the HOWTO while employed by SCO is unknown.

Further, we also see Hellwig involving himself in technical discussions revolving around SMP, and he also makes diagnoses and suggests workarounds for SMP code. Lastly, he also shows an awareness that his code needs to take SMP issues into account. He's also listed as the active maintainer of two other file systems under Linux, the FREEVXFS and SYSV filesystems.

So it should come as no surprise that he's made many contributions to another important Linux filesystem, the XFS filesystem contributed to the Linux kernel by those nice folks at SGI. While he's not listed as the filesystem's maintainer, he's so important to this effort that SGI has given him two more email addresses, 'hch at sgi dot com,' and 'hch at lab343.munich.sgi dot com.' He also has the authority to merge code into this subsystem, as you can see here. The document at this URL is full of emails that read like this example:

Date: Tue Jan 14 07:19:17 PST 2003
Workarea: lab343.munich.sgi.com:/home/hch/repo/slinx/2.5.x-xfs
Author: cattelan
Merged by: hch
Merged mods: 2.4.x-xfs:slinx:136126a
The following file(s) were checked into:
bonnie.engr.sgi.com:/isms/slinx/2.5.x-xfs"

In other words, on January 14th of 2003, Christoph Hellwig merged an XFS patch by Russell Cattelan into the 'lab343.munich.sgi.com:/home/hch/repo/slinx/2.5.x-xfs' work area. You'll also note that he's merging kernel 2.4 code into the 2.5 kernel.

Here's another document where he's working with the XFS code. Here he writes about checking files into 'bonnie.engr.sgi.com:/isms/slinx/2.4.x-xfs' .

He also acts like a teacher/boss. Note his comments in the following documents. Here he replies to Andrew Morton with a suggestion for making his XFS patch better and Mr. Morton replies.

Here are more URLs listing XFS commits. In the first URL he commits filesystem code from three different email addresses:

http://source.mvista.com/pipermail/linuxppc-commit/2002-October/002793.html

http://ftp.club.cc.cmu.edu/pub/linux/kernel/v2.5/snapshots/patch-2.5.58-bk1.log

http://www.kernel.org/pub/linux/kernel/v2.5/snapshots/old/patch-2.5.63-bk2.log

http://www.club.cc.cmu.edu/pub/linux/kernel/v2.5/testing/cset/cset-1.1039.1.1.txt

His work rates frequent mention in the 2.5.* kernel release notes. Once again, it happens under three different email addresses. Note how much of this work involves the XFS filesystem, here and here.

Here Hellwig deals with an XFS bug. And here he's listed in connection with several XFS bugs. See number 452, 835, 840, 861, and 870. In other words, he was deeply involved with porting the XFS file system from SGI into Linux, and he performed all of a chief programmer's usual functions - coding, debugging, merging patches and codebases, supervising, and instructing.

"Worse for SCO, in addition to Hellwig's work on XFS and SMP, we also see from this post that Hellwig's supervisors, in this case Ralf Flaxa, (rf@caldera.de) are aware that he is online and making changes to the Linux kernel:

Reading this thread and previous requests I see a need by ISVs and applications to determine at runtime certain processor related features. They can roughly be grouped into:

1. number-crunching power in general

o number of CPUs
o CPU family and type
o BogoMips (per CPU and/or overall)
o SMP capabilities (here kernel scalability could be an item)

2. features specific to a processor

o for x86 e.g. the "flags" and *_bug fields
o info about FPU, MMU and the like

I have asked Christoph to work on a proposal and present it to this list. I think we agree that there is a need and demand for a standard way to gather this info and that /proc/cpuinfo was just a bad start - so I would like to give it another chance. This is Ralf speaking as director of Linux development at Caldera and as technical lead for the LSB sample implementation.

You might also notice the kernel version numbers. Where SMP is concerned, he's not just talking about code in 2.2. He (or his code) discusses 2.4 (though I'm not sure which release he's discussing) and 2.4.10. Lastly, most of the posts are dated after the January 5th, 2001 release of the 2.4 kernel, so Hellwig not only contributed to SMP, his contributions involved 2.4. One of them is even dated July 13, 2003, long after SCO filed suit against IBM. The XFS code is much the same, though most of it deals with the 2.5 codebase.

Mr. Hellwig has been at the heart of three important initiatives aimed at making Linux "enterprise ready," JFS, SMP, and XFS.

What about NUMA? As it happens, SCO Linux 4 lists it as one of its features, along with JFS and SMP:

Features of SCO Linux 4 include:

Linux 2.4.19 Kernel – The core of SCO Linux Server 4.0 is the 2.4.19 Linux kernel. New features include broadened USB support, Logical Volume Manager, improved journaling file system support, POSIX-ACLS, new O(1) scheduler (improves SMP support), Asynchronous I/O, Enterprise Volume Management System (EVMS), PCI Hot Plug Support on supported hardware, NUMA support, and many other performance enhancing capabilities.

And this white paper lists NUMA on page 10. Caldera should have known about NUMA in Linux: here is a description of ccNUMA in Caldera's OpenLinux Documentation.

Well, that's three down and only one to go, RCU. I had more trouble finding proof that SCO distributed a kernel with RCU. But, thanks to the Groklaw community, and especially nw, we got that one done eventually. Here's a guy who reports on downloading the 2.4.19 kernel from SCO's ftp site and finding RCU patches:

The SCO RPMs for the 2.4.19 kernels do not contain the generic 2.4.19 GPL kernel source code (linux-2.4.19.tar.bz2). These RPMs contain the SCO GPL patches (some are which RCU patches to the kernel) and "specfiles" to create the SCO 2.4.19 GPL kernels from the generic 2.4.19 kernel source code. The reason the SCO RPMs do not contain the generic 2.4.19 GPL kernel source code (linux-2.4.19.tar.bz2) can be found in an email from a UnitedLinux developer at: http://marc.theaimsgroup.com/?l=linux-kernel&m=104218778023663&w=4

The generic 2.4.19 GPL kernel source code (linux-2.4.19.tar.bz2) can be found at: ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.19.tar.bz2 - Copy this file to the RPM source directory before building the SCO 2.4.19 GPL kernels.

Then, I couldn't seem to find any more on RCU, so I put out the call for folks to try to find a bit more, knowing there could be some skeptics in the crowd. Dr. Stupid, as always, came through with skill and charm. Here is his email:

Here's an article about SCO Linux 4, showing it's based on UL and thus SuSE.

Here's a SCO page talking about how to update SCO Linux 4, note that it gives the explicit kernel version - 'k_deflt-2.4.19-246.i586.rpm' Date is Feb 2003.

Google cache of above just in case.

Note that SCO did distribute the kernel updates themselves. Allegedly it's rather harder to get the kernel from them these days. But they talk of the "United Linux SP1 CD" which SuSE could send you if you ask them nicely. Here is a page which verifies that SCO was using the SuSE kernel (though he's referring to the pre-service pack 1 version)

And while this may have been posted before on Groklaw, it bears repeating in this context: Christoph Hellwig, nodding approvingly about RCU and helping out.

Also check here. That 'k_deflt-2.4.19-246.i586.rpm' was made available on many sites, such as here. Search on google for 'k_deflt-2.4.19-246.i586.rpm' for more)

Note that this mirror site has the kernel built for various processors (athlon, SMP etc) but they are all built from the same source RPM which is at the bottom... kernel-source-2.4.19.SuSE-152.i586.rpm . Yes folks, remember that all the United Linux chaps were shipping the exact same kernel: SuSE's kernel. But that kernel isn't the plain 2.4.19 kernel, it's patched (with 152 patches to be exact!)

So, I downloaded the source RPM and unpacked it, to see just what SCO was shipping, under the GPL, in its SCO Linux 4 product.

Well, well... in the 'kernel' folder of the source package there is a file which isn't in the vanilla 2.4.19 kernel. That file is called rcupdate.c and I'll give you a short excerpt:

/*
* Read-Copy Update mechanism for mutual exclusion
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
USA.
*
* Copyright (c) International Business Machines Corp., 2001
* Copyright (C) Andrea Arcangeli SuSE, 2001
*
* Author: Dipankar Sarma ,
* Andrea Arcangeli B^!

*
* Based on the original work by Paul McKenney
* and inputs from Andrea Arcangeli, Rusty Russell, Andi Kleen etc.
* Papers:
* http://www.rdrop.com/users/paulmck/paper/rclockpdcsproof.pdf
* http://lse.sourceforge.net/locking/rclock_OLS.2001.05.01c.sc.pdf
(OLS2001)
*
* For detailed explanation of Read-Copy Update mechanism see -
* http://lse.sourceforge.net/locking/rcupdate.html
*
*/

Leans back, lights cigar.... B^!

Oh, and btw, SCO could hardly have been ignorant about the enterprise level features going into UnitedLinux, as they crowed about them.

P.S. Moreover, the changelog for the kernel mentions rcu patches back in 2001, so the original kernel that SCO Linux 4 shipped with would have had RCU too.

So, there we have it. All four. And just in time for December 5, too. I do hope Judge Wells enjoys detail. While no one can yet know for sure what SCO meant in its response to the interrogatory, I think we can safely say that they cannot claim that they were unaware that they had released, and even were contributing toward, these enterprise-enabling features of Linux.