Kernel Traffic #205 For 14�Feb�2003

there's a race condition in filesystem

let's have a two inodes that are placed in the same buffer.

call fsync on inode 1
it goes down to ext2_update_inode [update == 1]
it calls ll_rw_block at the end
ll_rw_block starts to write buffer
ext2_update_inode waits on buffer

while the buffer is writing, another process calls fsync on inode 2
it goes again to ext2_update_inode
it calls ll_rw_block
ll_rw_block sees buffer locked and exits immediatelly
ext2_update_inode waits for buffer
the first write finished, ext2_update_inode exits and changes made by second proces to inode 2 ARE NOT WRITTEN TO DISK.

This bug causes that when you simultaneously fsync two inodes in the same buffer, only the first will be really written to disk.

It should, but it is a hazard. The problem is that every use of ll_rw_block has this bug, not only the one in ext2 fsync. The most clean thing would be to modify ll_rw_block to wait until buffer becomes unlocked, no one knows if it can produce some weird things.

Even Linus didn't know what he was doing, see this comment around the buggy part in 2.2, 2.0 and previous kernels.

ll_rw_blk.c:
        /* Uhhuh.. Nasty dead-lock possible here.. */
        if (buffer_locked(bh))
                return;
        /* Maybe the above fixes it, and maybe it doesn't boot. Life is interesting */
        lock_buffer(bh);

Nope, that's a _long_ time ago. It may well have been simply a MM bug that got fixed long since, ie something like

• dirty writeout happens
• writeout needs memory
• buffer.c tries to clean up pages
• hang.

Last week I finally got access to a decent (but old) technical specification for the ZT5550 redundant host controller. The document was published for the ZT5550C, but I am hoping that newer versions of the RHC just add more functionality to all the documented reserved bits in the document I am looking at.

The following patch adds a sysfs interface to most of the bits accessible via the indirect register (through the HCINDEX and HCDATA addresses in the Command and Status Register (CSR). The only bits I did not add access to were the ones that are cleared by reading. There are a lot of bits to get access to, which makes this patch a little bigger then I first expected, so I created a new config option so only people who actually want to mess with the RHC would pay for it.

Enabling this code will cause a new directory called zt5550_rhc to be created in the root of sysfs

This provides all kind of rope to hang yourself with, but it was fun messing with it. This also points out other areas where an interested party could further enable this particular board, for example adding the code to respond to a fault condition (after enabling fault handling through sysfs).

This patch was generated off of today's bk tree with the previously posted patch by Scott to fix the deadlock issue and the patch posted by Stanley to add sysfs support. Both of these patches are attached as a single patch to this email message.

For those wondering what's happening with klibc, here's an update...

I have it building relatively well within the kernel, and have modified the usr/gen_init_cpio.c file to add files to the cpio "blob". That all seems to work, but I don't seem to be able to extract the files properly (or at least that's what I'm guessing is happening).

If anyone wants to see the current progress, there's a big patch against 2.5.59 at: kernel.org/pub/linux/kernel/people/gregkh/klibc/klibc-2.5.59.patch.gz

and a bk tree with the different changes broken down into "logical" chunks at:

bk://kernel.bkbits.net/gregkh/linux/klibc-2.5

Any help with trying to debug init/initramfs.c to figure out what is going wrong would be greatly appreciated.

I've managed to mount the initramfs with MS_BIND into my root fs and found why /sbin/hotplug cannot be run currently. There is some off-by-one bug during file extraction that causes the first byte of the file to get left out. I.e. the file starts with "ELF\001" instead of "\577ELF".

This may or may not be related to another off-by-one bug that I'm seeing sometime when unpacking initramfs on s390x ("panic: length error").

The patch below is how I hacked prepare_namespace() to keep initramfs visible after boot.

syscalltrack-0.82, the 14th alpha release of the Linux kernel system call tracker, is now available. syscalltrack supports version 2.4.x of the Linux kernel on the i386 platform.

This release containes several new features, bug fixes and cleanups.

New in version 0.82, "Minty Chinchilla"

• This release adds support for matching and logging the current working directory. "I feel that the '%cwd' macro in logging format is really useful since it allows to know if 'open("passwd", ...)' relates to '/etc/passwd' rather than '/home/joe/tmp/passwd'." Patch from Simon Patarin.
• This release allocates the memory for kernel logging buffer using vmalloc, which allows you to allocate much more memory for them than the previous version. Patch from Simon Patarin.
• This release contains rewritten serialization/deserialization code for the rules library. The new code fits better with the overall design and is cleaner and more robust.
• This release contains a bug fix when detecting whether the kernel modules are loaded in the user space libraries. Modules should now be correctly recognized as loaded/unloaded in all cases. Bug spotted by Mike Shea.
• This release contains a bug fix for sctrace where sctracing a program with command line arguments could fail to find the program to trace.
• This release contains several testing improvements, including a new regression test script, from Orna Agmon.
• This release installs the syscalltrack binaries to ${prefix}/bin/name-version, to allow several syscalltrack versions to coexist. Kernel modules are installed to '/lib/modules/kernel-version/syscalltrack-version'.

The xfs patches for 2.4.20 have been respun as of 2003-02-05 23:39 UTC, including kdb v3.0.

For some time the XFS group have been producing split patches for XFS, separating the core XFS changes from additional patches such as kdb, xattr, acl, dmapi. The split patches are released to the world with the hope that developers and distributors will find them useful.

Read the README in each directory very carefully, the split patch format has changed over a few kernel releases. Any questions that are covered by the README will be ignored. There is even a 2.4.21/README for the terminally impatient :).

Here's a series of PCI Hotplug patches, a few related PCI core patches, and two small, related sysfs patches.

The hotplug driver patches consist of a lot of bug fixes due to problems found by the smatch and checker projects, and a big patch to remove pcihpfs and use sysfs instead from Stanley Wang. I've also moved the few functions in drivers/hotplug/pci_hotplug_util.c to drivers/pci/hotplug.c which is a better place for them.

There are some sysfs updates for pci devices from Dan Stekloff and a new function was added to sysfs to support the move from pcihpfs to sysfs. This sysfs patch was blessed by Pat Mochel.

This patch provides discontigmem support for the IBM x440. This code has passed through the hands of several developers: Chandra Seetharaman, James Cleverdon, John Stultz, and last to touch it, me :-) This patch requires full acpi support.

I've tested this patch on an 8 way x440 16 GB of RAM with and without HT (acpi=off).

The Linux Test Project test suite LTP-20030206.tgz has been released. Visit our website (http://ltp.sourceforge.net) to download the latest version of the testsuite that contains 950+ tests for the Linux OS. The site also contains other information such as: test results, a Linux test tools matrix, technical papers and HowTos on Linux testing, and a code coverage analysis tool. There is also a list of test cases that are expected to fail, located at (http://ltp.sourceforge.net/expected-errors.php)

The highlights of this release are:

• New options added to 'pan'.
  • quiet mode to reduce output ( -q )
  • pretty mode to format logfile output ( -p ).
• New options added and changes made to "runalltests.sh".
  • network tests can be run from runalltests (-N).
  • Optional cpu (-c), memory (-m), i/o (-i), and network (-n) load generation.
  • default result log directory is now /results.
• New tests added that cover:
  • hyperthreading
  • stressing mmap()
  • libmm
  • commands such as: cpio, ln, and cp
• Inclusion of all submitted and accepted patches/fixes through 02/05/03.

We encourage the community to post results, patches, or new tests on our mailing list, and to use the CVS bug tracking facility to report problems that you might encounter. More details available at our web-site.

As of the next release, we will be adding the option to license the ACPI AML interpreter (drivers/acpi/*/*.c) under the BSD license, as well as the current, GPL license.

While this will nominally increase your rights w.r.t. the code, the real reason for this is for us to more easily accept external contributor's changes into the interpreter's code (a good thing for everyone).

The Linux-specific ACPI code (drivers/acpi/*.c) is not affected by this change (i.e. it is still GPL-only).

This was mentioned a couple of months ago, but we're now finally getting around to doing it. :)

Thanks to Arnd Bergmann, it looks like the klibc and initramfs code is now working. I've created a patch against Linus's latest bk tree and put it at:

http://www.kroah.com/linux/klibc/klibc-2.5.59-2.patch.gz

(I can't get to kernel.org right now, sorry) and there's a bk tree at:

bk://kernel.bkbits.net/gregkh/linux/klibc-2.5

I'd really like to send this to Linus now, but I'm going to be away from email for about a week, so I'll wait will I get back. If anyone has any issues with this patch, please let me know.

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.59/2.5.59-mm9/

• Adam's cleanup and cutdown of devfs has been put back in again. We really need devfs users to test this and to report, please. (And not just to me! I'll only bounce it to Adam J. Richter <[email protected]> anyway)

  Adam has a userspace helper app `devfs-helper' at

  ftp://ftp.yggdrasil.com/pub/dist/device_control/devfs/

• The I/O scheduler changes here are stable now. All-round peformance is OK.

  I've been using it on the desktop for a week. Certain operations are noticeably different; for example a big `cvs co' would previously have had several-second stalls every 20 seconds or so. Now, things slow down but it keeps going. I'm sure the overall runtime will be slightly increased, but overall the decreased read latency is good.

• dcache_rcu has been put back.

OK. Looks like Linus is hot on the trail.

BTW, some nice people have been sending in smalldevfs testing results (successful). I've put that patch back up at

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.59/smalldevfs.patch

for other testers. It applies to 2.5.59 base.

I'm afraid that I have to bring this news to linux-kernel; people who have written code for the Linux kernel need to know about this, and we need to come to a decision about the action we wish to take. Taking no action sends a message that "we don't care what you do with kernel code, even if you violate the terms of the license."

It would appear that Castle Technology Limited, UK, have taken some of the Linux kernel 2.5 code, and incorporated it into their own product, "RISC OS", which is distributed in binary ROM form built into machines they sell. This code is linked with other proprietary code.

I have a detailed description which shows how the Linux source code can be slightly modified to produce the disputed code, with reasons each modification. This will be provided to people upon private email request.

Having discussed this with Linus, Linus is of the opinion that a public letter should be written to Castle Technology Ltd, copied to lkml and various news sites. However, I'd like to get this issue into the minds of people who have touched any of the following code:

• PCI subsystem
• IO resource allocation

The guy who reported the problem to me has already tried to contact the company concerned to ask for the source under the terms of the GPL, and this resulted in the "function signatures" being removed in the next version of the product, while the actual code remained. No other response was forthcoming.

Subsequently, during the first week of January, the guy has contacted the company again asking for the source covering the disputed code, this time copying me with the email. Again, no repsonse from Castle Technology has been forthcoming to date.

There is more than one player involved with RISC OS:

Pace Micro, of Saltaire, West Yorkshire

http://www.pace.co.uk/

http://uk.finance.yahoo.com/q?s=PIC.L&d=c&k=c1&a=v&p=s&t=5y&l=on&z=m&q=l

seem to own the Copyright.

And RISCOS Ltd., of Cardiff, Wales

http://www.riscos.com/

are involved as well.

I'd like to point out, however, that Pace sub-license RISC OS. As far as I am aware at present, RISCOS Ltd do not distribute the code in question, and neither does Pace Microtechnology Ltd.

It has come to my attention that some people are trying to implicate the above companies in this.

I would strongly suggest people do not start to make (unfounded) claims against neither RISCOS Ltd nor Pace Microtechnology Ltd unless they have proof, in which case such proof should first contact the appropriate copyright holders concerned.

perfctr-2.4.5 is now available at the usual place: http://www.csd.uu.se/~mikpe/linux/perfctr/

This just is a minor maintenance release, before the API fixes and extensions which are scheduled for perfctr-2.5.

Version 2.4.5, 2003-02-09

• Corrected the unit mask definition for the K7 SYSTEM_REQUEST_TYPE event in etc/perfctr-events.tab: WC is 0x02 not 0x04.
• Fixed two compile warnings which could be triggered in 2.5 kernels.
• Patch kit updates for 2.4.21-pre4/2.4.18-24(RedHat)/2.5.59-osdl2 kernels.

Here's a simple stochastic fairness queueing disk scheduler, for current 2.5.59-BK. It has known limitations right now, mainly because I didn't bother making it complete. But it should suffice for some rudimentary testing, at least.

I'm not going to go into great detail about how it works, see Andrea's initial post of the paper referenced. This version may not be completely true to the SFQ concept, but should be close enough I think. It divides traffic into a fixed number of buckets (64 per default), and perturbs the hash every 5 seconds (hash shamelessly borrowed from networking atm, see comment).

To avoid too many disk seeks, when it's time to dispatch requests to the driver, we round robin all non-empty buckets and grab a single request from each. These requests are sorted into the dispatch queue.

For performance reasons, io scheduler request merging is still a per-queue function (and not per-bucket).

In closing, let me stress that this version has not really been tested all that much. It passes simple SCSI and IDE testing, should work on any hardware basically.

In the nature of taking this concept to the extreme, here's a CFQ disk scheduler (it should be obvious by now, that I'm simply making up a TLA as I see fit :-), or Complete Fair Queueing. It never suffers from queue collisions.

So how does it work? As with SFQ, a hash of busy queues is maintained. If a queue for a given queue doesn't exist, one is simply allocated. The actual queueing of requests works like the SFQ scheduler I sent out yesterday, with little twist: we try to put at least cfq_quantum number of requests on the dispatch queue. If only a small number of processes are waiting for io, then this significantly helps throughput by minimizing the time spent between finishing one request and starting a new one.

Other changes/fixes from SFQ:

• Leave request mergeable even while on the dispatch list, to make the merge window as big as possible.
• The dispatch_sort() would sometimes not order requests correctly.
• Various small fixes.

Interestingly, dbench results show little variance between runs with this CFQ scheduler. Another point of interesting to folks may be that it would be trivial to add process io priorities on top of CFQ (or SFQ for that matter, but I consider CFQ to be the superiour scheduler).

If you play with this, let me know how it fares. Patch is against 2.5.60.

Release 0.2.0 of the Open POSIX Test Suite is now available at http://posixtest.sourceforge.net. This second release contains POSIX conformance tests for 50-80% of the POSIX functions of threads, signals, and semaphores. It also contains the full timers suite (tags TMR and CS) released in 0.1.0 with bug fixes.

The release notes that appear on download describe how to compile and run these tests.

The README page and the Open POSIX Test Suite website (above) give more information on the project goals and progress as well as information on how to contribute or contact us if you are interested.

Many thanks to Jim Houston and other members of the POSIX testing community for their bug fixes, patches, and suggestions on how to improve the 0.1.0 suite.

The Open POSIX Test Suite is an open source test suite with the goal of creating conformance test suites, as well as potentially functional and stress test suites, to the functions described in the IEEE Std 1003.1-2001 System Interfaces specification. Initial work is focusing on timers, threads, semaphores, signals, and message queues.

Feel free to contact [email protected] if you would like further information.