Kernel Traffic #275 For 2�Oct�2004

I just posted "blktool" on my SF page, http://sourceforge.net/projects/gkernel/ and in BitKeeper at bk://gkernel.bkbits.net/blktool

blktool aims to be an easier to use, and more generic version of the existing utility 'hdparm'. For example,

        $ hdparm -c1 /dev/hda
                becomes
        $ blktool /dev/hda pio-data 32-bit

        and

        $ hdparm -L0 /dev/hda
                becomes
        $ blktool /dev/hda media unlock

The utility is currently still fairly specific to IDE devices (as hdparm is), but that will change in the coming weeks as SCSI, I2O, and possibly some bits of hardware RAID control are added.

The audience for this application, like hdparm, is fairly narrow, specific to people who tweak their storage devices and _know what they are doing_. Improper use of this tool, like hdparm, can turn your disk into a doorstop.

This patch adds a new system call `waitid'. This is a new POSIX call that subsumes the rest of the wait* family and can do some things the older calls cannot. A minor addition is the ability to select what kinds of status to check for with a mask of independent bits, so you can wait for just stops and not terminations, for example. A more significant improvement is the WNOWAIT flag, which allows for polling child status without reaping. This interface fills in a siginfo_t with the same details that a SIGCHLD for the status change has; some of that info (e.g. si_uid) is not available via wait4 or other calls.

I've added a new system call that has the parameter conventions of the POSIX function because that seems like the cleanest thing. This patch includes the actual system call table additions for i386 and x86-64; other architectures will need to assign the system call number, and 64-bit ones may need to implement 32-bit compat support for it as I did for x86-64. The new features could instead be provided by some new kludge inventions in the wait4 system call interface (that's what BSD did). If kludges are preferable to adding a system call, I can work up something different.

I added a struct rusage field si_rusage to siginfo_t in the SIGCHLD case (this does not affect the size or layout of the struct). This is not part of the POSIX interface, but it makes it so that `waitid' subsumes all the functionality of `wait4'. Future kernel ABIs (new arch's or whatnot) can have only the `waitid' system call and the rest of the wait* family including wait3 and wait4 can be implemented in user space using waitid. There is nothing in user space as yet that would make use of the new field.

Most of the new functionality is implemented purely in the waitid system call itself. POSIX also provides for the WCONTINUED flag to report when a child process had been stopped by job control and then resumed with SIGCONT. Corresponding to this, a SIGCHLD is now generated when a child resumes (unless SA_NOCLDSTOP is set), with the value CLD_CONTINUED in siginfo_t.si_code. To implement this, some additional bookkeeping is required in the signal code handling job control stops.

The motivation for this work is to make it possible to implement the POSIX semantics of the `waitid' function in glibc completely and correctly. If changing either the system call interface used to accomplish that, or any details of the kernel implementation work, would improve the chances of getting this incorporated, I am more than happy to work through any issues.

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm1

• perfctr is ready to be merged up, but there are concerns that it duplicates perfmon capabilities, and that perfmon may be a better base from which to start. Am waiting for the dust to settle on that front.
• The packet-writing patches should be ready to go, but I haven't even looked at them yet, and am not sure that anyone else has reviewed the code.

• Added the kprobes facility.

  Generally we prefer to not merge infrastructure into the kernel unless it has in-kernel users. kprobes is exceptional, in that its applications are all custom-written to solve a particular problem.

  One problem we face with kprobes is that there is no easy way in which it can receive runtime (or regression) testing.

  This patchset does include an application (a network packet tracer) but it's not obvious that we should retain that in the kernel.

• This kernel probably still has the ia64 scheduler startup bug, although it works For Me.
• If people have significant patches in here, please regression test them to make sure everything landed OK, thanks.

The following two patches can be dropped from -mm. The functionality has moved to scripts/mksysmap

handle-undefined-symbols.patch
Fail if vmlinux contains undefined symbols

sparc32-ignore-undefined-symbols-with-3-or-more-leading-underscores.patch
sparc32: ignore undefined symbols with 3 or more leading underscores

Some of the metadata handling is needed kernel side. I'm hoping we can avoid most of it with the drive hotplug code. The corner case causing the problem is when no arrays are configured so there is no device/hwif present.

I looked at the libata stuff - it's part of the reason I sent Jeff the error dump/translate patch but right now libata is woefully ignorant of a large number of IDE/EIDE/ATA considerations.

dmraid 1.0.0-rc3 is available at http://people.redhat.com:/~heinzm/sw/dmraid/ in source, source rpm and i386 rpm.

dmraid (Device-Mapper Raid tool) discovers, [de]activates and displays properties of software RAID sets (ie. ATARAID) and contained DOS partitions using the device-mapper runtime of the 2.6 kernel.

The following ATARAID types are supported on Linux 2.6:

Highpoint HPT37X
Highpoint HPT45X
Intel Software RAID
Promise FastTrack
Silicon Image Medley

This ATARAID type is only basically supported in this version (I need better metadata format specs; please help): LSI Logic MegaRAID

Please provide insight to support those metadata formats completely.

Thanks.

See files README and CHANGELOG, which come with the source tarball for prerequisites to run this software, further instructions on installing and using dmraid!

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm2/

• Added Tony Luck's ia64 devel tree to the -mm "external trees" lineup.
• The monster memory leak which some people were seeing with audio CD writing should be fixed.
• If your floppy drives magically disappear, try booting with floppy=no_acpi and send a report.

• Added the reiser4 filesystem. Please give it a whizz. Please Cc [email protected] on any problem reports. (It spits a bunch of warnings on 64-bit compiles - that'll be fixed)

  Here's a little questionnaire which Hans filled out for me, to help people get started:

  • A really short guide to how to get up and running with this filesystem.

    Reiser4 is a file system based on dancing tree algorithms, and is described at http://www.namesys.com. One should be able to get it up and running just like any of the other filesystems supported by Linux. Configure it to be compiled either builtin or as a module. Create reiser4 filesystem with mkfs.reiser4, mount and use it. More detailed info can be found at http://thebsh.namesys.com/snapshots/LATEST/READ.ME.

  • The direct URL which people use to obtain the mkfs tool for this filesystem. Also fsck and anything else.

    Reiser4 userland tools can be obtained at ftp://ftp.namesys.com/pub/reiser4progs. ftp://ftp.namesys.com/pub/reiser4progs/README contains detailed instructions on how to compile and install these tools. Also all reiser4 progs have man pages.

  • Any known shortcomings, caveats, etc.

    Reiser4 has been tested on i386 yet only. Quota support is not ready yet. Should be ready soon. Reiser4 was tested extensively, and we got to where the mailing list was not able to hit any bugs, but then we told people that, got an order of magnitude increase in users, and they are able to hit bugs that we are working on now.

    Reiser's Law of Software Engineering: Each order of magnitude increase in users finds more bugs, in a quantity equal to the previous order of magnitude increase in users. Success for software developers is measured by how long the frustration lasts.

    Only the very core functionality is working. Exotic plugins, an API for multiple operation transactions and accessing multiple small files in one syscall, compression, inheritance, all have been postponed until after the core functionality is shipped. The compression plugin needs a code review before anyone should use it.

  • A statement on compatibility with reiserfs3 filesytems.

    To upgrade from reiserfs V3 to V4, use tar, or sponsor us to write a convertfs.

  • Bear in mind that people will immediately benchmark this filesytem, and first impressions count. Now is your chance to communicate any tuning guidelines, mount options or whatever which you'd like people to understand BEFORE they start publishing benchmark info.

    Reiser4 is not tuned for fsync/sync/O_SYNC performance yet.

    If you see results that are much different from those at www.namesys.com/benchmarks.html, let us know. If you see performance characteristics that don't quite make sense, email [email protected], such things are always of interest.

    reiser4 is not tuned for mmaping and dirtying more than physical ram like IOzone does. This is quite different in its code path from writing and dirtying more than physical ram. There are those who think that what IOZone does is rarely done by real programs, and therefor we should not bother to optimize what it does. All I know is, this month we are not optimized for it.

    Please consider its space savings when you benchmark it also.

The reiser 4 system call sys_reiserfs seems to need an additional patch, which is craftily hidden inside reiser4-only.patch

That patch creates fs/reiser4/linux-5_reiser4_syscall.patch, which I can only assume reiser 4 users should apply...

Kind of ugly.

Looking further, the horrors only increase. It looks like sys_reiser4() is an interface to load programs into the kernel, with reiserfs4 containing an interpreter.

I'll leave aside the issues of having a scripting language inside the kernel, since I'm sure other people will comment on it.

However, I am absolutely flabbergasted that Hans Reiser is using a syscall here, instead of a filesystem interface.

Furthermore, why do the parsing in the kernel, instead of compiling the human-readable strings in userspace and loading something easy to use into the kernel, like the selinux subsystem does?

Since this code is bound to be horribly controversial, it may be an idea to remove this from the reiserfs4 core patch. That way the battles over the filesystem, and its interactions with the rest of the kernel can be fought first, without having the whole reiserfs4 filesystem strand in the quicksand of "why do we need an interpreted language with completely new filesystem semantics in the kernel?"

It's my understanding that sys_reiser4() is basically defunct at this point.

It will probably be revived at some time in the future but we'd be best off crossing that bridge when we arrive at it, and ignoring the syscall part of the code at this time.

For review purposes it would be better if the syscall code and all the namesys debug support code simply weren't present in the patch. But one can sympathise with the need to keep it there for the time being. Please just read around it.

I would say unfinished and in need of a code review by me before anyone starts using it, instead of defunct. There is no good reason for it to be sent to Andrew as a patch file, and the guy responsible is on vacation. What it should be in as is an experimental do not touch config option turned off by default.

sys_reiser4 is needed for these purposes:

• to eliminate the (otherwise valid) argument that it is more performance efficient for attributes to be accessed via an API that is different from files, by allowing multiple files to be accessed in one system call
• to bundle multiple filesystem operations into one atomic write
• to prepare the groundwork for the semantic enhancements described in www.namesys.com/whitepaper.html
• to define a standard interface that users will find uniform across all apps for this functionality
• to allow VFS to remain undisturbed in the eyes of legacy apps while semantic enhancements go into the filesystem namespace in a form that is less crippled by compatibility issues.

Now that the core reiser4 functionality is stable, the lead programmers and I can spare some time to review sys_reiser4 and the compression plugin (also not yet ready for prime time). This will take us 6-12 weeks I would guess, as Digeo is keeping us 50% busy with work that earns our paychecks at the moment, darpa is also keeping me busy with www.namesys.com/blackbox.html, and I expect there will be a few bugs found in the core code over the next few months also.

After a bit of discussion on the dri lists, we have come to the decision that it is probably necessary to retire the above driver, no dri developer is currently using the above hardware and the driver is so different from the others it makes a lot of hacks in the drm needed...

If anyone does actually use this driver and hardware let us know or it'll be marked as BROKEN soon and then it will actually break :-)

I noticed that every file that could be built as a module was sucking in sched.h (and therefore, every other include file under the sun).

This patch

• removes the sched.h from module.h
• Moves the capable() definition from sched.h to capability.h
• split out the wake_up_* stuff to linux/wakeup.h
• Removed sched.h includes from a bunch of drivers that didn't need it due to the above work.
• Fixes up all the breakage I was able to find under x86. Fixing other arch's is simple enough, they just need to include sched.h explicity in a few places now (or jiffies.h, or capability.h or wakeup.h))

I've not done any measurements to see if this is noticable on a compile, as I'd expect it to be mostly in the noise anyway (though last time I did this in 2.5.early, it did shave off the best part of a minute off my worst-case-scenario build), but untangling the spaghetti of includes a little should at least mean gcc uses less memory during the build.

I hereby declare open season on linux/sched.h!

In preparation for moving all user-related bits out of sched.h and coopting linux/user.h for this purpose, this patch converts all inclusions of linux/user.h to asm/user.h

OK, let's go! ;-)

Let's see how often we can kill it's include lines. To start from a clean base, I looked at vanilla 2.6.8.1 first before trying out your patches.

analysis was i386-only, my personal config builds, allyesconfig does not (neither does it with an unpatched kernel)

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm3/

• Added three more bk trees:

  bk-fb: Some ARM framebuffer driver (rmk)
  bk-mmc: ARM-specific media drivers(?)
  bk-watchdog: watchdog drivers

• I'm totally unclear on what's happening with the release_task sleep-while-atomic bug, and with the CPU hotplug BUG. This kernel will probably emit might_sleep warnings. Turn off CONFIG_PREEMPT if it gets irritating.
• Added Nick Piggin's CPU scheduler to see what happens. See inside the patch for details. Please test, benchmark, report.
• This is (very) lightly tested. Mainly a resync with various parties.

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm4/

• Added the kexec code. Again. This was in -mm a year or so ago but didn't make it.

• This kernel has an x86 patch which alters the copy_*_user() functions so they will return -EFAULT on a fault rather than the number of bytes which remain to be copied. This is a bit of an experiment, because this seems to be the preferred API for those functions. It's a see-what-breaks thing.

  And things will break. If weird behaviour is observed, please revert usercopy-return-EFAULT.patch and send a report.

I'd prefer it if you (and everyone else) could give a meaningful English-language Subject: to patches, please.

A well-chosen patch Subject: becomes a sort of globally-unique key by which the patch is tracked - I munge it into a patch filename and it propagates all the way into bitkeeper. It can be used for searching email folders, googling, inter-developer discussion, etc, etc.

Just posted version 2 of rng-tools at http://sourceforge.net/projects/gkernel/

This release fixes a problem related to 2.6.x kernels.

rng-tools is currently for users of hardware random number generators (RNGs), and the included daemon rngd fill the kernel entropy pool from userspace with the results of the output.

Future directions include:

• support for VIA 'xcrypt' instruction in userspace, avoiding the need for a kernel driver
• multithreaded daemon
• additional entropy sources besides the dedicated h/w RNG