Kernel Traffic #300 For 29�Mar�2005

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.11-rc3/2.6.11-rc3-mm2/

• Added the mlock and !SCHED_OTHER Linux Security Module for the audio guys. It seems that nothing else is going to come along and this is completely encapsulated.

• Various other stuff. If anyone has a patch in here which they think should be in 2.6.11, please let me know. I'm intending to merge the following into 2.6.11:

  alpha-add-missing-dma_mapping_error.patch
  fix-compat-shmget-overflow.patch
  fix-shmget-for-ppc64-s390-64-sparc64.patch
  binfmt_elf-clearing-bss-may-fail.patch
  qlogic-warning-fixes.patch
  oprofile-exittext-referenced-in-inittext.patch
  force-read-implies-exec-for-all-32bit-processes-in-x86-64.patch
  oprofile-arm-xscale1-pmu-support-fix.patch

Unless I misunderstood the code, this one is available for quite some time: <http://www.olafdietsche.de/linux/accessfs/> or a newer, self-contained version <http://lkml.org/lkml/2005/1/11/221>

Or you could use a real solution - filesystem capabilities: <http://www.olafdietsche.de/linux/capability/> and if you don't like this one :-), there's also an alternative existing here: <http://www.stanford.edu/~luto/linux-fscap/>

Most people felt that a more general capabilities module would be nice to have. But, no one offered any code, or volunteered to work on it.

I have no objection to that approach, but am not willing or able to do it myself. My opinion is that expanding the scope of the LSM would significantly increase its security risk. That job needs to be done very carefully, by someone with a deep understanding of the kernel's internal use of capabilities.

Perhaps, Christoph's suggestion could become part of a more general module, which might replace the RT-LSM in the 2.8 timeframe. Our LSM is a modest solution aimed at solving the immediate needs of audio developers and users with minimal impact on kernel security or correctness.

i somewhat share that view. (despite all the promises from the audio folks - if they are just half as agressive resisting removal as they were pushing integration then it will never be removed ;-)

but i'm not sure how rlimits will contain the whole problem - can rlimits be restricted to a single app (jackd)? The most canonical use of rlimits is per-user (per-group), so the rlimit could end up _widening_ the effects of the hack ...

I'd like to announce, yet-another-hotplug based userspace project: hotplug-ng. This collection of code replaces the existing linux-hotplug package with very tiny, compiled executable programs, instead of the existing bash scripts.

It currently provides the following:

• a /sbin/hotplug multiplexer. Works identical to the existing bash /sbin/hotplug.
• autoload programs for usb, scsi, and pci modules. These programs determine what module needs to be loaded when the kernel emits a hotplug event for these types of devices. This works just like the existing linux-hotplug scripts, with a few exceptions.

But why redo this all in .c code? What's wrong with shell scripts? Nothing is wrong with shell scripts, unless you don't want to have an interpreter in your initramfs/initrd and you want to provide /sbin/hotplug and autoload module functionality. Or if you have a huge box that spawns a zillion hotplug events all at once, and you need to be able to handle all of that with the minimum amount of processing time and memory.

So, how small are these programs? Take a look:

   text    data     bss     dec     hex filename
   4669      32     124    4825    12d9 hotplug
   5077       8     348    5433    1539 module_pci
   4925       8     412    5345    14e1 module_scsi
   5349       8     348    5705    1649 module_usb

Those are all static binaries, linked with klibc (which is included in the hotplug-ng package, just like udev.)

This compares to the following bash scripts:

-rwxr-xr-x  1 root root  4412 Feb 10 15:28 /sbin/hotplug
-rw-r--r--  1 root root   702 Sep 24 08:04 /etc/hotplug/blacklist
-rw-r--r--  1 root root  5293 Sep 24 08:04 /etc/hotplug/hotplug.functions
-rwxr-xr-x  1 root root  3739 Sep 24 08:04 /etc/hotplug/pci.agent
-rwxr-xr-x  1 root root  1459 Sep 24 08:04 /etc/hotplug/scsi.agent
-rwxr-xr-x  1 root root 13466 Sep 24 08:04 /etc/hotplug/usb.agent
-rw-r--r--  1 root root 39306 Sep 24 08:04 /etc/hotplug/usb.distmap
-rw-r--r--  1 root root  4364 Sep 24 08:04 /etc/hotplug/usb.handmap
-rw-r--r--  1 root root   189 Sep 24 08:04 /etc/hotplug/usb.usermap

All of which are loaded into memory for each hotplug event (for specific hotplug events, only that bus type of file is loaded.)

But what about speed? With a completely unscientific measurement on my old, slow laptop, it takes about 2 seconds from the time I plug a usb device into the machine, for the proper module to be loaded. With the hotplug-ng program, it takes less than a second.

And for those of you who might remember the old dietHotplug program that also did the same thing in a tiny amount of space, this project obsoletes that one. dietHotplug had to be rebuilt for every kernel that was used on the system, hotplug-ng uses the ability for modprobe to determine the module that needs to be loaded based on the module aliases (modprobe as it currently works stops loading modules when it finds an alias that matches. This does not work for drivers that claim to support "all devices" and then later on fail on devices that they really don't support. For that, all matching drivers need to be loaded for the system to work properly. The linux-hotplug scripts handle this correctly, so if you rely on this functionality, please stick with that package for now. I'll be modifying modprobe to add this feature in the near future.).

The code can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/hotplug-ng-001.tar.gz
for those who wish to poke around in it.

I still have a few more programs to write to get it up to the same functionality as the existing hotplug scripts (firmware, ieee1392, etc.) but those will be done soon. I'd like to get people's comments on the idea, and welcome suggestions and even patches :)

hotplug-ng development is done in a BitKeeper repository located at:
bk://linuxusb.bkbits.net/hotplug-ng

Wow, thats pretty awesome. Just the other day I said, "Why is hotplug written in sh? Isn't that horribly inefficient way of handling something that needs to be done quickly using the least amount of resources possible?" It seems you were reading my mind.

Please, continue this project and encourage distros to switch to it (when it exceeds hotplug in functionality and stability). Ubuntu currently is trying to reduce boot time, and I bet something like this would factor in (even a few seconds helps).

This is a heads up that the BK tree for the kernel is currently at 59,000 changesets give or take a few. The BK that you are using uses unsigned shorts for the internal names of each delta which means you folks are about 100 days away from things no longer working.

The good news is that the openlogging tree for the kernel has 135,000 changesets so we've obviously long since fixed this problem.

The bad news is that you will need to upgrade your BK binary in order to pass over the 64K changeset boundary. The data is stored on disk in ascii so it doesn't matter if you upgrade until you hit the problem but sooner or later you will need to upgrade.

We'll get the fix into bk-3.2.4 which should be out by the end of the month. When we release that we'll send out notice and it would be good if people gave it a try and let us know if they hit issues because in a couple of months everyone is going to have to upgrade.

It's possible that we'll be changing the BK license to conform more with our commercial license but we won't do that without running it by Linus & Co to make sure that it's acceptable. One change we'd like to make there is to clarify the non-compete stuff. We've had some people who have indicated that they believed that if they used BK they were agreeing that they would never work on another SCM system. We can see how it is possible that people would interpret the license that way but that wasn't our intent. What we would like to do is change the language to say that if you use BK you are agreeing that you won't work on another SCM for 1 year after you stop using BK. But after that you would be able to hack on anything that you wanted. That was more of what we had in mind in the first place but we didn't make it clear. If you all thought that using BK meant you had no right to hack on SCM ever again, that's just not fair. We need to protect our IP but you should have the right to choose to go hack SCM if that's what you (our first choice is that you came and worked here, we really like kernel hackers, but if you don't want to that's cool too).

Attached is an implementation of /proc/cpumem. /proc/cpumem shows the valid physical memory ranges.

• i386 and x86_64
• implement valid_phys_addr_range() and use it. (the first argument of the i386 version is little uncomfortable.)
• /dev/mem of the i386 version should be mofified. but not yet.

example: amd64 8GB Mem

# cat /proc/cpumem
0000000000000000 000000000009b800
0000000000100000 00000000fbe70000
0000000100000000 0000000100000000
#

start address and size. hex digit.

OK, time to stop polishing and start publishing.

This is to announce yaird, Yet Another mkInitRD, a rewrite of mkinitrd based on hotplug algorithms.

MOTIVATION

Why a rewrite? The versions of mkinitrd that I studied, Debian sid and Fedora FC3, have some problems: they capture a lot of knowledge about the boot process, but don't always understand when a new kernel uses a different module name than the old kernel, may rely on modules compiled into the kernel, and don't always catch errors at the earliest opportunity.

Assumption: there are three issues that cause most of these problems.

• Backward compatibility: sysfs provides a wealth of information about hardware, but if you have to support 2.4 kernels, you have to limit yourself in the use you can make of that information.
• Originality: hotplug does a pretty good job of finding the appropriate modules for a device, basically because it closely follows the algorithms the kernel uses for matching hardware with modules. Deviating from those algorithms is unlikely to produce a more robust result.
• Shell scripting: beyond a certain level of complexity, the shell syntax makes it difficult to focus on the data structures you're trying to process and makes it difficult to do rigorous error checking.

Yaird is intended to find out whether that assumption is correct: if so, a program to build initrd images will be more reliable if it's written in perl, if it uses sysfs to determine what hardware needs to be supported, and if it closely follows the methods hotplug uses find the modules needed to support some hardware.

STATUS

There is working code: yaird booted the machine this note is written on. Code is available online, there also is a paper that discusses the workings of the application in detail.

http://www.xs4all.nl/~ekonijn/yaird/

So far, the program works correctly on every machine it's been tested on, but with only two test boxes that does not mean much.

Basic creature comforts are in place: "configure; make" but no RPM or deb files, a README and help option but no manual page.

Features:

• understands SATA and IDE. USB sticks and SCSI should work, but are untested.
• understands MDADM and LVM2, activates only necessary devices, understands stacks like LVM on top of stripe on top of mirror.
• handles both initrd and initramfs.
• understands USB keyboards.
• understands hotplug blacklist.
• knows that a module compiled into the kernel does not need insmod.
• understands /etc/fstab, including niceties such as labels, uuids and octal escapes.
• image generation understands symbolic links and shared libraries. (should support 32bit emulation on 64bit kernels; untested).
• templating system to simplify tuning the initrd image to the distribution.
• avoids hard-coded device numbers; does not require devfs.
• testing mode gives detailed overview of the systems hardware and the modules needed to support that.

There are rough edges in practically every feature: this is suitable for testing, but not for production use.

TODO

1. Testing so far is 100% successful, but with just two boxes to play with, that's not saying much. If you can find space to test the code on your system, your results are highly appreciated. At this point, hardware testing is most valuable: I already know that dm-crypt is unsupported for now, but whether this stuff can boot a powerbook, sparc, or just about anything else is an open question.
2. Feedback. This may be an interesting idea, but how does it relate to other new stuff floating about? In particular, what about the work that's going on putting udev on initramfs: are these approaches complementary or alternatives? Different perspectives on where this stuff fits in would help.
3. Support more configurations. dm-crypt is unsupported for now, and so are multipath, swsusp, EVMS, NFS and loopback mounts. Implementing this stuff becomes interesting once there are some tests results that the basic ideas work in practice.

Having a mkinitrd that's not a shell script is a godsend. I would endorse yaird on that fact alone :)

I've long wanted a "mkinitfoo" that would create .cpio.gz for initramfs by default. So, good job there, too.

Eventually your script will need pull in, into the initramfs, klibc and programs linked against klibc.

I've been working in implementing extended attributes support in the JFFS2 filesystem.

During the (short) time I worked on it, I just decided to try to bring back the thread on SELinux in hand-held and embedded devices and see if there's someone interested in contributing to it and collaborating to make something out of it.

The current work is just a draft, I've started with the standard Vanilla kernel sources plus mtd JFFS2 sources, used to patch the vanilla ones for latest code (I'm confused on which one has the most updated tree or if there are special differences between mtd's trees and vanilla's), and implemented the skeleton using the reiserfs xattr code base.

Then Russell Coker commented to me that I should use the handhelds.org kernel, so, I have the doubt on which one use, even if porting the changes made to JFFS2 code base to vanilla sources wouldn't be difficult, at least I suppose.

I've opened a few wiki pages for discussion and documenting until it gets further developed:

http://wiki.tuxedo-es.org/tuxedo/JFFS2xattr
http://wiki.tuxedo-es.org/tuxedo/SELinuxEmbedded

In addition, having someone experienced with xattr API could be great, as development documentation seems inexistent, among James Morris' merged xattr consolidation code.

Here goes the second pre of v2.4.30.

It contains a bunch of important networking fixes, most noticeably the brlocks rework.

Plus USB fixes, megaraid2 driver update, JFS update, amongst others.

Version 0.0.4 of yaird is now available at:

http://www.xs4all.nl/~ekonijn/yaird/yaird-0.0.4.tar.gz

Yaird is a perl rewrite of mkinitrd. It aims to reliably identify the necessary modules by using the same algorithms as hotplug, and comes with a template system to to tune the tool for different distributions and experiment with different image layouts. It requires a 2.6 kernel with hotplug. There is a paper discussing it at:

http://www.xs4all.nl/~ekonijn/yaird/yaird.html

There are rough edges in practically every feature, and numerous features still need to be added: this is suitable for testing, but not for production use.

Thanks to all who gave feedback, sent patches and were brave enough to test this stuff. Below are highlights from the change log and todo list.

Summary of user visible changes for version 0.0.4:

• Process kernel command line options: init=, ro, rw.
• Boot into single user mode supported
• Support modules outside /lib/modules
• Support modules with extension other than .ko
• Warn about duplicates in modules.dep
• Generated image now waits for device to become visible in /sys, and gives error message if it doesn't
• Support 2.6.10 sysfs layout: SCSI now has a new subdirectory 'target'. (Thanks to Harald Dunkel for testing)
• Warn about unrecognised paths in /sys
• Empty lines in /etc/fstab are valid. (Patch Goffredo Baroncelli)

On top of the todo list are now:

• add command line option (--root=/dev/hdb) to simplify testing.
• add tree copying to the templates, to allow all of firmware to be copied to the image. Or all of /lib/modules, if you want to have hotplug on the image.
• get klibc run_init working. Test by switching the Debian template to initramfs. This should make Debian and Fedora templates more similar, it is also groundwork for possible hotplug-ng support.
• any patches you may wish to send :)