Kernel Traffic #200 For 13�Jan�2003

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.53/2.5.53-mm2/

Mainly stability work:

• If pte_chain_alloc() fails to allocate GFP_ATOMIC memory, the kernel oopses. This is a long-standing rmap problem. Present also in the 2.4 rmap patches and, as far as I know, production Red Hat kernels.

  So it is clearly a very rare problem but it is not acceptable to have an unchecked kmalloc in the core of the 2.5 VM.

  The approach which I took was to change the page_add_rmap() API to require the caller to pass in a preallocated pte_chain. And change all callers to allocate their pte_chains with GFP_KERNEL.

  This change is fairly ugly, but every other hare-brained scheme I could come up with had holes. This one adds maybe 20 instructions to pagefaults and works...

  The swapoff path has not yet been converted - this can still oops.

  The locking isn't quite right yet, if shared pagetables are enabled.

• If radix_tree_insert() fails to allocate GFP_ATOMIC memory, a system call will return -ENOMEM, resulting in application failure.

  This was fixed by implementing a reservation API within the slab allocator. Before taking locks the caller of radix_tree_insert will ask slab to preallocate sufficient objects in this CPU's slab head array to guarantee that the allocation of up to seven (on ia32) radix_tree_nodes cannot fail.

  This permitted the removal of the radix tree mempool. That's a 130 kbyte saving. (260k on 64-bit).

• Some aggressive pruning of various system-wide memory reserve settings:

  • The page reservation limits in the page allocator have been reduced from ~256 pages per zone to ~4 pages per zone.
  • The preallocation levels in the slab head arrays (which were ridiculously large) have been reduced from 32k-128k to, typically, a single page.
  • The per-cpu-pages head arrays in the page allocator have been reduced from ~64 pages to 2 pages.

  The net effect of these changes is to remove almost all of the kernel's reserved memory buffers. Instead of maintaining several megabytes of free memory the kernel will only maintain some tens of kilobytes.

  And guess what? Everything still works.

  I won't be submitting these changes - they are here for robustness testing. But it certainly does indicate that the settings of these thresholds need to be reviewed. And that there don't appear to be any low-on-memory deadlocks in the VM (with ext2, at least..)

• An updated dcache_rcu patch which should fix a rename-related race which Al Viro noted.

Further minor reductions may be possible from some additional clean-ups and perhaps a simplification to the programming interface.

This code is probably very buggy. I've just gotten it working on the machine on which I'm composing this message. It is not ready for integration yet. I would particularly appreciate help with the dentry manipulation and locking, which I think I've probably botched.

The notable differences between devfs and mini-devfs are:

1. devfsd has been replaced with the user mode helper /sbin/devfs_helper which is exec'ed for registration and initial lookup events (I could catch all the events that devfs does, but I don't think that's really necessary). I have yet to port devfsd to this interface. When sysfs support is added to the new kernel parameter system, I will make this a kernel parameter, so it can be off initially, and only turned on by systems that use it when the boot process is ready for it. This will also eliminate the problem where the boot process currently tries to do 200+ execs as each pseudo-terminal is registered.
2. The removable device code that cleared and reread partition tables from removable media all the time preventing use of userland partitioning with devfs is eliminated. (Non-devfs systems never had this problem.)
3. devfs_handle_t is now a synonym for struct dentry*.
4. A lot of the devfs routines are unimplemented. I haven't noticed much code that uses them, and I'm not sure that any code really should. I think arch/ia64/sn uses devfs_get_first_child, devfs_get_next_sibling. I need to understand what if any of the other routines are really necessary and why (for example, why can't we use struct dentry). My computer seems to run fine without them.

TODO:

First of all, I'd like to debug this code and I'd welcome any help. The only malfunction (aside from routines that aren't written) that I've noticed is that previously xdm would give up after trying to start the X server about five times (it is not configured). With this smaller devfs, xdm keeps trying to start the X server. Also, I'm pretty sure that my code is at least releasing and reacquiring dcache_lock when it shouldn't, and I think there may be some similar inode->i_sem issues.

In the future, I'd like to shrink the devfs interface to devfs_{create,delete}. If we prohibit file renaming in devfs, then drivers can be sure they've removed the devfs nodes that they've created when they delete the paths that they've created. A side-effect of this would be that devfs_handle_t would be eliminated. I still want the ability to pass a struct file_operations* to devfs_create as it may enable the elimination of fixed device numbers.

I think I'd like to change fs/super.c slightly to make it easier to statically allocate the struct super_block for filesystems that can have only one instance even if they are mounted in multiple locations (devfs, procfs, sysfs, usbdevfs, etc.).

I started hacking on this code by making an approximately ten line change to ramfs just to have it call a user level program on lookups and file creation. I had hoped to change the devfs routines to just generically operate on whatever was mounted on /dev. If the devfs API were shrunk substantially, it might be worth trying that approach again.

I have made a new version of my mini-devfs (attached below). I have also made a first version of my devfs_helper program to handle the functionality of devfsd on systems that use my mini-devfs. devfs_helper is avaiable from the following location:

ftp://ftp.yggdrasil.com/pub/dist/device_control/devfs_helper/devfs_helper-0.1.tar.gz

This version of devfs_helper only supports "EXECUTE" and "MODLOAD" actions, which may be the only events that are really necessary. In the future, I envision eliminating the "MODLOAD" action, since it is equivalent to "EXECUTE modprobe -C /etc/modprobe.devfs $devname".

devfs_helper is currently 211 lines of C code. In comparison, devfsd-1.3.25/*.[ch] is 3143 lines.

Also, here is version 2 of my mini-devfs. I am embarassed to say that I omitted Richard Gooch's copyright notice on his code that I copoied into mini-devfs/numspace.c. This patch corrects that and fixes a variety of other little problems. It also changes the directory of this facility to fs/mini-devfs/. One change that is necessary for operation with devfs_helper is that the event names are now in all capitals ("REGISTER" and "LOOKUP") to match the format of /etc/devfsd.conf.

Just to keep everyone up to date, here is a third iteration of my patch converting devfs to a ramfs-based file system. This one uses an almost unmodified version of devfs/util.c to restore automatic device number allocation and devfs_{,un}register_tape().

This code now tries to implement almost all of the devfs functionality that anything outside of arch/ia64/sn uses. The most significant except that I'm aware of is the ability to create a plain file with custom file operations, which is done the Memory Type Range Register code, but that code also provides a proc interface for the same thing, and I think the proc interface is what everyone uses right now anyhow.

If Christoph's patch for deleting a bunch of unused stuff from devfs gets into 2.5.54, that should make my patch smaller, and I'll post a new version then. If nobody objects, then perhaps I'll make that version replace fs/devfs rather than creating a separate fs/mini-devfs.

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.53/2.5.53-mm3/

• 2.5.53-mm2 was a bit sick in the timekeeping and slab department. That should be fixed here.
• The idea of using the slab head arrays for object preallocation has been abandoned. It involved too many slab changes, and slab just explodes in your face when touched. So I've used a custom reservation pool in the radix-tree code instead.
• I've spent two days chasing the memory leak which Con has reported and have thus far not been able to reproduce it (managed to collaterally discover a swapoff lockup and an htree leak though). It's probably an ext3/VM interaction. Please keep an eye out for this.

Happy new year to you all, hopefully most of you are back from the dead and the hangovers are all long gone. And if not, I'm told reading a large kernel patch is _just_ the medication for whatever ails you.

The 2.5.54 patch is largely mainly a big collection of various small things, all over the place (diffstat shows a long list of small changes, with some noticeable activity in UML, the MPT fusion driver and some of the fbcon drivers).

Various module updates (deprecated functions, updated loaders etc), usb, m68k, x86-64 updates, kbuild stuff etc etc.

This is yet another release of the IPMI driver for Linux. This release cleans up the rather broken locking that was in the driver and fixes the linux command line parsing so the driver may be properly compiled into the kernel. Patches are relative to 2.4.20 and 2.5.54.

This release adds minor bugfixes to the watchdog and fixes for handling buggy hardware. It adds the ability to have the user be notified of a pretimeout if not using NMIs for pretimeout notification.

As usual, you can get the drivers from SourceForge. The home page is http://openipmi.sourceforge.net. http://sourceforge.net/projects/openipmi gets you directly to the page with the info.

PS - In case you don't know, IPMI is a standard for system management, it provides ways to detect the managed devices in the system and sensors attached to them. You can get more information at http://www.intel.com/design/servers/ipmi/spec.htm.

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.54/2.5.54-mm3/

Several patches here which fix pretty much the last source of long scheduling latency stalls in the core kernel - long-held page_table_lock during pagetable teardown.

The preemptible kernel now achieves around 500 microsecond worst-case latency on a 500MHz PIII (with a slow memory system). This is about as good as the 2.4 low-latency patch. Maybe better.

This is with ext2, and only with ext2. Other filesystems need work to reach that level of performance.

Non-preemptible kernels will benefit as well. This sort of means that preemptibility is only really needed for specialised multimedia/control type apps. Opinions vary ;)

Filesystem mount and unmount is a problem. Probably, this will not be addressed. People who have specialised latency requirements should avoid using automounters and those gadgets which poll CDROMs for insertion events.

This work has broken the shared pagetable patch - it touches the same code in many places. I shall put Humpty together again, but will not be including it for some time. This is because there may be bugs in this patch series which are accidentally fixed in the shared pagetable patch. So shared pagetables will be reintegrated when these changes have had sufficient testing.

Hugh, could you please closely review these changes sometime? Thanks.

I've needed this patch (well, this is a cleaned up version of what I used actually) for some time on PPC and on some embedded platforms. The issue that typically happens is when the kernel is booted with an IDE device still doing it's POST sequence (or just beeing reset, that is with no firmware or a firmware that doesn't wait for the device to be ready before booting the kernel).

The patch just waits up to 35 seconds (30 seconds per spec, plus a small margin to deal with a couple of bogus drives I saw that took 31 seconds) for the BUSY bit to go away on an HWIF.

It's mandatory in the IDE spec to pull-down D7 to ground on an inteface, so that an interface with no driver connected should return a value with bit BUSY 0x80 cleared, thus will not trigger this wait loop. I did a sanity check against 0xff anyway to deal with a couple of bogus interfaces I encountered though.

I don't expect this patch to break any existing working configuration, so please send to Linus for 2.5. If you accept it, I'll then send a 2.4 version to Marcelo as well. This have been around for some time and, imho, should really get in now.

There is a ton of stuff pending for 2.5 IDE. Unfortunately 2.5 isn't in a state I can do any usable testing so it will have to wait. The Marcelo 2.4 tree is current and I'm doing the work in 2.4 first now.

Rusty seems to have a lot of the module stuff in hand so hopefully I'll get back onto 2.5 after LCA

Well, actually, I'd like to see this patch in 2.4 asap too ;) It should apply "as is" with some offset.

As Eric W. Biederman noticed, it may not be enough for some really broken devices, but will not harm neither on these, and will fix the problem on a whole lot of better ones. It's definitely necessary with some WD hard disks and the "combo" DVD/CDRW drive shipped by Apple on some ibooks (Apple firmware typically does a reset of all drives just before booting the kernel, without waiting)

After a year in stasis, I've completely rebuilt my kernel patch that implements page coloring. Improvements include:

• Page coloring is now hardwired into the kernel. The hash queues now use bootmem, and page coloring is always on. The patch still creates /proc/page_color for statistics, but that will go away in time.
• Automatic detection of external cache size on many architectures. I have no idea if any of this code works, since I don't have any of the target machines. The preferred way to initialize the coloring is by passing "page_color=<external cache size in kB>" as a boot argument.
• NUMA-safe, discontig-safe

Right now the actual page coloring algorithm is the same as in previous patches, and performs the same. In the next few weeks I'll be trying new ideas that will hopefully reduce fragmentation and increase performance. This is an early attempt to get some feedback on mistakes I may have made.

lmbench shows no real gains or losses compared to an unpatched kernel; some of the page fault and protection fault times are slightly slower, but it's close to the rounding error over five lmbench runs.

Here are all the performance results I have for the patch:

1. Compile of 2.4.20 kernel with gcc 3.1.1 on 466MHz DS10 Alphaserver with 2MB cache: repeatable 1% speedup (573 sec vs. 579 sec)
2. 1000x1000 matrix multiply: 10% speedup on Athlon II with 512kB cache (Dieter N?tzel)
3. Without page coloring, the alpha gets 80% of max theoretical bandwidth for working sets at most 1/8 the size of its L2 cache. For larger working sets than that the achieved bandwidth is only 30%-50% of max. With page coloring, the 80% figure applies to the entire L2 cache.
4. FFTW (alpha): 30% speedup for 64k-point FFTs, 20% speedup for 1M-point FFTs

Patch is available at

www.boo.net/~jasonp/page_color-2.4.20-20030104.patch

What is the status of dmfs going into mainline?

I saw that Greg KH posted a patch with some corrections to dmfs for 2.5.50?

IMO it would be nice to have a kernel config option that makes the ioctl method optional when dmfs is set to y or m in kernel config. That will not only save a bit of code space, but it will also serve to encourage use of dmfs. :)

The last version I released is here:

http://people.sistina.com/~thornber/patches/2.5-unstable/2.5.51/2.5.51-dmfs-1.tar.bz2

There are still a couple of easy to fix issues with it (eg. the kmalloc while a spin lock is held that Andrew Morton pointed out :).

Both Andrew Morton and Greg KH expressed concerns with the way I've mapped the dm semantics onto the filesystem (http://marc.theaimsgroup.com/?l=linux-kernel&m=103975736919315&w=2). So Greg is currently trying to get a sysfs interface working.

We need to get a concensus of opinion in the community as to what is a good interface. I'm not going to be rushed into including something in dm that could cause critism for years to come. dmfs is what Alasdair Kergon and I have proposed, we're just waiting for an alternative to kick off the discussions ATM.

The sixth iteration of my devfs code shink is available here:

ftp://ftp.yggdrasil.com/pub/dist/device_control/devfs/smalldevfs-2.5.54-v6.patch

I believe the deletions make the patch so big that the linux-kernel mailing list filters prevent me from submit an email that includes it.

This patch reduces include/linux/devfs*.h and fs/devfs from 3655 lines to 1239, a reduction of 2450 lines, nearly a factor three. That may not be as impressive as the original 5X reduction, but that is mostly because I've restored a bunch of functionality that I hope to eliminate in the future.

I'd like to thank Richard Gooch for writing devfs. I think it was a great idea and the effort involved in implementing it, especially when it was not clear that it could work well, was probably about 30-100 times my effort in shrinking it. I immediately became a convert within a day of trying it. I'd be happy for Richard to take over this code and continue maintaining devfs. If he doesn't want to, I'm willing to and I'm also happy to let someone else do it if they want.

This is nearly the same patch that I attempted to post on January 2, but apparently some well intentioned spam filter blocked it. I had this problem once before, also when submitting a big patch with a lot of deletions sent as a MIME attachment. This time I'm submitting the patch as part of the text of my message.

The there are no code changes between this version and the one that I tried to post on January 2. In the meantime, I've used it and stared at it more, and now I'm posting this as a candiate for integration into Linus's kernel.

The January 2 version introduced two significant changes: isolating the filesystem driver to a separate file that only exports two symbols (devfs_vfsmount and init_devfs_fs), and making that patch a change to fs/devfs rather than a new filesystem. (If anyone would prefer that I submit this as a separate file system, please let me know.)

If you want devfsd functionality (well, at least the "REGISTER" and "LOOKUP" events), you can get my user level program devfs_helper, which is a reduced functionality replacement program for devfsd from the following URL.

ftp://ftp.yggdrasil.com/pub/dist/device_control/devfs_helper/devfs_helper-0.1.tar.gz

devfs_helper is program that is exec'ed on each event rather than being a daemon that waits on events. When the new module_param code is further developed, I will default the devfs_helper to be turned off until a user level program sets the name of the program.

Finally, I'd like to move forward toward getting this into Linus's kernel. Any blessings, curses, requests for changes or advice on the best way to proceed would be appreciated.

Great, glad to hear it.

I would suggest at least browsing Documentation/driver-model/*.txt and reading the appended document. (I've responded to several similar emails in the past, and realized there was no master document for describing this, so I decided to write one.)

Hopefully, this is useful to you and others. I'll be adding this to Documentation/driver-model/.

In general, the driver model infrastructure (and kobject infrastructure) are all about using generically defined objects and routines, rather than duplicating them again and again. Because of that, most of the changes happen at a high level (i.e. in the bus driver) rather than at the low-level (i.e. in the driver modules).

The appended document describes how to convert a bus driver to the new model, which covers the representation of devices and device drivers. It is a gradual process that can be done in several steps.

If you don't feel up to taking on an entirely unconverted driver, please feel free to help out with ones that are already underway, including:

• PCI
• USB
• SCSI
• IDE
• MCAM
• ISA

Alternatively, if someone doesn't feel up to converting the drivers, something that would be really handy would be a list of all bus types and device classes that the kernel supports, and the drivers that belong to each. It's not necessarily an easy list to compile, but again something that can happen gradually. ;)

Well, the ntfs driver from the 2.4.20 vanilla kernel has really dangerous write support for the ntfs partitions. It is strongly discouraged to use it. You can use though the backport driver from the 2.5 kernel series (aka ntfs-tng). It allows you to overwrite the files using mmap() and write(). So, neither size changes, nor attribute changes. You'll find mode detailes along with the driver itself at http://linux-ntfs.sf.net/.