Kernel Traffic #272 For 5�Sep�2004

Red Hat and (the former) Sistina Software are pleased to announce that we will host a two day kickoff workshop on GFS and Cluster Infrastructure in Minneapolis, July 29 and 30, not too long after OLS. We call this the "Cluster Summit" because it goes well beyond GFS, and is really about building a comprehensive cluster infrastructure for Linux, which will hopefully be a reality by the time Linux 2.8 arrives. If we want that, we have to start now, and we have to work like fiends, time is short. We offer as a starting point, functional code for a half-dozen major, generic cluster subsystems that Sistina has had under development for several years.

This means not just a cluster filesystem, but cluster logical volume management, generic distributed locking, cluster membership services, node fencing, user space utilities, graphical interfaces and more. Of course, it's all up for peer review. Everybody is invited, and yes, that includes OCFS and Lustre folks too. Speaking as an honorary OpenGFS team member, we will be there in force.

Tentative agenda items:

• GFS walkthrough: let's get hacking
• GULM, the Grand Unified Lock Manager
• Sistina's brand new Distributed Lock Manager
• Symmetric Cluster Architecture walkthrough
• Are we there yet? Infrastructure directions
• GFS: Great, it works! What next?

Further details, including information on travel and hotel arrangements, will be posted over the next few days on the Red Hat sponsored community cluster page:

http://sources.redhat.com/cluster/

Unfortunately, space is limited. We feel we can accommodate about fifty people comfortably. Registration is first come, first served. The price is: Free! (Of course.) If you're interested, please email me.

Let's set our sights on making Linux 2.8 a true cluster operating system.

The problems were mostly political. Maybe we tried to push too early, but 1-3 years back, people weren't really interested in agreeing on some common components or APIs. In particular a certain Linux vendor didn't even join the group ;-) And the "industry" was very reluctant too. Which meant that everybody spend ages talking and not much happening.

However, times may have changed, and hopefully for the better. The push to get one solution included into the Linux kernel may be enough to convince people that this time its for real...

There still is the Open Clustering Framework group though, which is a sub-group of the FSG and maybe the right umbrella to put this under, to stay away from the impression that it's a single vendor pushing.

If we could revive that and make real progress, I'd be as happy as a well fed penguin.

Now with OpenAIS on the table, the GFS stack, the work already done by OCF in the past (which is, admittedly, depressingly little, but I quite like the Resource Agent API for one) et cetera, there may be a good chance.

OK, what I've learned from the discussion so far is, we need to avoid getting stuck too much on the HA aspects and focus more on the cluster/performance side for now. There are just too many entrenched positions on failover. Even though every component of the cluster is designed to fail over, that's just a small part of what we have to deal with:

• Cluster Volume management
• Cluster configuration management
• Cluster membership/quorum
• Node Fencing
• Parallel cluster filesystems with local semantics
• Distributed Locking
• Cluster mirror block device
• Cluster snapshot block device
• Cluster administration interface, including volume managment
• Cluster resource balancing
• bits I forgot to mention

Out of that, we need to pick the three or four items we're prepared to address immediately, that we can obviously share between at least two known cluster filesystems, and get them onto lkml for peer review. Trying to push the whole thing as one lump has never worked for anybody, and won't work in this case either. For example, the DLM is fairly non-controversial, and important in terms of performance and reliability. Let's start with that.

Furthermore, nobody seems interested in arguing about the cluster block devices either, so lets just discuss how they work and get them out of the way.

Then let's tackle the low level infrastructure, such as CCS (Cluster Configuration System) that does a simple job, that is, it distributes configuration files racelessly.

I heard plenty of fascinating discussion of quorum strategies last night, and have a number of papers to read as a result. But that's a diversion: it can and must be pluggable. We just need to agree on how the plugs work, a considerably less ambitious task.

In general, the principle is: the less important it is, the more argument there will be about it. Defer that, make it pluggable, call it policy, push it to user space, and move on. We need to agree on the basics so that we can manage network volumes with cluster filesystems on top of them.

as most of you are probably aware of it, there have been complaints on lkml that the 2.6 kernel is not suitable for serious audio work due to high scheduling latencies (e.g. the Jackit people complained). I took a look at latencies and indeed 2.6.7 is pretty bad - latencies up to 50 msec (!) can be easily triggered using common workloads, on fast 2GHz+ x86 system - even when using the fully preemptible kernel!

to solve this problem, Arjan van de Ven and I went over various kernel functions to determine their preemptability and we re-created from scratch a patch that is equivalent in performance to the 2.4 lowlatency patches but is different in design, impact and approach:

http://redhat.com/~mingo/voluntary-preempt/voluntary-pree mpt-2.6.7-bk20-H2

(Note to kernel patch reviewers: the split voluntary_resched type of APIs, the feature #ifdefs and runtime flags are temporary and were only introduced to enable a easy benchmarking/comparisons. I'll split this up into small pieces once there's testing feedback and actual audio users had their say!)

unlike the lowlatency patches, this patch doesn't add a lot of new scheduling points to the source code, it rather reuses a rich but currently inactive set of scheduling points that already exist in the 2.6 tree: the might_sleep() debugging checks. Any code point that does might_sleep() is in fact ready to sleep at that point. So the patch activates these debugging checks to be scheduling points. This reduces complexity and impact quite significantly.

but even using these (over one hundred) might_sleep() points there were still a number of latency sources in the kernel - we identified and fixed them by hand, either via additional might_sleep() checks, or via explicit rescheduling points. Sometimes lock-break was necessary as well.

as a practical goal, this patch aims to fix all latency sources that generate higher than ~1 msec latencies. We'd love to learn about workloads that still cause audio skipping even with this patch applied, but i've been unable to generate any load that creates higher than 1msec latencies. (not counting driver initialization routines.)

this patch is also more configurable than the 2.4 lowlatency patches were: there's a .config option to enable voluntary preemption, and there are runtime /proc/sys knobs and boot-time flags to turn voluntary preemption (CONFIG_VOLUNTARY_PREEMPT) and kernel preemption (CONFIG_PREEMPT) on/off:

        # turn on/off voluntary preemption (if CONFIG_VOLUNTARY_PREEMPT)
        echo 1 > /proc/sys/kernel/voluntary_preemption
        echo 0 > /proc/sys/kernel/voluntary_preemption

        # turn on/off the preemptible kernel feature (if CONFIG_PREEMPT)
        /proc/sys/kernel/kernel_preemption
        /proc/sys/kernel/kernel_preemption

the 'voluntary-preemption=0/1' and 'kernel-preemption=0/1' boot options can be used to control these flags at boot-time.

all 4 combinations make sense if both CONFIG_PREEMPT and CONFIG_VOLUNTARY_PREEMPT are enabled - great for performance/latency testing and comparisons.

The stock 2.6 kernel is equivalent to:

voluntary_preemption:0 kernel_preemption:0

the 2.6 kernel with voluntary kernel preemption is equivalent to:

voluntary_preemption:1 kernel_preemption:0

the 2.6 kernel with preemptible kernel enabled is:

voluntary_preemption:0 kernel_preemption:1

and the preemptible kernel enhanced with additional lock-breaks is enabled via:

voluntary_preemption:1 kernel_preemption:1

it is safe to change these flags anytime.

The patch is against 2.6.7-bk20, and it also includes fixes for kernel bugs that were uncovered while developing this patch. While it works for me, be careful when using this patch!

Certainly 2.6+preempt is not as good as 2.4+LL at this time, but 2.6 isn't too bad either. Even under heavy filesystem load it's hard to exceed a 0.5 millisecond holdoff. There are still a few problem in the ext3 checkpoint buffer handling, but those seem pretty hard to hit. I doubt if the `Jack' testers were running `dbench 1000' during their testing.

All of which makes me suspect that the problems which the `Jack' testers saw were not directly related to long periods of non-preemption in-kernel. At least, not in core kernel/fs/mm code. There have been problem in the past in places like i2c drivers, fbdev scrolling, etc.

What we need to do is to encourage audio testers to use ALSA drivers, to enable CONFIG_SND_DEBUG in the kernel build and to set /proc/asound/*/*/xrun_debug and to send us the traces which result from underruns.

As for the patch, well, sprinkling rescheduling points everywhere is still not the preferred approach. But adding more might_sleep() checks is a sneaky way of making it more attractive ;)

Let me repeat that I am unconvinced as to the diagnosis of the current audio problems - more analysis might prove me wrong of course.

And I'm unconvinced that we need to do anything apart from identifying and fixing the remaining spinlocks which are holding off preemption for too long.

IOW, I am questioning the very need for a "voluntary preemption" feature at all when "involuntary preemption" works perfectly well.

the reason is difference in overhead (codesize, speed) and risks (driver robustness). We do not want to enable preempt for Fedora yet because it breaks just too much stuff and is too heavy. So we looked for a solution that might work for a generic distro.

here are the code size differences. With a typical .config (debugging options disabled), the 2.6.7-mm7(+voluntary-preempt) UP x86 kernel gets the following .text sizes:

orig:      1776911 bytes
   preempt:   1855519 bytes  (+4.4%)
   voluntary: 1783407 bytes  (+0.3%)

so if voluntary-preempt can get close to real preempt's numbers for practical stuff then we get most of the benefits while excluding some of the nastiest risks and disadvantages.

(Long-term i'd like to see preempt be used unconditionally - at which point the 10-line CONFIG_VOLUNTARY_PREEMPT Kconfig and kernel.h change could go away.)

ftp://ftp.kernel.org/pub/linux/kernel/people/ akpm/patches/2.6/2.6.8-rc1/2.6.8-rc1-mm1/

• Lots of little fixes, mainly

• Numerous scheduling latency fixes, mainly in the ext3 area.

  This is a first pass - these patches need redoing and a bit of infrastructure consolidation.

• Outta here: I won't be in a position to handle patches until July 26. Off to http://www.tech-forum.org/upcoming/open_source_software_06-10 -04.htm and then Kernel Summit and then OLS.

Who would do such a stupid thing ? Me the impatient trying to write before the drive ends to load the disc...

The bad thing is that it leaves the drive in an ususable state:

Error trying to open /dev/hdc exclusively (Device or resource busy)... retrying in 1 second.
Error trying to open /dev/hdc exclusively (Device or resource busy)... retrying in 1 second.

It does not happen with 2.6.8-rc2.

the attached patch limits the number of concurrent hotplug processes. Main idea behind it is that currently each call to call_usermodehelper will result in an execve() of "/sbin/hotplug", without any check whether enough resources are available for successful execution. This leads to hotplug being stuck and in worst cases to machines being unable to boot.

This check cannot be implemented in userspace as the resources are already taken by the time any resource check can be done; for the same reason any 'slim' programs as /sbin/hotplug will only delay the problem.

I dont know exactly which scenario Hannes has seen, but its quite simple to trigger this scenario with almost any s390/zSeries.

Using the Hardware Management Console or z/VM you are able to hotplug (deactive/activate/attach/detach) almost every channel path. As a channel path can connect a big bunch of devices lots of machine checks are triggered, which causes lots of hotplugs. The same amount of machine checks could happen if a hardware failure happens.

Some month ago I played around with a diet version of hotplug. This program was fast and small enough to make my scenario work properly. Nevertheless, as hannes already stated this will only delay the problem.

I'm interested in implementing something I'll call isolated sched domains for single cpus (to minimize the latencies involved when doing things like load balancing on certain select cpus) on IA64.

Below I've included an initial patch to illustrate what I'd like to do. I know there's been mention of 'platform specific work' in the area of sched domains. This patch only addresses IA64, but could be made generic as well. The code is derived directly from the current default arch_init_sched_domains code.

The patch isolates cpus that have been specified via a boot time parameter 'isolcpus=<cpu #>,<cpu #>'.

The attached patch is against 2.6.8-rc1-mm1. Tested on SMP, UP and SMP+HT here and it seems to be OK.

I have included the cpu_sibling_map for ppc64, although Anton said he did have an implementation floating around which he would probably prefer, but I'll let him deal with that.

Anyway, x86-64 is not equivalent before and after this patch. The main thing is that they've been using SD_CPU_INIT for NUMA nodes, but will now use SD_NODE_INIT. Probably neither is optimal, but I don't think Andi has had much time to look at it. I should be able to take a look at it soon.

Attached is the PAGG patch for kernel 2.6.7. Some may recall I posted an earlier PAGG patch for 2.6.7. This version has improved handling for the init function pointer functionality introduced earlier.

We'd be very interested in seeing this be accepted in to the kernel. If there is anything we should adjust to make it more likely to be accepted, please reply.

I'm working on BRLTTY[1], a user-space daemon which handles braille displays on UNIX platforms. One of our display drivers recently gained the ability to receive (set 2) scancodes from a keyboard connected directly to the display. This is a very cool feature, since the display in question has a bluetooth interface, making it effectively into a complete wireless terminal (input and output through the same connection).

However, this creates some problems. First of all, we now have to deal with keyboard layouts. Additionally, since we currently insert via TIOCSTI I think this might get problematic as soon as one switches to an X Windows console and modifiers come into play.

Does anyone know (and can point me into the right direction) if Linux has some mechanism to allow for user-space keyboard data to be processed by the kernel as if it were received from the system keyboard? I.e., keyboard layout would be handled by the same mapping which is configured for the system.

About modifiers, I submitted a patch to Dave to handle them properly.

But ascii to scancode translation still depends on scancode to ascii translation performed by the kernel indeed and the question still applies. I'll have a look at uinput.

Following patch implements the kernel events layer, which is a simple wrapper around netlink to allow asynchronous communication from the kernel to user-space of events, errors, logging, and so on.

Current intention is to hook the kernel via this interface into D-BUS, although the patch is intended to be agnostic to any of that and policy free.

D-BUS can be found here:

http://dbus.freedesktop.org/

Other user-space utilities (including code to utilize this) can be found here:

http://vrfy.org/projects/kdbusd/

This patch only implements a single event, processor temperature detection. Other useful events include md sync, filesystem mount, driver errors, etc. We can add those later, on a case-by-case basis. I would like to be more careful with adding events than we are with adding printk's, with some aim toward a stable interface.

Usage of the new interface is simple:

send_kmessage(group, interface, message, ...)

Credit to Arjan for the initial implementation, Kay Sievers for some updates, and the netlink code.

Predicting the future is hard, but I suspect this number to be small. Let's say 10 in core kernel code?

If this takes off as a solution for error reporting, that number will be much larger in drivers.

I would alsmost rather all possible messages get stuck somewhere common so driver writes can't add these ad-hoc and we can avoid a proliferation of either similar or pointless messages.

Forcing these into a common place lets people eyeball if a new messages really is necessary --- and it makes writing applications to deal with these things easier (since you don't have to scan the entire kernel tree).

I must say that my gut feeling here is that bolting an arbitrary new namespace into the kernel in this manner is not the way to proceed.

I hope we'll hear more from Greg on this next week - see if we can come up with some way to use the kobject/sysfs namespace for this.

Although heaven knows how "tmpfs just ran out of space" would map onto kobject/sysfs.

ketchup is a script that automatically patches between kernel versions, downloading and caching patches as needed, and automatically determining the latest versions of several trees. Available at:

http://selenic.com/ketchup/ketchup-0.8

New in this version by popular demand:

• falls back to .gz files if .bz2 files aren't available
• can find BK snapshots in old/ directories (aka the jgarzik memorial hack)
• option to rename directories to linux-<v> after update
• can read default options from KETCHUP_OPTS

Example usage:

$ ketchup 2.6-mm
2.6.3-rc1-mm1 -> 2.6.5-mm4
Applying 2.6.3-rc1-mm1.bz2 -R
Applying patch-2.6.3-rc1.bz2 -R
Applying patch-2.6.3.bz2
Applying patch-2.6.4.bz2
Applying patch-2.6.5.bz2
Downloading 2.6.5-mm4.bz2
Downloading 2.6.5-mm4.bz2.sign
Verifying signature...
gpg: Signature made Sat Apr 10 21:55:36 2004 CDT using DSA key ID 517D0F0E
gpg: Good signature from "Linux Kernel Archives Verification Key
<[email protected]>"
gpg: aka "Linux Kernel Archives Verification Key
<[email protected]>"
owner.
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the
Primary key fingerprint: C75D C40A 11D7 AF88 9981 ED5B C86B A06A 517D 0F0E
Applying 2.6.5-mm4.bz2

I know this is off-topic, but a court in my home town Munich has decided that a cease and desist letter Harald Welte sent to a router producer (Sitecom) who used netfilter/iptables in his router but didn't publish the sources of the firmware with a penalty of up to 100 000 Euro is valid (a German version of the decision of the three judges is at http://www.jbb.de/urteil_lg_muenchen_gpl.pdf).

It's quite nice to hear that a court has decided that the GPL is enforceable under German law.

The patch below enables the IBM Multiport Serial Adapter for the Linux OS. This driver is for a family of multiport serial adapters including 2 port RVX, 2 port internal modem, 4 port internal modem and a split 1 port RVX and 1 port internal modem. We have applied & tested this patch against our iSeries and pSeries machines (there are no known existing defects following this testing).

We would like this code accepted into the kernel. It was previosly submitted to the linux-serial mailing list, and we believe we have addressed any coding issues raised. Are there any additional concerns? How do you suggest we proceed to have this code accepted?

Looks sane. Did you really intend that it be buildable for all architectures? (There's nothing wrong with this - it's good. But it's a bit unusual for this sort of driver).

"IBM multiport serial adapter" seems to be a rather generic description of this device. Is this the only multiport serial adapter which IBM has ever, or will ever make? If not, then perhaps we could make the description a little more specific?

I have attached a patch for implementing IRQ handlers in threads, for latency-reduction purposes. It requires that softirqs must be run in threads (or else they could end up running inside the IRQ threads, which will at the very least trigger bugs due to in_irq() being set). I've tested it with Ingo's voluntary-preempt J7 patch, and it should work with the TimeSys softirq thread patch as well (though you might get a conflict with the PF_IRQHANDLER definition; just merge them into one).

Some notes:

1. This may not work properly with some interrupt controller code, which doesn't do the obvious thing with mask_and_ack() and end(). This includes the IO-APIC code, which has an empty end() for edge triggered interrupts and an empty mask_and_ack() for level-triggered interrupts. The mask_and_ack() needs to really mask the interrupt, as otherwise the hardware will not deliver lower-priority (to it) interrupts, which may have a higher-priority thread.
2. This patch does not disable local interrupts when running a threaded handler. SMP-safe drivers shouldn't be directly bothered by this (as the interrupt could as easily have happened on another CPU), but there may be some interactions with softirqs and per-cpu data, if a softirq thread preempts an IRQ thread, or an IRQ thread gets migrated to a different CPU. I'm particularly worried about the network code. If possible, I'd like to find and fix such breakages rather than use local_irq_disable(), as that would prevent IRQ proritization from working, and prevent IRQ threads from being used to isolate the rest of the system from long-running IRQs (such as non-DMA IDE).
3. The i8042 driver had to be marked SA_NOTHREAD, as there are non-preemptible regions where it spins, waiting for an interrupt. Ideally, this driver (and others like it) should be fixed to either do a cond_resched() or use a wait queue.
4. This might be a good time to get around to moving the bulk of the arch/whatever/kernel/irq.c into generic code, as the code said was supposed to happen in 2.5. This patch is currently only for x86 (though we've run IRQ threads on many different platforms in the past).
5. Is there any reason why an IRQ controller might want to have its end() called even if IRQ_DISABLED or IRQ_INPROGRESS is set? It'd be nice to merge those checks in with the IRQ_THREADPENDING/IRQ_THREADRUNNING checks.
6. This patch causes in_irq() to return true if an IRQ thread is running, as some drivers use it in common code to determine how to act. in_interrupt(), however, will return false in such a case. The exact meaning of these macros in the presence of IRQ threads isn't very well defined, and I hope this results in sane behavior.

Wouldn't the IRQ threads be subject to the same heuristics that the scheduler uses with ordinary threads, in order to avoid unnecessary CPU migration? Plus, IRQs ordinarily get distributed across CPUs, and in most cases shouldn't have a very large cache footprint (especially data; the code can be in multiple CPU caches at once), so I don't think this is a susbtantial degradation from the way things already are.

If desired by the user, an IRQ thread could be bound to a specific CPU to avoid such problems (in which case, they'd probably want to set the smp_affinity of the hard IRQ stub to the same CPU).

My experience with clients who have been using TimeSys' stuff has been abysmal. The fact of the of the matter is that most people who used this were practically locked-in to TimeSys' services, unable to download anything "standard" off the net and using it with their kernel. In one example, we had to ditch the kernel the client got from TimeSys because we had spent 10+ hours trying to get LTT to work on it without any success whatsoever.

As I had said on other lists before, I don't see the point of creating that much complexity in the kernel in order to try to shave-off a little bit more off of the kernel's interrupt response time. The fact of the matter is that neither this patch nor most of the other patches suggested makes the kernel truely hard-rt. These patches only make the kernel respond "faster". If you really need hard-rt, then you should be using the Adeos nanokernel. With Adeos, you can even get a hard-rt driver without using RTAI or any of the other rt derivatives.

there's really two camps that are emerging in the real time Linux field, dual and single kernel. The single kernel work that's current being done could very well get Linux to being hard RT, assuming that you solve all of the technical problems with things like RCU, etc... in 2.6.

The dual kernels folks would be in less of position to VAR their own stuff and sell proprietary products if Linux were to get native hard RT performance if you accept that economic criteria. Who knows what the actual results will be.

It could be that all of this work with Linux could bury prioprietary OS product (such as LynxOS here) or it could open doors to other things unknown things that were never possible previous to Linux getting some kind of hard RT capability. It's certainly a scary notion to think about with many variables to consider. Linux getting hard RT is inevitable. It's just a question of how it'll be handled by proprietary OS vendors, witness IBM for a positive example. A negative one would be Sun.

Now that Windriver System (the idiot folks that never understood Linux before laying off tons of folks and disbanned the rather famous BSD/OS group which I was apart of, etc...) and Red Hat is in the picture, it's all starting to cook up.

The hard RT people I know of and work with want to be able 1) to get microsecond level bounded interrupt latency with no exception to this rule, and 2) to be able to choose the right level of dispatch latency on a thread-by-thread basis, from a few microseconds to a few hundreds of, but in any case _bounded_ and predictable in the worst case. For this to happen, they are willing to accept stringent limitations functionaly-wise if need be to obtain the first, but still get access to the regular Linux programming model and APIs if the second fits their apps. They already know how they could mix both properly in what would look like a single system from the application pov.

For these people, the current undergoing work aimed at improving the current determinism of the vanilla kernel is everything but a danger: it's fundamental and very good news, because it could make 2) a reality sooner or later. However, point 1) remains an issue, and unless you find a solution for mixing fire and water, i.e. determinism which requires unfairness by design and throughput seeking fairness on the average case, you would likely end up considering that the Linux RT people's radical approach of using a dual-kernel does not make them uneducated bozos (Ok, except me perhaps, but this is not my point). To get microsecond level guaranteed interrupt latencies, the problem is far beyond solving random latency spots here and there: it's an architectural issue.

To achieve this, we (i.e. the educated ones like Karim helping the uneducated bozo like myself; yep, this is a teamwork) have come to the conclusion that we needed a portable infrastructure that allows a complete prioritization of interrupts, and hw events of interest in general (e.g. traps/exceptions). Some infrastructure that exposes the same interface regardless of the platform it runs on. It's called Adeos, the source code identation is terrible (after 20 years practicing it, I still find means to worsen my coding style, funky eh?!) but it's a working example of such kind of infrastructure. The advantage of such kind of thin layer is that you can plug any hard real-time core over it. This layer can remain silent when unused, it can be configured out, it is just an enabler. You don't have to put the hell of a havoc in a stable GPOS core to modify some key architectural characteristics of the Linux kernel in order to buy hard RT capabilities to everyone, which could be construed as smashing a squadron of flies with nukes.

There is no such thing as hard-RT in the real world.

In reality, there's no point in making the software far more reliable than the hardware, power supply, and so on. Somebody may pour a can of Mountain Dew into the vent holes.

Your software is OK as long as other causes of failure are much more likely. One might even say you spent too much of your budget perfecting the software! In the end it all comes down to $$$ (or Euros, or Yen...), doesn't it?

People don't mathematically demonstrate anything about modern systems, at least not while being honest.

loop-AES changes since previous release:

• Adapted and merged Russell King's loop.c flush_dcache_page() fix. Most sane processors were not affected, but some processors with goofy aliasing caches were indeed affected (2.4 and 2.6 kernels).
• Added optimized assembler implementations of AES and MD5 functions for AMD64 and compatible processors.
• Pentium-2 optimized assembler implementations of AES and MD5 are really i386 compatible, so now those assembler implementations are enabled for all x86 processors.
• Fixed Makefile to be compatible with distros that include "" characters in KERNELRELEASE string.
• Added dkms.conf configuration file for Dynamic Kernel Module Support. Charles Duffy wrote original version.
• Added support for /lib/modules/`uname -r`/source symlink.
• Converted MODULE_PARM macros to module_param (2.6 kernels only).
• Added workaround for scripts/modpost breakage (2.6 kernels only).

bzip2 compressed tarball is here:

http://loop-aes.sourceforge.net/loop-AES/loop-AES-v2.1c.tar.bz2
md5sum b404d9d679b7096dd3fb089345c52320

http://loop-aes.sourceforge.net/loop-AES/loop-AES-v2.1c.tar.bz2.sign

Additional ciphers package changes since previous release:

• Fixed Makefile to be compatible with distros that include "" characters in KERNELRELEASE string.
• Added dkms.conf configuration file for Dynamic Kernel Module Support. Adapted from version that was originally written by Charles Duffy.
• Added support for /lib/modules/`uname -r`/source symlink.
• Added workaround for scripts/modpost breakage (2.6 kernels only).

bzip2 compressed tarball is here:

http://loop-aes.sourceforge.net/ciphers/ciphers-v2.0i.tar.bz2
md5sum 1a5e1d967bca0cde71a32e533ef26ce9

http://loop-aes.sourceforge.net/ciphers/ciphers-v2.0i.tar.bz2.sign