Kernel Traffic #211 For 30�Mar�2003

We've been working on a gateway between BitKeeper and CVS to provide the revision history in a form which makes the !BK people happy (or happier).

We have the first pass of this completed and have a linux 2.5 tree on kernel.bkbits.net and you can check out the tree as follows (please don't do this unless you are a programmer and will be using this. Penguin Computing provided the hardware and the bandwidth for that machine and if you all melt down the network they could get annoyed. By all means go for it if you actually write code, though, that's why it is there.)

    mkdir ws
    cd ws
    cvs -d:pserver:[email protected]:/home/cvs co linux-2.5

Each of the releases are tagged, they are of the form v2_5_64 etc.

Linus had said in the past that someone other than us should do this but as it turns out, to do a reasonable job you need BK source. So we did it. What do we mean by a reasonable job? BitKeeper has an automatic branch feature which captures all parallel development. It's cool but a bit pedantic and it makes exporting to a different system almost impossible if you try and match what BK does exactly. So we didn't. What we (actually Wayne Scott) did was to write a graph traversal alg which finds the longest path through the revision history which includes all tags. For the 2.5 tree, that is currently 8298 distinct points. Each of those points has been captured in CVS as a commit. If we did our job correctly, each of these commits has the same timestamp across all files. So you should be able to get any changeset out of the CVS tree with the appropriate CVS command based on dates.

We also created a ChangeSet file in the CVS tree. It has no contents, it serves as a place to capture the BK changeset comments. Each file which is part of a changeset has an extra comment which is of the form

        (Logical change 1.%d)

where the "1.%d" matches the changeset rev. So you can look for all files that have (Logical change 1.300) in their comments to reconstruct the changeset. NOTE! That information is actually redundant, the timestamps are supposed to do the same thing, let us know if that is not working, we'll redo it. I expect we'll find bugs, please be patient, it takes 4 hours of CPU time on a 2.1Ghz Athlon to do the conversion, that's a big part of why this has taken so long. That's after a week's worth of optimizations.

Each ChangeSet delta has a BK rev associated with it in the comments. We'll be giving you a small shell script which you can use to send Linus patches that include the rev and we'll modify BK so that it can take those patches with no patch rejects if you used that script.

We have a first pass of a real time gateway between BK and this CVS tree done. Right now it is done by hand (by me) but as soon as it is debugged you will see this tree being updated about 1-3 minutes after Linus pushes to bkbits.

Once you guys look this over and decide you like it, we'll do the same thing for the 2.4 tree.

We're also talking to an unnamed (in case it doesn't work out) Linux company who may host bkbits.net for us. If they do that, we'll turn the GNU patch exporter feature in BKD. That means that you'll be able to wget any changeset as a GNU patch, complete with checkin comments. I'm working with Alan on the format, I think we're close though I have to run the latest version past him.

If all of this sounds nice, it is. It was a lot of work for us to do this and you might be wondering why we bothered. Well, for a couple of reasons. First of all, it was only recently that I realized that because BK is not free software some people won't run BK to get data out of BK. It may be dense on my part, but I simply did not anticipate that people would be that extreme, it never occurred to me. We did a ton of work to make sure anyone could get their data out of BK but you do have to run BK to get the data. I never thought of people not being willing to run BK to get at the data. Second, we have maintained SCCS compatible file formats so that there would be another way to get the data out of BK. This has held us back in terms of functionality and performance. I had thought there was some value in the SCCS format but recent discussions on this list have convinced me that without the changeset information the file format doesn't have much value.

Our goal is to provide the data in a way that you can get at it without being dependent on us or BK in any way. As soon as we have this debugged, I'd like to move the CVS repositories to kernel.org (if I can get HPA to agree) and then you'll have the revision history and can live without the fear of the "don't piss Larry off license". Quite frankly, we don't like the current situation any better than many of you, so if this addresses your concerns that will take some pressure off of us.

Another goal is to have the freedom to evolve our file formats to be better, better performance and more features. SCCS is holding us back. So you should look hard at what we are providing and figure out if it is enough. If you come back with "well, it's not BitKeeper so it's not enough" we'll just ignore that. CVS isn't BitKeeper. On the other hand, we believe we have gone as far as is possible to provide all of the information, checkin comments, data, timestamps, user names, everything. The graph traversal alg captures information at an extremely fine granularity, absolutely as fine is possible. We have 8298 distinct points over the 2.5.0 .. 2.5.64 set of changes, so it is 130 times finer than the official releases. If you think something is missing, tell us, we'll try and fix it.

The payoff for you is that you have the data in a format that is not locked into some tool which could be taken away. The payoff for us is that we can evolve our tool as we see fit. We have that right today, we can do whatever we want, but it would be anywhere from annoying to unethical to do so if that meant that you couldn't get at the data except through BitKeeper. So the "deal" here is that you get the data in CVS (and/or patches + comments) and we get to hack the heck out of the file format. Our changes are going to move far faster than CSSC or anyone else could keep up without a lot of effort. On the other hand, our changes are going to make cold cache performance be much closer to hot cache performance, use a lot less disk space, a lot less memory, and a lot less CPU.

before Linus started using BK, close to 50% of the revision data that is now saved was completely lost in the process of him merging patches by hand into his repository. I mean be realistic, do you think that Linus kept perfect track of EVERY single ' ' ';' ')' that he changed when merging a patch with minor rejects with his repo? Do you think that every single time that he made a 1 line change or merged a 1 line change that was sent to this list it was documented and recorded? I doubt it. So now we are able to get a publicly available CVS repository with close to two times the data that was ever available before, and infinitely more than was ever available to anyone outside of Linus' own head.

I personally think that Larry has done an amazing job supporting this project and it's goals, and I will give him a big "Thanks for all your support and hard work" at this time. I think that those of you complaining about this as bitmover clearing the road to steal our data should take a long hard look at what you are really saying and consider what BK has given us that we never had before because nothing before was ever usable by Linus.

The ChangeSet file has many csets and we only capture around 1/2 of them in CVS ChangeSet file. The extra ChangeSets are grouped together with the merge cset where they were added to the path we are recording. That is correct, but it is not the whole story.

What happens is that most csets modifiy a non overlapping set of files. So while we didn't get every delta to the ChangeSet file, we did capture >90% of the actual changes to the source files in the tree.

You are giving us approximately 90% of our data in exchange for the one thing that made using bitkeeper not a total sellout; the fact that the revision history of the repo was still accessible without proprietary software.

I honestly appreciate the work that you and BitMover do for the kernel, but not giving us access to 100% of _our_ data is unacceptable to me. Quite honestly, I think your move is to restrict the possible alternatives to the BK client (the CSSC based ones like I and others had done), which were able to extract 100% of the data, even if they couldn't make use of it in the same way as bitkeeper. Atleast it was there.

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier to get away with locking the revision history into a proprietary format.

As long as we continue to get all the data in an open format, I'm not sure this really matters, personally. If there's some data loss, let's focus on that issue ... but it seems there isn't at the moment.

I'd rather we *didn't* go trying to clone BK and make it file-format compatible underneath ... that seems more incendiary than useful. Cloning other products is always a loosing game, the best you can do is catch them. Personally, I'd prefer we spent the effort making a usable simple SCM that 95% of us can use that does merges and stuff, and not bother trying to follow someone else in file format.

Of course, I'm in no position to dictate to others what they should implement, do what you like ... just my personal opinion. But there's always the possiblity we can make something that fits kernel development *better*, rather than playing catchup to BK all the time ;-)

My personal opinion is that BK maps only so so well onto the kernel development effort. It's not horrible, it's closer than any other SCM, but it could be better. The kernel guys tend to be "more loose" than commercial guys, i.e., stuff is tried, it sits in Alan's tree for a while or DaveJ's tree and then is rejected if it is found to be bad. You really need a sort of "lossy" SCM system, one which is willing to throw data away. BK is absolutely not about losing information, we view everything as valuable, even bad ideas. That matches the commercial world better than the Linux world.

I _think_ that Arch is closer. You will definitely give up some stuff if you move to Arch but you will also gain some stuff. Arch is willing to pick and choose, we aren't, we're sort of an all or nothing answer. Pavel is all hot and bothered about PRCS but PRCS is sort of BK without the distribution, gui tools, and scripting. It's a step backwards as far as I can tell (don't get me wrong, we've acknowledged the coolness of PRCS on our website for years and I tried to team up with Josh, I'm a fan). You should really look at Arch, it may be a better fit. And these days, if you could find a better fit, none of us at BitMover would shed a tear if you moved off BK. This has *not* been a pleasant experience for us.

You shouldn't care less of the disk format. You *can't* run bk in the first place to reach those files, it's by pure luck that somebody is been fine to give away his right to write free software (oh and proprietary software too but we don't care :) in the SCM arena and to provide this info to you via rsync or whatever proxy or open protocol that tytso mentioned is doable.

Before you can remotely care about the disk format you've to reverse engeneer the network protocol first, having more proprietary stuff there won't make differences for us. And of course it makes perfect sense for Larry to hide the stuff better, but even if he encrypts it, the secret key has to be in the bk binary, I mean, it's all in open source assembly anyways, if you figured out the network protcol, you shouldn't have an order of magnitude more of troubles to figure out the new file format too. NOTE: I don't want to discuss the legal details of reading the open source assembly, this was only an example ;).

really, what we care is the data, and what I discussed in the last weeks with Larry about the kernel CVS at first sigh seems enough for kernel developers, what matters is the _mainline_ evolution. All other trees matters much less (and NOTE: all important non mainline trees don't use bitkeeper anyways). If getting the changesets with dates will be too hard I assume Larry could help on it. Some script should do it pretty well thanks to the logical tag in the log. I know it's not the most useful format for export but this is reliable, documented and open and it makes it trivial to checkout and search the file logs. which makes it very usable immediatly w/o the need of new software which is good for us kernel developers in the short term. This is a good short/mid term solution.

IMHO cloning bitkeeper would be an option if Larry would be supporting it, but that is obviously not the case.

There is no point to complain about the change of format of files in the Larry has all the rights to change the file format even after you reverse engeneered it the first second third fourth time, so all your effort will break in seconds. You can spend the rest of your life to keep up with Larry and he'll always be ahead of you. We have to do this with the SMB protocol because there's no open ""exporter"", but here Larry provided the data, and the data belongs to the community in the first place so there's no need to slowdown innovation here trying to catch up with closed proprietary protocols. And note: if you don't like that linux is developed with bk you should speak with Linus not with Larry. That is Linus's choice, Larry couldn't make that change.

If you complain about the file format change, it means you realized right now you did a mistake in depending on bk in the first place.

I think we reached a point of balance here that will solve all the collisions. The CVS is a "stability" point. The lack of data-availability with CVS or similar open protocol would force us to reverse engeneer bk to access the data, and the availability of CVS immediatly make us wasting time reverse engeneering bk. Cloning bitkeeper is a waste of time if the CVS just exports the data correctly.

Please focus on this: the only thing we miss is the visibility of the jfs tree and similar other bits that aren't even guaranteed to be merged in mainline. But that doesn't worry me at all, in one year from now if the jfs tree didn't merge correctly it won't matter what was in such dead tree.

If you want to contribute, stop these threads, and start importing CVS into a more powerful SCM and let us know an URL where we can access the data from there. I will only answer to a working URL, either that or live with CVS. The SCM can be evolved over tiem. If this new underground domain will be better than bitkeeper than jfs and Linus as well could join us in the future. In the meantime CVS will do fine and it guarantees the openess of the linux info. As you probably know I don't have much time in helping with SCM developement by I can t try give my $.02 (or at the very least I want to be still allowed to give my $.02! ;).

I won't answer further emails about this issue to avoid hurting the l-k traffic too much (the last bk threads even made me overlook the BK->CVS announcement after a one day and half of email backlog go figure ;).

And of course many thanks to Larry for the BK->CVS effort! While I think it was due, it certainly takes some relevant effort to do it.

I've been very happy with BK, been using it shortly after Linus started doing so. Mostly out of curiosity at first, later because it was actually quite useful. I even see myself as a fairly pragmatic individual, but even so I do find it increasingly difficult to defend my BK usage.

So please stop thinking you can judge that easily by pushing me into your nice little 'some people will never be happy bla bla' category.

From what I can gather, the question is very simple:

"Can we get our data out of BK into some kind of open format?"

It's an important question. If the answer is "yes, but only the stuff that can be mapped onto CVS" then that's a significant data loss, and if BitMover changes the data format without documentation, then there is no longer a way to get all the data out.

Presumably the CVS exporter could get augmented with some kind of metadata export... perhaps an XML schema that describes how the various points are to be linked or whatnot... it won't turn CVS into BK overnight (so Larry can still sleep at night), but it would give BitMover the freedom to change their data format.

bk export -tpatch -r1.900 > patch.1.900
bk changes -v -r1.900 > comments.1.900

Been there forever. So has ways to get all the metadata from the command line without having to reverse engineer the file format. See

http://www.bitkeeper.com/manpages/bk-prs-1.html

it's all there. Always has been.

Wayne wanted me to point that it is easy to write the BK to CVS exporter completely from the command line, we prototyped it that way, the only reason we rewrote part of it in C was for performance. The point being that you guys could have done this yourself without help from us because all the metadata is right there. Ditto for anyone else worried about getting their data out of BK now or in the future. The whole point of prs is to be able to have a will-always-work way to get at the data or the metadata, it makes the file format a non-issue.

More importantly, even if someone isn't allowed to use the BK command line tool because once upon time, a long time ago, they submitted a patch to arch or subversion, they can still find someone is allowed to set up a bk daemon under the terms of the FUL, connect to the BK daemon using a http client, and extract the full diff of any changeset that way. This doesn't have to be the bkd on bkbits.net; anyone who is authorized to use BK under the terms of the FUL can set up a bk daemon to be listening on a port of any machine for which they have shell access (it doesn't even require root privs). And every last changeset can always be made available using this path.

So to the people are complaining that they won't be able to get out their data if a future version of BK uses a more powerful representation than SCCS files ---- would you like some more whine with your cheese?

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.64/2.5.64-mm8/

• Several fixes to the anticipatory scheduler. It is the default IO scheduler again.

  The main thing which was fixed here was an interesting deadlock involving keventd, the I/O scheduler, vfork and request_module().

• I should have mentioned that 2.5.64-mm7 included a CPU scheduler tweak from Mike Galbraith which apparently fixes up the various starvation problems which people have been experiencing. That is also in 2.5.64-mm8.

the attached patch fixes a fundamental (and long-standing) bug in the sleep-average estimator which is the root cause of the "contest process_load" problems reported by Mike Galbraith and Andrew Morton, and which problem is addressed by Mike's patch.

the bug is the following: the sleep_time code in activate_task() over-estimates the true sleep time by 0.5 jiffies on average (0.5 msecs on recent 2.5 kernels). Furthermore, for highly context-switch intensive and CPU-intensive workloads it means a constant 1 jiffy over-estimation. This turns the balance of giving and removing ticks and nils the effect of the CPU busy-tick, catapulting the task(s) to highly interactive status - while in reality they are constantly burning CPU time.

the fix is to round down sleep_time, not to round it up. This slightly under-estimates the sleep time, but this is not a real problem, any task with a sleep time in the 1 jiffy range will see timekeeping granularity artifacts from various parts of the kernel anyway. We could use rdtsc to estimate the sleep time, but i think that's unnecessary overhead.

the fixups in Mike's scheduler patch (which is in -mm8) basically work around this bug. The patch below definitely fixes the contest-load starvation bug, but it remains to be seen what other effects it has on interactivity. In any case, this bug in the estimator is real and if there's any other interactivity problem around then we need to deal with it ontop of this patch.

this bug has been in the O(1) scheduler from day 1 on basically, so i'm quite hopeful that a number of interactivity complaints are fixed by this patch.

The Linux 2.2 and Linux 2.4 kernels have a flaw in ptrace. This hole allows local users to obtain full privileges. Remote exploitation of this hole is not possible. Linux 2.5 is not believed to be vulnerable.

Linux 2.2.25 has been released to correct Linux 2.2. It contains no other changes. The bug fixes that would have been in 2.2.5pre1 will now appear in 2.2.26pre1. The patch will apply directly to most older 2.2 releases.

A patch for Linux 2.4.20/Linux 2.4.21pre is attached. The patch also subtly changes the PR_SET_DUMPABLE prctl. We believe this is neccessary and that it will not affect any software. The functionality change is specific to unusual debugging situations.

We would like to thank Andrzej Szombierski who found the problem, and wrote an initial patch. Seth Arnold cleaned up the 2.2 change. Arjan van de Ven and Ben LaHaise identified additional problems with the original fix.

The dumpability is per mm, which means that you have to consider all the cases of a thread being created in parallel to dumpability being enabled.

So consider a three threaded process. Thread one triggers kernel thread creation, thread two turns dumpability back on, thread three ptraces the new kernel thread.

Proving that is safe is non trivial so the current patch chooses not to attempt it. For 2.4.21 proper someone can sit down and do the needed verification if they wish.

This patch really breaks UML using the skas mode of thread tracing skas3 patch on quite a significant amount of machines out there. The skas mode is a lot more secure than the traditional UML tt mode. I guess this is related to the below...

I am running a UML site that a lot of hospitals ad clinics in Bangldesh depend on for there email. It allows them to work around the corruption and agrandidement of the ISPs over there. The skas3 mode patch is needed for the site to run securely. Tracing thread mode does not cut it.

There are also a large number of other telehoused ISP virtual hosting machines that use this stuff, and it is actually proving to be quite reliable.

I have attached the skas3 patch that Jeff Dike is currently using, and the patch that you have produced. Could you please look into the clash between them, and get it fixed.

Thank you - there are lots out there who will appreciate this.

So what happens now?

Is this critical enough for 2.4.21 to go out? Or can it wait like some other fairly serious stuff such as the ext3 fixes? What about the current state of IDE?

Would it make sense to repackage 2.4.20 into something like 2.4.20-p1 or 2.4.20.1 with only the critical stuff applied?

There shouldn't be a huge need to rush 2.4.21 as-is, really. If you want an immediate update, get the fix from your vendor.

Plus, it's a local root hole, and there are still tons of those left out there to squash.

At some point it probably would make sense to start deprecating .gz format files from kernel.org.

I am envisioning this as a three-phase changeover:

a) Get all mirrors to carry .bz2 format. This would affect the following sites:

DUTH:format=gz
GARBO:format=gz
HCMC:format=gz
IGLU:format=gz
LINUXAID:format=gz
LLARIAN-NET:format=gz
MINET-FR:format=gz
NC-ORC:format=gz
PCSS:format=gz
PROGRAMVAREVERKSTEDET:format=gz
PUB-FTP-UNIVERSITY-OF-OLDENBURG:format=gz
RN-RNO:format=gz
TASK:format=gz
TELEPAC:format=gz
TENGU-EASYNET-FR:format=gz
UNC-METALAB:format=gz
WEBLAB:format=gz

b) Once that is done, change the robots to no longer require .gz files; .bz2 files uploaded would be signed but no .gz file would be generated.

-> If we get a complete loss of data here, all .gz files would be lost.

c) At some point, deprecate .gz uploads entirely and remove all the old .gz files. After that point .gz files uploaded would be treated just like .Z, .zip or any other "unmanaged" compression format.

Now, the questions that come up are:

i) Does this sound reasonable to everyone? In particular, is there any loss in losing the "original" compressed files?

ii) Assuming a yes on the previous question, what time frame would it make sense for this changeover to happen over?

there is at least one reason for the "original" .gz files. Here are the logical steps:

a) any Linux distribution contains their own "linux" package with the source base being "vanilla" Linux .tar.gz file

b) switching such to .tar.bz2 will make building the package longer because of longer extract times

c) re-running tar to generate a .tar.gz from .tar.bz2 and store the .tar.gz instead will make customers suspicious --- i.e. they will have to ask "is this _really_ a plain Linux tree or do I need to run diff(1) to verify, just in case?"

See the reasoning? However, I agree that this reason is very weak. But you were interested in any reasons, including weak ones

Personally I fetch the .bz2 tar files for a few base kernel versions, but I fetch the .gz patch files.

This is so that I can "zgrep" through the patch files looking for which version changed some feature or API.

bzgrep exists, but it is way too slow.

So if there were only .bz2 patch files, I would fetch them and convert them back into .gz files on my local mirror.

Which is ok of course, but then the signatures don't match any more.

Well, that's my really weak reason for liking .gz patches.

OK, there seems to be enough resistance to this to put it off for a year or two. Since that means I don't have to do any work, that plan has inherent appeal to me.

In other words, the current setup will remain for now. HOWEVER, I would like to recommend that mirror sites start carrying .bz2 files if they want to carry only one format.

ftp://ftp.<country>.kernel.org/pub/linux/utils/kernel/ksymoops/v2.4

ksymoops-2.4.9.tar.gz           Source tarball, includes RPM spec file
ksymoops-2.4.9-1.src.rpm        As above, in SRPM format
ksymoops-2.4.9-1.i386.rpm       Compiled with 2.96 20000731, glibc 2.2.5
patch-ksymoops-2.4.9.gz         Patch from ksymoops 2.4.8 to 2.4.9.

Changelog extract

• For code lines that dump before EIP and have variable length instructions, decode in two chunks with suitable headings.
• Fix broken mips64 address mapping. Maciej W. Rozycki.
• Pass more mips registers. Maciej W. Rozycki.
• Add INSTALL note about broken distributions. Reported by James W. Laferriere.

The change to decode the Code: line in two chunks allows architectures that have variable length instructions to safely dump the code before eip. The code from eip onwards is always reliable, the code before eip may not be reliable, this is reflected in the headings before each chunk of decode output.

Updating ksymoops alone has no effect on the decode. The arch specific kernel dump routine must :-

(a) Add the string " VLI" to the line containing the eip/psr/pc/ip/psw to tell ksymoops that this dump has variable length instructions. For example EIP: 0060:[<c014fedf>] VLI Not tainted

(b) Dump bytes before and after eip, on one code line. ksymoops handles up to 64 bytes of Code: data.

(c) Enclose the eip byte in <> or ().

If you do not want to decode before eip with variable length instructions, do not change your arch tree. ksymoops will continue to decode as before.

Architectures that already dump code before the eip do not need to be changed. AFAIK they all have fixed length instructions.

Some people have reported problems building ksymoops, with unresolved references in libbfd (htab_create, htab_find_slot_with_hash). Try http://www.cs.helsinki.fi/linux/linux-kernel/2002-13/0196.html first, if that does not work, contact the binutils maintainers. This is not a ksymoops problem, ksymoops only uses libbfd. Any unresolved references from libbfd are a binutils problem.

Here are some more i2c driver cleanups. These do the following items I mentioned in my last set of patches:

• clean up #ifdef mess in i2c controllers
• fix the printk() calls to use proper levels
• add i2c controller driver core support

There is also a i2c spelling patch in here, and I've added the i2c-isa driver, which almost isn't even a driver at all, based on the size of it.

Even with adding a new driver, the sum of these patches is still a smaller number of lines than we started with, which is always nice.

Please pull from: bk://kernel.bkbits.net/gregkh/linux/i2c-2.5

Things left to do after this:

• clean up i2c driver core support to work better.
• add i2c device core support.
• port remaining drivers from i2c cvs to kernel tree.

lk-changelog.pl aka. shortlog version 0.83 has been released.

This script is used by Linus and Marcelo to rearrange and reformat BK ChangeSet logs into a more human-readable format, and the official repository is bk://gkernel.bkbits.net/BK-kernel-tools/

The changes are listed at the end of the script below.

You can always download this script and GPG signatures from http://mandree.home.pages.de/linux/kernel/

My thanks go to Vitezslav Samel who has spent a lot of time on digging out the real names for addresses sending in BK ChangeSets.

LSE Tech Call Minutes March 21, 2003

Minutes Compiled by Hanna Linder. Please send corrections or additions to [email protected]

I. Matthew Fanto: The Cello Disk Scheduling Framework

If you have seen on the lkml the schedulers being worked on, like: cfq, deadline, and anticipatory io schedulers. Cello is another one. It is a two level disk scheduler.

The two levels are a request is made and is put into a class specific queue. Currently there are three different applications classes:

real time,
interactive best effort,
throughput intensive.

Different algorithms and insertion methods work better for each type so it maintains different queues for each class type.

After assigning to queue it assigns weights. there is a 4th queue which is class unaware and it chooses which tasks to take based on weights and queue.

There are two ways to assign weights:

         proportionate time allocation.
                doesnt take into account amount of data being transfered.
         proportionate byte allocation.
                takes into account the amount of data going to transfer.

there are advantages to each method.

Most of the code is done. Bill Irwin saw the code last night. No one else has yet. Hanna encouraged him to send it out anyway.

Matt said he has a paper he will put on line. Jens Axboe read the paper a while ago and wanted to see it actually work.

Cliff offered to help Matt run it under the STP so they can help test it. Sometimes STP isnt happy so make sure to cc cliff white to get help getting it working.

Gerrit reiterated the importance of sending out the code.

Originally got the project from mjc (Michael Cohen) and found lots of performance increases based on his implementation in the user space. He asked Matt to work on it in the kernel.

Matt is going to send out a patch today to lse-tech.

II. Cliff White - Scheduling benchmarks

Has been redoing the AIM stuff. Should have an early alpha release next week. Should work with both aim7 and aim9. He is switching some of the static stuff to dynamic to make it easier to work with. He hopes to have something runnable next week.

What it does is take a list of microbenchmarks, decides on a num of tasks for each user, forks a certain number of children. each child runs a number of tasks in a random order.

Currently looking at a problem where if 20-30 children forking seeing 3-4 seconds delta between children starting and stopping even though they are all doing the same thing. The lifetime of the parent is about 50 seconds. There is a lot of contention between the children so it may or may not be a problem. At the very least this looks like a good way to stress the scheduler.

(Minus some stuff which is NDA'd because it involves unreleased chips etc)

• Promise 20376
• Audit Promise drivers
• BIOS timing stuff
• Simplex mode reassignment intelligence
• IDE-SCSI crashes on 2.5
• IDE-SCSI/reset race on 2.4/2.5
• Forward port remaining drivers to 2.5
• Add ATAPI virtual DMA
• Add DMA active irq poll trick
• Clock switching for Highpoint 372N
• Support for SATA bridge on HPT
• Intel ICH5 errata audit
• Intel Centrino idents and errata audit [Merged for 2.5]
• Explain rather than just fix the CMD680 mmio collision problem
• Finish hotplug handling
• Revert identify hacks now ide-default is present
• Allow multiple driver binding for ide-cd/ide-scsi etc
• Locking for unload driver
• Locking for modular load onto a busy interface
• ADMA full support
• Mark Lord/Andre ideas on LBA28/LBA48
• Finish verifying 256 sector I/O or larger on LBA48 [How to handle change dynamically on hotplug ?]
• Clean up ide_unregister paths
• Finish ide pcmcia code hot unplug interface registers
• Finish ide pcmcia unregister path retry logic
• IRQ detect broken for some setups
• Check full PCI clocking info on HPT37x
• Rewrite HPT37x controller type logic
• Debug TRM290
• Fix up the hwif based sectors per transfer limit
• How to handle generic class IDE devices by class only
• Opti support for the 558 ?
• Can we resolve NDA's with SiS ?
• Audit ALi driver use of config register bits on bridge
• Document the calling properties for each driver function
• Work out how to fix up all the TCQ crashes
• Does taskfile I/O now work after the bug fixes ? [do we care 8))]
• Multiple taskfile load support for controllers that have it [big performance win]
• IDE specification issue - mishandling error abort on ATA6
• 20276 with i960 SX6000 mishandling
• Investigate breakage in ide-floppy on 2.4
• Merge new ACPI + relax into the -ac tree
• Get Arjan's info on IDE violations in simulator

That is basically impossible. How are you going to handle the case where you have a queue full of 256 request writes, and the plugged in disk chokes on them? And insolvable unless you start setting aside requests simply for this purpose. Also breaks the pseudo atomic segments that a single request represents. This is just way beyond ugly...

This is a generic problem of course, and the typical answer is to go by the rules of the lowest common denominator if hot plug can cause you queue limits to be violated (may be other problems than simply max sector count).

LUFS 0.9.5 has been released.

LUFS (http://lufs.sourceforge.net) is a hybrid userspace file system framework. It comes with a bunch of useful file system modules: sshfs, ftpfs, gnutellafs, gvfs, locasefs, gvfs, cardfs, cefs, etc.

2.4 & 2.5 kernel patches available at http://sourceforge.net/project/showfiles.php?group_id=57332

I have set up a tree on kernel.org for SCM repositories. Currently it contains a mirror of the bk->cvs repository tree at:

ftp://ftp.kernel.org/pub/scm/linux/kernel/bkcvs/

The main intent for this is to make it possible to rsync the full repository; we currently don't have CVS pserver access or anything like that (maybe later.)

This tree is mirrored hourly from kernel.bkbits.net.

I'm hoping someone with a BK license and a kernel.org account could add the bk exports file to this tree, as well.

Release 0.9.0 of the Open POSIX Test Suite is now available at http://posixtest.sourceforge.net. This third release contains POSIX conformance tests for the POSIX functions:

Threads (90% - not including tagged or rwlock-related functions)
Signals (90% complete)
Message queues (100% complete)
Semaphores (100% complete)
Timers (100% complete - tags TMR and CS)

It also contains bug fixes from 0.2.0.

The release notes that appear on download describe how to compile and run these tests.

The README page and the Open POSIX Test Suite website (above) give more information on the project goals and progress as well as information on how to contribute or contact us if you are interested.

Many thanks to Jim Houston, Jerome Marchand and other members of the POSIX testing community for their bug fixes, patches, and suggestions on how to improve the 0.2.0 suite.

The Open POSIX Test Suite is an open source test suite with the goal of creating conformance test suites, as well as potentially functional and stress test suites, to the functions described in the IEEE Std 1003.1-2001 System Interfaces specification. Initial work is focusing on timers, threads, semaphores, signals, and message queues.

Feel free to contact [email protected] if you would like further information.

I'd like to hold off in submitting the i2c chip drivers just yet, due to the changes for sysfs that are going to be needed for these drivers.

As an example of the changes necessary, here's a patch against the i2c cvs version of the eeprom.c driver that converts it over to use sysfs, instead of the /proc and sysctl interface. It's still a bit rough, but you should get the idea of where I'm wanting to go with this. As you can see, it takes about 100 lines of code off of this driver, which is nice.

I'm copying the sensors list too, as they wanted to see how this was going to be done.

Ok, in finding out a bit more of what the EEPROM driver is trying to do, it looks like it qualifies for the "binary file in sysfs" exception. It's just exporting a binary blob of data that exists in hardware through the kernel to userspace, for userspace to do with it what it wants to.

I'll work on converting the driver to that interface later tonight, unless someone beats me to it :)

Release 1.1.2 of JFS was made available today.

Drop 65 on March 25, 2003 includes fixes to the file system and utilities.

Utilities changes

• fix undefined reference to errno (G. D. Haraldsson)
• allow jfs_mkfs to run on regular file
• fix for-loop going past last element of vopen array
• sanity checking on variable this_ag
• s_label displayed incorrectly when 16 chars long

File System changes

• Clean up code flushing outstanding transactions to the journal
• Replace ugly debug macros with simpler ones
• Add get_index_page to eliminate unneeded I/O
• Fix hang while flushing outstanding transactions under heavy load
• Avoid deadlock under very heavy load
• Don't zero s_op during failed mount cleanup

Note: The 2.4.21 and 2.5 kernel.org development kernels are kept up to date with the latest JFS code. The file system updates available on the web site are only needed for maintaining earlier 2.4 kernels.

For more details about JFS, please see our website: http://oss.software.ibm.com/jfs

Hey, I'm a retard. I don't actually do the tagging with a script, since I usually want to create the tree privately first, and run a final compile cycle on it. So I tag it by hand, and then I have a script that outputs a list of commands that I just cut-and-paste directly. It's ugly, but it works (and it means that running the script doesn't _do_ anything: I have to actually take one last look at what I'm going to do before I start it all up).

The "release script" is this piece of crap:

        #!/bin/sh
        echo "bk export -w ../linux-$2"
        echo "bk export -h -tpatch -rv$1.. > ../patch-$2"
        echo "export PATH=\$PATH:/home/torvalds/BK/tools"
        echo "changelog v$1 v$2 > ../ChangeLog-$2"
        echo "shortlog --width=72 < ../ChangeLog-$2 > ../ChangeLog"
        echo "cd .."
        echo "tar cf linux-$2.tar linux-$2"
        echo "gzip -9 patch-$2"
        echo "gzip -9 linux-$2.tar"
        echo "touch LATEST-IS-$2"

and that's it. I'd just do "../release-script 2.5.66 2.5.67" from my kernel directory when I want to generate a 2.5.67 release.

Ok, I'm embarrassed to even admit to doing this. Don't rub it in. It's useful to me because sometimes I don't do a full release - I just want to pre-generate a change-log of the diff, for example, to judge whether I forgot about something.

This is announcement for the release of nfs-utils 1.0.3

It is available from sourceforge:
http://nfs.sourceforge.net
or kernel.org
http://www.{countrycode}.kernel.org/pub/linux/utils/nfs/

This release is primarily for people using or planning to use nfsd on 2.5 series kernels.

Other user's may upgrade if they like. There are a few small fixes which might be of interest. See the Change Log.

The system call interface for exporting filesystems via NFS is defined using types like "__kernel_dev_t" which should only be used internally to the kernel. Using these in a user-space interface was a poor decission.

There are patches available for 2.5 which change __kernel_dev_t, at least for i386, to be 32 bit instead of 16 bit. This is an incompatable change and nfs-utils does not work correctly on kernels with these patches.

This version of nfs-utils uses an alternate interface which is available in 2.5 to export file systems. This interface is not sensitive to changes in kernel internal type definitions, and so will work independantly of these patches.

If the new interface is not available (as it is not in 2.4), this version of nfs-utils will fall back to the old style interface. Thus this version should work on all platforms and all kernel releases.

It may be that when these patches are finally included into the mainline, we will able to keep the old systemcall interface working. However as this is not certain, using 1.0.3 (or later) is probably the best choice.