I am convinced that all these games are unnecessary, and that the problem is fundamentally different. Not fixing up the current code, but just looking at the problem differently - making the deadlock go away by virtue of avoiding the critical regions.

I think the suggestion to change the semantics of "swapout" is a good one. Now we have the mm layer passing down the vma to the IO layer, and hat makes everything more complex. I would certainly agree with just changing that semantic detail, and changing swapout to something like

        .. hold a spinlock - we can probably just reuse the
           page_table_lock for this to avoid multiple levels of locking
           here..

        file = fget(vma->vm_file);
        offset = file->f_offset + (address - vma->vm_start);
        flush_tlb_page(vma, address);
        spin_unlock(&vma->vm_mm->page_table_lock);

        error = file->f_ops-gt;swapout(file, offset, page);
        fput(file);

        ...

and then the other requirement would be that whenever the vma chain is physically modified, you also have to hold the page_table_lock.

And finally, we rename the "page_table_lock" to the "page_stealer_lock", and we're all done.

Does anybody see anything fundamentally wrong here? It looks like it should fix the problem without introducing any new locks, and without holding any locks across the actual physical swapout activity.

Well, considering that shm_swapout() currently looks like this:

        static int shm_swapout(struct vm_area_struct * vma, struct page * page)
        {
                return 0;
        }

I don't think the SHM case is all that problematic: we could easily just have a dummy vma->vm_file there. In fact, it probably should do so anyway: the SHM code _really_ does not need the private member.

There are strong arguments for saying that if the thing you're mapping actually _needs_ the vma in order to swap out, then the thing is broken. SHM certainly used to be horribly broken in this area, but that's no longer true.

There can be no contention on the page_table_lock in the absense of the page stealer. The reason is simple: every single thing that gets the page table lock has already gotten the mm lock beforehand, and as such contention is never an issue.

Contention _only_ occurs for the specific case when somebody is scanning the page tables in order to free up pages. And at that point it's not contention any more, at that point it is the thing that protects us from bad things happening.

As such, the hold time of the spinlock is entirely immaterial, and coalescing the page table lock and the vmlist lock does not increase contention, it only decreases the number of locks you have to get. At least as far as I can see.

Don't think of it as overloading the page table lock. Notice how the page table lock really isn't a page table lock - it really is just "protection against vmscan", and it's misnamed mainly because the only part we protected was the page tables (which isn't enough).

So think of it as a fix to the current protection, and as that fix makes it protect more than just the page tables (makes it protect everything that is required), the name should also change.

All the (current) new bootcode was written with the idea of "binary level compatibility" firmly in mind. I had actually worked backward from the original as86 output to come up with the equivalent gas instructions when the assembly was questionable, simply because I knew that that encoding worked. I did that with the idea that 'hey, if it compiles to the same opcodes, were fine' -- that was the sole plan at the moment: to have a 2.4 kernel that doesn't need as86/ld86 to build.

Yes that makes the source wrong on a pure "syntax" level in quite a few spots, but does it truly matter if (a) it assembles to the correct opcodes, (b) gives some much needed testing and real life usage of gas, so that it finally will do the right things when it comes to 16 bit asms, and (c) is planned on being (for lack of a better word) "optimized" back to correctness later on down the road (when everyone's using, or better yet are forced to use, the latest binutils).

You bring up some valid points, but I feel the intent of the patch was misjudged.

Just a few points on the latest binutils (which now might be a requirement of the next 2.3): The changes to the bootcode were intended to be included with a kernel that had a minimum binutils of 2.9.1.0.7, because I didn't feel it was worth a binutils upgrade to 2.9.5 to each and every person in order to have "syntactic perfection" (due to the fact that only the *latest* binutils produce the most correct code for most every case). I just wanted joe user to be able to still compile, but without the dependency on as86.

I've heard we might go to binutils 2.9.5 now, and if it happens I will happily rework the code to be more clearly, properly written.

Hope that explains my thoughts a bit.

Note that I'm probably going to remove the 1G split.

The 3GB user-space is better for users, and with BIGMEM support the advantages of 2G kernels are much less anyway. I don't see any horribly compelling reasons for showing that particular difference to users any more: the 2G split was a hack to avoid doing bigmem, but now...

This is especially true now that I have fairly clean patches from Ingo to take the BIGMEM stuff up to 64GB - the difference of whether you can directly map 1GB or 2GB of that physical memory is negligible, as 95% of the memory will be high anyway on the big boxes.

Some points I will make with regards to this point though, which you may wish to consider when resending it:

1. I suffer from Red-Green colour blindness myself, with the result that I am not the ideal person to determine colour schemes for web pages. It is largely for this reason that I keep my pages reasonably simple, and don't normally use lavish colour schemes.
2. Because of the above, I designed the pages to be readable easily by myself, and by people suffering from the same form of colour blindness. In general, I find pages on a Cyan background the easiest to read, which is why I have used that colour scheme on my pages.

I am of course willing to listen to any criticism on this front, and to try out any suggestions that come my way, but if I can't read the resulting pages myself, they will not go online. I consider this to be reasonable.

the colour scheme I'm looking for is one where the text can remain either Black or NavyBlue, but the background colour can vary to indicate different things. Basically, I need four different background colours that satisfy ALL of the following requirements:

1. Provides a good contrast to the text colour under all of the various colour blindness options.
2. Does so equally well under all graphical web browsers, and especially under both Netscape and Internet Explod^Hrer.
3. Does so equally well under all combinations of operating system for each of those browsers.

Any suggestions relating to this will be very much appreciated.

remembering back to when the BBC Micro was released in the UK. and the explanation of why it flew in the face of conventional wisdom by providing white writing on a black background. Vt100's, DOS windows and linux terminals do likewise whilst X11/MS windows have the reverse. I expect people want what they're used to, and the easiest way to make a "window" acceptible as the replacement of a piece of paper is to make it look like one.

I find light writing on a dark background to be easier.

Its cheaper to make books with black writing because white ink is harder to make (It has to be opaque, whereas black ink only has to stain).

Don't make a monster. Make something that is simple, and expand on it as needed. I do not like the way you usually sit on the code until you are happy, and then release it in one big bunch without any input from others.

I prefer the incremental approach.

I don't "sit on the code". The process is far from opaque. The updates are describe on web pages, mailing lists and hypermail archives. Yes, many private test versions are sent to individuals and then have limited alpha releases. I do seek to minimize the number of "real" releases, so I don't have to track a zillion released driver versions.

But I wonder why I put in the effort -- all of my work is thrown away when you accept random, tested-only-on-one-machine patches from anywhere, and then refuse to put in the tested, updated drivers.

Donald, your latest "tested" drivers have been a QA disaster, and you should know it.

The reason your updated drivers don't get included in the latest kernels are _very_ simple:

• you don't send me patches in a timely manner. There are drivers that you haven't sent patches to me for several years, yet you maintain what you call "stable" versions on your web page. Shades of ISDN, sometimes.
• very occasionally your new "stable" drivers won't even work on the two or three machines I test on. This has sometimes meant that the eepro100 driver in particular has lagged behind a _lot_, for the simple reason that newer drivers did not work at all for me.

The "not send timely patches" thing applies to the PCI code in question. I have never EVER seen a patch from you to implement the PCI search function. Not now, not a year ago, not EVER.

And then you complain when I apply patches that ARE sent to me? Get a grip, Donald.

From my side, I send an updated driver, you report "it didn't work for someone" (and that person never sent a problem report to the driver mailing list), and it doesn't go in.

I'm trying to save you work, and have a broader early-test base, by having a separate test process that doesn't require people to update to the latest, probably-broken development kernel. You should *expect* that a driver has a three or six month old date on it.

I basically never EVER search for patches on the net. If they don't come to me in email, I take that to mean that the author isn't interested in getting them into the standard kernel. The end.

And if the author cannot be bothered to send me timely updates, the author also looks damn silly if he then complains that I don't use his code.

Note the "timely". It's not enough to send me the patch once in a blue moon (Donald doesn't do even that - even though I've asked him face to face, and he always says he will). Again, if the author cannot be bothered to MAINTAIN his patches, it's not worth my time even trying to do it for him: not because I'm a lazy bastard (which I am), but simply because especially with drivers I need help with maintenance, and an old unmaintained driver with known problems is better than a new unmaintained driver with unknown problems.

For example, I do not generally apply large patches. Ask Alan Cox, for example, who does a hell of a good job of maintaining a large portion of the kernel (hats off to Alan), and then ALSO does a hell of a good job in splitting the patches up in their original components.

When I get something like the current "ac" patches from Alan, I don't get just one email with the patch - I _literally_ get about 50 patches that are independent - one per driver, one per subsystem, one per new feature like PCI detection. And then I apply about 48 of them, and explain to him about the two I didn't like and why I didn't apply them.

Most people don't see that kind of code maintenance - they see the full "ac" patch-set, and then some time later they see the patches in my kernel. And sometimes people wonder why the end result isn't the same as the "ac" tree - now you know why. It's either because I decided to not apply a part of it, or (quite often) it's because Alan didn't even send part of it to me, because he knew it had some other issues.

In August you rejected all of the then-current updates because they had visible support for multiple kernel-support. You wanted all backwards compatibility code removed. Based on that I spent several weeks changing and testing over a dozen PCI drivers so that the base versions had support for 2.3 only. Now those are changes are rejected.

I spend more than half of my working time working on Linux drivers, including the time-consuming, boring work of answering email and tracking down usually-spurious bug reports. I've done it in a way that minimizes your workload, and makes the updates and new drivers immediately available to those with stable kernels. And now I find out that you don't consider me to be involved with the Linux kernel work.

Donald, read my emails.

They have NOT been rejected.

They haven't even been CONSIDERED. There's a big difference there.

And the reason they haven't even been considered is that you have not sent in any patches to me. I have not seen the patches, I have only seen the results of Alan applying the patches to the ac tree and various bug-reports.

And I will repeat my rule: I do not apply large patches with many separate changes. I apply patches that do one thing (like implement a PCI search function).

As Linus has described, and what you seem to have problems understanding, is that _this_ very technique _maximizes_ Linus's workload.

This seems to be the source of your confusion. Batching up sets of fixes over a long (ie. not timely) period of time, and then sending them all at once for integration is about the worst thing you can do.

Let me give you two situations to show you why this is true:

situation 1) You fix a bug today, you send Linus just that small change to one specific driver, to fix that bug.

This change is fine and nobody complains of breakage. You run through this process about 3 or 4 times.

On the 5th change, again small and specific and incremental, Linus and others note that people begin to complain of problems. These people also proclaim that before this 5th change went in things were perfectly fine for them.

situation 2) You spend 4 months, and fix dozens of bugs, and also have reworked major sections of code in a driver to support new cards, or do whatever. You send one big fat patch to Linus after this 4 month period, many users complain that the driver suddenly stops working for them in a big way.

In situation #1 Linus needs only to back out patch #5 or seek a remedy from you for that specific change, the fixes in patches 1 through 4 are not backed out and not lost. Whereas in situation #2 he has to back the whole thing out, and all the work is gone until things are remedied, and you'll probably repeat the cycle as other bugs are found whilst you keep resubmitting this huge patch, and eventually Linus (like right now) will get disgusted with the whole mess.

Now can you see why your "external from the main tree for months" development process sucks and causes grief for everyone?

That's almost never the case.

For drivers that support many cards I usually have ten different modifications outstanding, each one to attempt to fix a specific reported problem. Most changes either don't fix the problem, or are known to cause other problems. It's often a week or two before a test versions is reported on, so many times there is no way to serialize the changes.

Again, THIS is the problem. They shouldn't be outstanding. They should be out there, in the standard kernel. If they work, they work. If they don't, we find out. And if they don't work, then others can at least help. Right now, if they don't work, people usually don't even know, because the people for whom the old driver is fine will never even bother to try your internal test-of-the-day-driver.

Resulting in that when you _do_ think it's fine, it isn't. And at that point it's too dang hard to tell exactly what it was that broke.

The problem may be that you want to maintain a heck of a lot of drivers, and maybe you should try to delegate a bit, so that you can give the drivers you _really_ want to work on the priority they deserve.

Think how much faster, and more voluminous, reporting you would receive if your changes went into the main trees on a timely basis?

You can have the largest test lab, you can have thousands or millions of cards to verify things on, but it will never be any match for the cast of wacky configurations that users have who actively test and beat on the official release and pre-patch kernels done by Alan and Linus have.

Think of me as CVS with a brain and with some taste. Nothing more, nothing less. Development is done all over the world, and something has to keep it reasonably coherent, so that when people want to find a kernel that compiles most of the time and does what they expect, they can do so.

Most of the stuff is completely independent of each other, and you can live in your own microkernel universe 99% of the time. But having a central repository means that when global changes are needed, they CAN be done. It's painful, and it breaks code, but at least it is technically possible - which it wouldn't be with everything scattered all over.

My opinion is that a driver bug should not be backed out, given maintainer being known to be a serious guy, unless the maintainer agrees with, or ask for that. Anyway, a bug must be fixed and delaying it a long time is a bad option, in my opinion.

Note, that if it ever appears that the maintainer did the wrong thing, I trust him to understand how he has been wrong and change the way it will deliver changes for further updates. A maintainer that did waste time of users because he delivered broken stuff when this could be avoided is normally extremally sorry of that, and I am sure Donald has been so if this happened with some of his driver versions.

Just do not backout Donald's patches that seems broken and I bet you that everything will be just fine, or at least not worse that other breakages that sometimes occur during kernel development.

What?

It's not about "seems broken"

It's about the issue that there are real and definite bugs, and if a lot of things changed there is no good way to find out exactly what change caused the problem - especially not with the problem popping up for people who do not necessarily know C (or the device) enough to make a informed judgement other than "version X works for me, version Y does not".

The whole point of open source is to expose the development, and NOT have the mentality that "it will be fixed in the next release". There should be many small incremental releases, because whatever Donald or others say, especially with drivers you are often in the situation that you cannot from looking at the source see whether something is broken or not.

So it needs to be released often, and TESTED often. Which implies that the test-drivers should be part of the standard development kernel, because if they aren't, they aren't going to get very wide testing.

For example, what has happened multiple times is that the 1% for whom some particular old network driver does not work will try out Donalds new drivers, and what do you know? It works for them! And people think that that means that the new driver has to be much better than the old one, right?

Wrong. The new driver is NOT necessarily better at all. Not only has it been tested by much fewer people, it has been tested by a SELF-SELECTED group of people. Which may mean that the new driver fails horribly for a lot of people where the old driver was fine - because the new driver effectively has ZERO testing for common hardware that worked fine with the old driver.

This is not worth discussing further. Timely incremental changes are just so OBVIOUSLY better to anybody who has done any real maintenance that the argument is pointless. It's true in non-Linux settings too - why do you think commercial software companies have regression tests and a large testbed of different machines that are always active?

If Donald doesn't do the nice incremental patches, then somebody else will end up doing them. But that also means that Donald loses the right to then complain about others doing the work that he somehow considers "his".

I have occasionally forwarded stuff to Donald. More often, I forward things to Alan and Davem, simply because they tend to know networking issues well, and I tend to get more response out of them ;)

And no, I _don't_ feel responsible for specific device drivers, and sure, I could take the approach that "what do I care if a driver is broken?". And in fact, without a maintainer, that is what I often end up doing, although it it turns out to be a problem (it isn't always) I try to prod people who are silly enough to send me patches to maybe become a maintainer..

But what would happen if all the drivers were somewhere out in web-land, and to find the drivers for your random combination of hardware you'd have to search five different web-sites from five different maintainers? Do you really think a system can survive that way and find more users? And ever be tested as a "sum of all parts"? I don't.

So in real life, you need to have a "standard base" that includes all the stuff most people are reasonably expected to use. It doesn't include some of the more esoteric stuff (hard realtime, specific drivers for hardware that exists only in rather special places etc), but it certainly has to have drivers for things like a random tulip card, would you not say?

And maintaining that "standard base" is what I do. Others do it too, notably SuSE and Redhat. Much of that "standard base" is based on their work, in addition to the obvious work by the actual people who create the actual drivers and new features.

But exactly because Linux is _NOT_ a "one entity does everything" proposition, it is NOT the case that I go out on the net and find everything I want to have in the full package. I very much depend on people like David Miller, Alan Cox, Ingo Molnar, and a hundred other people who not only maintain their own subsystems, but also help me in maintaining those subsystems as part of the larger whole.

The problem in question is when a maintainer maintains the subsystem, but does not try to help in maintaining the whole. See? Instant disconnect.

Note that we've had that happen before for other drivers, and the sane way to handle it tends to be to split the driver up.

For example, see the IDE driver. It has core IDE functionality, but then multiple "sub-drivers" to do tuning etc. That may not be required for the tulip driver, but if the driver is starting to look fragile, then by all means at least give it some serious thought as an option. Often source duplication is a much lesser evil, and the evil of trying to handle everything with one approach can be a complete disaster.

Remember: the UNIX philosophy is to NOT try to create the "GNU Emacs of drivers". Do one thing, and do it well, even if it means that you need to have another driver for a card that isn't really all that similar any more.

See how the ncr53c7,8xx SCSI driver is handled: there are actually two drivers, where the newer driver only handles the newer cards, and as a result BOTH drivers are simpler (and the newer driver can be noticeably faster because it doesn't have to worry about issues that only happen with the old and arguably broken cards).

Splitting a driver tends to mean that the support for old cards stagnates. But that's fine - old cards do not change over time, and it makes it easier to change both the new and the old driver without having to worry about breaking the "other" one, because they are now independent.

For example, to just point out that this is not a problem just for network drivers: the same thing is definitely apparently happening with the PCMCIA host controllers. They are all just "similar enough" that one driver tries to handle them all, but as a result that one driver is quite fragile, and when somebody tries to fix an old ISA card that can break cardbus support and vice versa.

It's happened to many drivers over time, and if somebody really is willing to try to clean it up, I just want to say that at least as far as _I_ am concerned, I don't think it is a bad idea to have several drivers for similar hardware. You can try to share as much code as you reasonably can (see for example the 8390 driver that does this fairly successfully), without thinking that there has to be ONE driver that handles ALL 8390 cards.

Alan is a bit terse every now and then. In this case, that's because he's been explaining this over and over again.

It is kind of tricky, let me try to explain it.

You have two drivers that share an interrupt. Normally the two drivers both get called when an interrupt happens, they both check if there is something to do for their device, and they both clear the interrupt if their device was the one interrupting.

In this case, we have one real (e.g. disk) device, and one "user-device" on an interrupt. What can happen (but not the first time you try it!: The first time you try it it will work...) is that the user-device interrupts, disables the interrupt, and tries to wake the program. However, this program causes a pagefault which requires the disk. So the disk driver may then reenable interrupts. If it does, this will cause an immidate call of both interrupt routines, but the disk is not yet ready. The user-level driver will then redisable interrupts, to let the user-level program figure it out. Deadlock.

My point is that when developing a driver for a device without documentation, where you don't know how to acknowledge the interrupt (yet), this strategy works:

• Select an UNSHARED interrupt line.
• Call disable_irq() when the interrupt is received, and enable_irq() when you're ready to receive another.

Of course this is slow, prone to failure, won't work everywhere and generally bad. No driver in the kernel rhould even do this.

But for a hack to test something, it does work.

now more and more (IMHO off-topic) posts about what hardware to use are starting to appear on linux-kernel and linux-smp, I asked if there would be interest in a new list for the purpose of discussing Linux and hardware.

Since quite a bit of interest has been shown, I created the list as soon as I came back from ALS -- even before I have recovered from jetlag and resulting lack of sleep :)

It seems to find bad memory so easily with such a simple program.
*shrug* :)

I've put a slightly updated version up here:

I added a comment, fixed a spelling error and cosmetics in the fprintfs, but didn't change anything else (so hopefully it won't lose its magical powers). :)