Kernel Traffic Zack Brown
Code Freeze FS: JFS FS: NFS FS: ReiserFS FS: ext2 FS: ext3 Microkernels Networking Random Number Generation SMP Version Control Andreas Dilger Hang on to your hats folks, the first step is a doozy. The last time ReiserFS was covered on KT was in , and it seemed that the seas had calmed and the clouds had parted. However... In the course of discussion, Hans Reiser accused, Alan knew perfectly well that Linus tentatively planned to put reiserfs into 2.4.0, and yet Alan came out with a schedule in which reiserfs was deferred until 2.5. Thus the squabble. Alan has come up with an endless series of FUD reasons why ReiserFS should not go in, none of them valid. If Alan wants to put in some better architecture for journaling, fine, but reiserfs going in now does not prevent doing that. Alan Cox replied in turn: You are actually beginning to piss me off, which is quite an achievement. Your pathetic little attempts to bully people into merging your file system by trying to make them look biased if they dont follow your little whim are not going to work. If you get a bonus from your funders for being in 2.4 tough. Thats not how Linux works. I've read Machivelli's The Prince. I've read The Art Of War. I've fought Telco management stupidty in the trenches. I worked in the computer games industry. You simply aren't in the right league to play such games so please don't bother. Now if you must whine, whine in private. You are becoming the biggest hindrance to the acceptance of the file system your engineers are writing for you. If you want to get it in 2.4 then fix the bugs like the Postgres problem, keep the code clean and sort out things like the interactions with NFS handles, and your strange extra errno codes. Thats far more productive than being an idiot on the kernel list. Hans replied that he'd work on the technical problems, and added, I'll ignore the other stuff you said, and thank you for contributing so much to our community over so many years, for you have indeed done so. Colin McCormack replied to Alan as well: Reiser questions Cox's personal integrity. Cox then suggests that Reiser's comments could hinder his code's inclusion in 2.4. For Cox to even suggest this as a pretext for exclusion merely confirms Reiser's perceptions. Decisions may be made on personal (not engineering) grounds. I would have hoped that Linux bundling decisions were exclusively engineering-based, ``For however strong a new prince may be in troops, yet will he always have need of the good will of the inhabitants, if he wishes to enter into firm possession of the country.'' [Machiavelli, the Prince] I should add, on a personal note, that I only speak freely because I have no intention of ever adding a line to Linux kernel, unless it's forked. There were several replies to this and one long subthread. Christopher Fisk didn't see Alan and Hans' exchange in the light Colin did, and felt instead that Reiser was spending so much time stirring up things on linux-kernel that his attention probably wasn't on fixing the things that made Alan decide not to include the code. There was no reply to this, but Alan Cox also replied to Colin's take, specifically the statement that he (Alan) had suggested he would bar code from the kernel because of Hans' comments. Alan said to this, No I dont. I suggest he's pissing off people who would otherwise help use and debug his fs. And I have third party mails and comments that prove that assertion. There was no reply to this, but James Sutherland started the long subthread in reply to Colin. He started off replying to Colin's characterization of Alan's motives, saying, No. He indicated that Reiser was not making any progress or friends by making personal (unfounded) allegations. There are right and wrong ways to attempt to establish support for your plan: slagging off one of the most respected and productive members of the community on a false premise is probably not the right way... Replying to Machiavelli's quote, James added, The quote appears to support my side of this, not yours... Reiser is trying to force his code into the kernel prematurely, and rather than building support, he is just attacking those who oppose him - using his strength of troops to override the wishes of the inhabitants. James added, some of the things Reiser says may count against the premature inclusion of RFS in the kernel, and concluded that including RFS without adequate integration testing and support/maintenance would be substantially worse than not including it. Reiser's antagonism strongly suggests that there will be limited cooperation over future development/maintenance; if this is the case, the code must be excluded. Finally, James asked why Colin wouldn't contribute to the kernel unless it was forked, and Colin replied, Well, it's developed by someone who half-formed an opinion on microkernels years ago and has not revised it since; who thinks a module is something you buy at Ikea; who can't/won't use a version control system and who bitches at people who do (I'm thinking of the ISDN people, here); who refuses to accept patches in .bz2 format (god knows why), who seems to adopt rather arbitrary stances on technical matters, [...big breath...] and it shows. To James' assertion that some of Hans' comments might count against including ReiserFS prematurely into the kernel, Colin also said, Reiser has some opinions regarding the political and economic nature of Linux development. The views may have merit, or none, but that's completely unrelated to the nature of the code he's producing/produced. To this last, Theodore Y. Ts'o entered the discussion, with, While I agree with this in principle in general (although I wonder how much of the code is actually Han's; it seems that he's managed to con a bunch of programmers to do his work for him), I remain concerned that we not set a precedent that the way to get something into the kernel is to act like a paranoid asshole. That to me would be rewarding the wrong sets of behaviour. Two subthreads branched off from there. Richard Gooch agreed with Theodore that it would be bad to encourage that kind of behavior, but at the same time, he chastized Theodore for accusing Hans of "conning" people into coding ReiserFS. Richard added, He shouldn't be rewarded for bad behaviour, but he shouldn't be singled out for exclusion either. He has not sinned alone. Chip Salzenberg put in, My experience in the Perl community has taught me that tolerating or even celebrating buttheads who happen to be technically prolific is, in the end, very costly. It creates a skin-thickness requirement. Some who might have contributed leave rather than putting up with verbal abuse. A once-friendly community is be sacrificed on the altar of technical advancement... ironically, to the eventual detriment of both. In a later post, Chip added: Hans happened to trigger the discussion, and I happen to dislike his conspiracy theories, so I could see how it would look like I'm picking on him specifically. But I'm not. I'm just taking this opportunity to discuss the theory and practice of open source cooperation. On the other hand: It's not necessary to wait for the one and only worst social interaction on L-K before the issue can be discussed. On the gripping hand: I don't think Hans' particular brand of irritation should keep reiserfs out of the kernel. If Hans were, hypothetically, to become impossible to work with someday, we would have no problem with maintenance, because the reiserfs programming team would still be available ... and they've shown themselves to be friendly and cooperative, as far as I can recall. Elsewhere, Hans replied to Theodore's statement that Hans had "conned" programmers into doing his coding for him. Hans replied: This is a remarkable statement, I mean when you dislike someone you get downright religious about it don't you? Just so that I fully understand you, do you view workers as victimized exploitees of evil capitalists? Is an architect who comes up with ideas and algorithms and stares at the wall and systematically schmoozes with every other member of the team about their aspect of the design, and then usually asks other team members to do the coding for him, is this a lazy parasite who doesn't do real work in your view of things? The guy who works 40 hours a week to earn the money to pay the other programmers, and then lives like a college student while he works another 15-20 hours a week sending emails arguing over and directing their algorithms for 6 years before the project makes a penny on its own, he's a victimizer in your book, yes? We are all a bunch of basically generous persons squabbling over how best to give free software to the world and getting a little silly doing it. I can become an instant millionaire anytime I choose to take one of my standing offers to abandon reiserfs for a proprietary software company, and I am sure the rest of you have had similar offers. All of us would be making a lot more money if we had chosen to do proprietary software. Try not to get so wrapped up in accusing the other person of paranoiac evilness that you lose your perspective on that, ok? Hans also added his "FUD List" of things that had been said against ReiserFS or him:
  1. ReiserFS has serious VFS problems [it had unnecessary checks in it, but it was said in such a way that we wasted man-weeks reviewing before we could say for sure that it was not something more serious, the person who said it is unable to come up with anything more than these (now removed) unnecessary checks to complain about.]
  2. The disk format will change [Disk format changes will be merged into the mainstream kernel once per major kernel release. The mainstream version will be called "reiserfs". Development work tied to disk format changes will be done on a filesystem called "reiser4" which will be merged into "reiserfs" at the end of the 18-24 month linux development cycle. New versions will be able to read old version disk formats using our object oriented format handling code (which is getting more and more sophisticated and powerful with each version we come out with.) New versions will not convert old formats unless told to do so by the user. We are backporting reiserfs 3.6 to linux 2.2 so that even if users do convert they will be supported. ]
  3. ReiserFS should wait for somebody else to do journaling for it, and only then be merged in.
  4. Oooh, that evil Reiser, he didn't like our suggestion that he wait for somebody else we pick to do his journaling for him, he's going to be difficult to work with and should be kept out of the kernel for that reason. (Actually, it will be easy to get us to spend a few days to modify our code to use interfaces with specifications needed by other filesystems if you ask, so long as you don't ask us to wait for code you haven't written before making use of ours.)
  5. Oooh, that evil Reiser, he suggested that we engage in social exclusion and factionalization rather than just judging software by its merits, we need to punish him for saying such nasty things by excluding his software. (The irony of this seems completely lost on them. Thanks to Colin for doing the complete devastation of their logic.)
  6. And now the new, Oooh, that evil Reiser, he cons programmers into working for him so that he can live a 70 hour work week life of leisure. (Yes, now that I don't have a day job I do work longer hours than ever before, what a fool I be....)
 Chris Mason replied to several points. To item 1, he said, It was time well spent, as the code ended up much nicer when Vladimir was done. We shouldn't complain here, we had bugs, and now they are fixed. To item 4, Chris said, Hans, please read all the threads again. It was a confusing mess of flames and misunderstandings, and we certainly didn't do much to keep the flame/content ratio down. And finally, to item 3, Chris replied: This is the real reason I replied, it needs to be cleared up. What I got from Alan and Stephen's comments was that a common journal layer would be a good thing. The real issue at hand is the memory pressure interface, and how the journals interact with the buffer cache. We have our own end_io handlers for the log blocks, and the only other copied code is a reiserfs version of unmap_buffer. Both Stephen and Alan have replied saying they don't intend to force reiserfs to use the ext3 journal layer. Part of the confusion is my fault for offering to give the ext3 JFS a try, I'm not going to abandon the reiserfs journal layer, but I will try to improve it. Hans replied with an example. He spoke of someone who had written an ext2 resizer, while the maintainer of the relevant sections of code (Stephen C. Tweedie) had not bothered to look at the patch for six months. He concluded, Here is some poor guy who has put in the rather substantial labor required to write an online resizer, and his code is just going to waste and there is nothing he can do about it. He has even spent time rewriting the code to minimize the kernel portion of it so that these other guys will have less to read. It wasn't enough. Stephen replied that he'd reviewed the code in April, and added, I'd prefer this patch to be an early 2.5 inclusion than a 2.4 feature, since we were already _well_ into codefreeze before I ever saw the proposed ext2 changes. Online ext2 resizing is important, but I'd rather see any core ext2 changes get a _lot_ of testing before a major release, and certainly we'd need good reason to restructure such critical code during codefreeze. And Theodore also replied to Hans, saying: .... and now, for the rest of the story. Andreas Dilger has asked us to integrate a patch which makes it easier for him to maintain his on-line resizing patch. It moves a lot of code around from ext2_read_super() to a new function. It is supposedly doesn't change any lines of codes, but it is a very large patch, and so it's painful to integrate. Given that it had no functional changes, I gave it a low priority. Given that other pieces of the kernel which I am responsible for (and which are already in the Linux mainline) which have SMP race conditions (/dev/random) or which give kernel OOPS when you open the device (the rocketport driver), the gentle reader will perhaps excuse me for not attending to a 200 line patch which added no functional changes, but which carried the risk of potentially destablizing a critical piece of the kernel. Remember, unlike Hans Reiser, I don't have a staff of a dozen people working for me. While this means that sometimes my bandwidth is a little limited, it has the advantage that I don't get placed into conflict-of-interest situations where the desire to earn enough support income to pay for my employees doesn't tempt me to push for integrating non-bugfix changes during a code freeze. To Andreas's credit, at no point did he come to me and ask me to integrate patches that actually integrated the on-line ext2 resizing code into the mainline kernel. And if he had, I would have told him that we were in code freeze (even six months ago, we were in code freeze), and that the ext2 filesystem is too critical to risk destablizing it once we entered 2.3.99. In fact, as far as I know, he was planning on maintaining the on-line resizing patch as a separate patch, and was not trying to integrate it into Linux 2.4. The first that I heard that there was some desire to sneak the full on-line resizing functionality into Linux 2.4 came from Hans Reiser. To Hans' statement that the author of the ext2 online resizer probably wouldn't approve of his post, Theodore said: The cynical person might perhapos think that this might be a Micheavillean attempt by Hans to try to manufacture this as an excuse to get reiserfs into the kernel. ("See, you let ext2 have changes, clearly the code freeze doesn't have any meaning so you should let reiserfs in!"). The more charitable explanation is that there is a fundamental philosophical disagreement over what "code freeze" means, and what levels of reliability and code maturity is required before letting filesystem code into production use. I repeat, as far as I know Andreas has never advocated that we try to make fundamental functional changes to the ext2 code at this late date. Only Hans seems to be strongly argueing for this. Why? I will leave it to others to speculate; but ext2 isn't even his filesystem. To this last, Alan replied, Nothing so sinister. Andreas originally asked about merging that bit to make things cleaner. He hasnt asked me about merging the whole thing. Ever. Hans said, I have the impression that he would like the whole thing to go in but has despaired of it. It seems quite sad to me. I suspect him of being an overly nice guy like Chris.:-) This thread should die though, better that I spend my time ensuring that it doesn't happen to patches sent in for reiserfs. And the thread ended.
Kernel Release Announcement Alan Cox Tigran Aivazian Linus Torvalds announced Linux 2.4.0-test2: There's a "test2" kernel out there now, integrating most of the -ac patches, and some code that wasn't in -ac. Normally, when you integrate almost 5MB of patches, bad things happen. This time, a miracle occurred. As I uploaded the resultant kernel, a specter of the holy penguin appeared before me, and said "It is Good. It is Bugfree". As if wanting to re-assure me that yes, it really =was= the holy penguin, it finally added "Do you have any Herring?" before fading out in a puff of holy penguin-smoke. Only a faint whiff of rancid fish remains as I type in these words.. In short, not only are most of Alan's patches integrated, I have it on higher authority that the result is perfect. So if it doesn't compile for you, you must be doing something wrong. Richard Gooch posted a patch and said, Still the same problem I told you about a couple of days ago: it doesn't compile with the recommended compiler (gcc 2.7.2.3). Maybe this time you'll apply my patch? ;-) Later, Garst R. Reese said, Long ago I came to the conclusion that if the kernel was evolving it would evolve in an evolving environment, so I kept my binutils and compiler reasonably updated, at least to the last official release. This strategy has been quite successful for me. I have also noted that Changes is usually ancient history, specifically with respect pcmcia, but I am sure that Chris does his best. But maybe it would help if there was some version information with new releases it would help. E.g,. cat /proc/version and ld -v Tigran Aivazian also rose to the challenge, posting a patch for a fix that had gone into Alan Cox's tree, but hadn't made it into Linus' test2. There followed a medium/long technical discussion of why Tigran's patch hadn't gone in, involving some deep exploration into the C standard and its relationship to practical development. There were a smattering of other bug reports, but no other discussion.
Disks: SCSI FS: ReiserFS Virtual Memory Christopher Zimmerman reported that a single 'bonnie' process with a 2G file had deadlocked linux-2.4.0-test2 on his 8way 3ware raid 0. Henrik Storner noticed a similar lockup on his Red Hat 6.2 with Reiserfs. He reported his hardware as including a PII/350 with 128 MB RAM, and a Symbios 53c875 SCSI controller with an IBM DDRS-34560 attached. Elsewhere, Christopher reported that test2-ac1 seemed to fix the problem. Later, he added, linux-2.4.0-test2-ac1 seems golden to me. I've spent the last 8 hours trying every and anyway to kill my test boxes Everything seems to be rock solid. Later, he reported his machine crashed with "VM: killed klogd" errors, but Jens Axboe replied, That's most likely an unrelated problem, looks like -ac1 is still not perfect mm wise.
Jens Taprogge reported oopsen with test2 when running the 'S20modclean' script from Debian Woody (unstable). He listed his hardware as being a Toshiba Satellite 4000CDS. Glenn C. Hofmann also reported lots of oopsen with Woody. Jens reported no better luck with 2.4.0test3-pre1. Elsewhere, Mitja Sarp also reported lockups on the Toshiba Tecra 8100. No real debugging took place; maybe folks assumed Woody would turn out to be the real problem.
I2O Sound: i810 Philipp Thomas Derek Martin Jens Axboe Arjan van de Ven Tim Waugh Bartlomiej Zolnierkiewicz Alan Cox released 2.4.0-test2-ac2 and remarked, Since people started running ac1 I've put out an ac2 with the various reverts people pointed out fixed and some important furthe fixes done. He listed from the Changelog:
  • Fix the elevator bug (Jens Axboe)
  • Fix gcc 2.7.2 building (Various)
  • Fix building with CONFIG_NET=n (Arjan van de Ven)
  • Fix memory copy cases with gcc 2.96 (Philipp Thomas)
  • Fix 8390 non modular not exporting syms | different fix but Sang-yong found the bug (Sang-yong Suh)
  • Fix compiled in I2O bug (Arjan van de Ven)
  • Fix UFS indirect bug (Marcel Waldvogel)
  • Fix ppdev init bug (Mikolaj J. Habryn)
  • pcibios_penalize_isa_irq cannot be __init (Chris Wedgewood)
  • Fix scripts/kernel-doc regexp (Tim Waugh)
  • Fix comx symbols needed | Probably the comx proc code is what needs sorting for real (Kjaratan Maraas)
  • Fix coda crash (Steven Smith)
  • FIx ambiguous CPU name problem on config (Niels Jensen)
  • Fix sisfb breakage (Bartlomiej Zolnierkiewicz)
  • FIx i810 audio memory scribbles (Vladimir V. Klenov)
 Derek Martin reported that ftp.us.kernel.org seemed to have lost the entire 2.4 tree and all of Alan's patches. H. Peter Anvin replied: When this happens please send email to ftpadmin@kernel.org and include the *IP number* of the server that gives you trouble. There are currently 26 different ftp.us.kernel.org servers, and we need to know which one. If you don't know how to get the IP number of the server you're using, you can also go to http://www.kernel.org/mirrors/, click on a country (e.g. United States) and access the list of servers directly.
Disks: IDE Disks: SCSI Stephen Torri Tim Waugh Andre Hedrick Andre Hedrick reported that a lot of people had volunteered to help him document the ATA/IDE subsystem, and it gave him the idea for a more organized, more general system of documenting the kernel in its entirety. He suggested that such an effort could be self-directed, aiming against particular FUD attacks as they appeared. But Alan Cox replied, Each to his own goals. I want to be able to type 'make book' and get a book that I can hand to anyone wanting to do Linux drivers 8) Garst R. Reese agreed, adding that he felt the feature to be absolutely essential for any kernel documentation project to survive. Rik van Riel added, I started kernel-doc@nl.linux.org for this a long itme ago, but didn't find the list very active after the initial rush ;( Later it seemed folks wanted to revive Rik's mailing list for the current project. Elsewhere, Douglas Gilbert mentioned, some good work has been done in the Documentation/DocBook directory which is what I believe Alan was alluding to. If people want to try the sgml/docbook route then I found Tim Waugh very knowledgeable. He helped me produce this document describing the SCSI subsystem in lk 2.4: http://www.torque.net/scsi/linux_scsi_24/. Gary Lawrence Murphy put in: The LDP HOWTO-HOWTO is also evolving rapidly into a Linux Authoring Guide which will explain the mysteries of DocBook. The document now contains a very good template for newcomers, surveys of authoring tools and the basic instructions for creating an authoring environment using OpenJade. Anyone with specific questions about DocBook documentation is also invited to write to me; since I'm being paid to document the kernel, I have a vested interest in getting you all up to speed ;) Tim Waugh replied, with a pointer to http://people.redhat.com/twaugh/docbook/selfdocbook. Elsewhere, Garst said, I would just like to restate that I think it extremely important that we choose an outline and format. My personal preference would be to use LyX, but this could be because I find the documentation of DocBook and sgml-lite so inscrutable :) Most of the details can be sorted out when we have a list, but we will have a basic dependancy on the code writers to supply the basics, and our success will depend on them percieving it as a valuable service, rather than just something else eating away their time. Stephen Torri agreed, but cautioned against getting into text-editor-wars. Elsewhere, Gary Lawrence Murphy said, What we need, and which will be the hardest to get, is not for the programmers to open their schedule books for us, but to get them to dilligently follow the GNOME commenting style --- if every function, or at least every public function of every C source file were to have a 'proper' comment block, kernel docs would be self-generating. That should be our goal. Anything else and we are faced with the same instant obsolescence that thwarted the previous kernel doc projects. There was no reply.
H. Peter Anvin Sarah Nordstrom reported, Just wanted to let anyone who was interested in the POWER port, that i've pretty much let it drop, due to a lack of documentation available. Everyone had told me that, but i wanted to see what i could do. The documentation is just plain not available for any opensource project at this time, and it doesn't look like IBM's planning on making it so, as it's a dead platform really. Ohwell, i've got an HP PA-RISC 735/99 i'm going to get Linux on now, and mabye make some contributions to that port ;o) Tom Gall replied: Are you on the linuxppc-workstation list at lists.linuxppc.org? I posted pointers to the documentation that you need. I posted this not more than a couple of weeks ago. I can resend it to you directly when I'm back in my office if you like... The long and the short of it, this stuff is publicly available assuming you're still interested. There was no reply to that, but elsewhere, H. Peter Anvin remarked that he thought IBM was doing their own internal POWER-4 port. Cort Dougan explained, Power4 and Power3 are different creatures from the original Power architecture (the one Sarah was referring to). The Power[34] are far more PowerPC-like and not a gothic nightmare of twisted logic like the Power. Sarah confirmed, i was refering to POWER-1, which is not similar enough to PPC, like 3 and 4 are, to be easilly ported. Also, the machines with POWER1 are MCA, which makes porting that much more difficult, and without the documentation, pretty much out of the question. Elsewhere, Tom said: I should mention that the PowerPC Linux team is just starting work on a 64 bit version of the PowerPC Linux kernel. We're hoping this will fit into something like arch/ppc64 as soon as work begins on 2.5.x. This work isn't just an internal to IBM and we want definately to work with the open source community on this. Most of the discussion for the 64 bit PowerPC kernel has been on linuxppc-workstation@lists.linuxppc.org. I encourage you to subscribe to this list if you are at all interested in this.
FS: ext2 Victor Khimenko Andreas Jaeger Mike A. Harris Mike A. Harris asked if 2.4 would support files greater than 2G on 32-bit architectures with any filesystem. Andreas Jaeger replied that yes it did support large files with ext2, but it would require a glibc recompile (version 2.1.3) and probably a 'tar' recompile as well. But H. Peter Anvin corrected, Not so. I'm using glibc 2.1.2, tar 1.13.11 and gzip 1.2.4 straight from the RedHat 6.1 distribution, and tar -z handles files > 2 GB just fine. The only reason for this, of course, is that tar was designed to be used with streaming media (and in this case it is even operating on a compressed stream fed through a pipe), thus it never seeks. Elsewhere, Victor Khimenko also explained the 'glibc' limitation. At one point, Mike asked if the resulting 'glibc' binary would still work under 2.2.x, and H. Peter replied, You need the new kernel. You don't really need the glibc upgrade to support something as simple as .tar.gz files -- you only need the glibc upgrade to support large-file seeking.
Garst R. Reese Garst R. Reese asked why '-Werror' was included as a 'gcc' option in 2.4.0-test3-pre2. As far as he knew, no kernel had ever successfully compiled with that option included. David Ford explained, This way the source will get cleaned up before 2.4.0 is released. Linus Torvalds also replied to Garst: My kernel certainly compiles. I checked that every single filesystem can be compiled with the flag. But yes, the fact that _I_ am anal about not having warnings does not mean that everybody else is. I added -Werror to see if people would fix some of the things it shows, and I've already gotten some patches. More are certainly needed. I'll probably end up dropping -Werror at some point before the real 2.4.0, if for no other reason than the fact that there are lots of gcc versions out there that complain about bogus things (the signed/unsigned comparison warnings in some gcc versions are basically not possible to fix, which is why no "real" gcc version does this warning by default to my knowledge). And anybody who isn't interested in fixing them might as well remove the -Werror from their makefile too. But maybe it will cause some cleanup.