My first aproach to implement them is 'fail-safe and fail-soft':

• write meta-data back to disk in safe order.
• initialize meta-data on disk at first use.
• keep link-count of inodes zero when creating files.

Second, minimize loss of performance:

• it isn't neccesary to synhronous-update inode-bitmaps and block-bitmaps. E2fsck will repair them perfectly.
• it isn't neccesary to synhronous-update inode and indirect-block, this does'nt cause any problems.

People who wants to use extensively ramdisks should use a blocksize of PAGE_SIZE to avoid the fragmentation of not-freeable memory. NOTE: you can't generate a 4k filesystem using a blocksize of 1k simply because the buffer-size used from mke2fs must be the same used by the filesystem code otherwise you won't read on the memory you written before (on a ramdisk you can't emulate larger softblocksize). So to use a blocksize of 4k you must:

        insmod rd rd_blocksize=4096
        mke2fs -b 4096 /dev/ramdisk
        mount /dev/ramdisk /mnt

then you'll avoid the fragmentation of protected buffers (really I don't ever know if the ramdisk driver will work correctly on 2.3.14-pre2 but that's a different issue and it has only to deal with the VM not with the ramdisk internals that have only to set the protected bit upon lowlevel writes).

Insmodding rd without parameters you'll use a hardblocksize of 1k and everything will work as usual. btw currently rd.c works as usual even with 512 byte of hardblocksize because both the blockdevice layer, and ext2fs prefer to start reading/writeing with a softblocksize of 1k (and mke2fs generate filesystem with 1k blocksize as default). With the fixes now the hardblocksize and the softblocksize will be both set to 1k and this looks cleaner to me.

The last issue is with the ramdisk images: the helper function (identify_ramdisk_image) that reads the superblock only check how many blocks it contains but it doesn't check the blocksize of the filesystem. So basically you must use a blocksize of 1k if you want rd_load_image to work properly. At least now it's a know feature and the check at the end of the patch will be done right.

OK, here's a summary of what I see to be the problem with ramdisk as of 2.3.7. Note that it's partly predicated on a not-so-good understanding of the page cache, so some of the analysis is very subject to error:

First the problem: With kernel version 2.3.7 , and continuing through the current 2.3.13 version, the ramdisk driver does not function properly when mounted as root, whether loaded from initrd or from a floppy disk. Non-root ramdisk can also function improperly if direct device IO and file IO are both used on a ramdisk (for example, by loading a filesystem image by dd, then mounting it and attempting to read files). I believe this is indicative of a broader, but more subtle, problem that could affect any device.

The specific symptoms of the ramdisk problem: The filesystem is recognized, mounts properly, directory reads are OK, but file reads return bogus data. When using ramdisk as root, the kernel fails with "Kernel panic: No init found. Try passing init= option to kernel." regardless of whether there is a valid init= option present. Debugging has revealed that the kernel does find the directory entry for the init executable, but when it reads the executable file, it gets back all 00s, and so it rejects the file as non-executable. If a (non-root) ramdisk is prepared with mke2fs, mounted, files are written, then read, that appears to work OK.

The short version of why it's happening: Ramdisk keeps its data in the buffer cache, file IO checks the page cache. The file reads are missing in the page cache, and creating new buffer entries (filled from the ramdisk device, which will return 00s), even though the data already exists in the buffer cache.

The longer version of why it's happening: The rd device driver itself doesn't know anything about the actual data, it lets the buffer cache worry about storing the data and servicing reads. Whenever a read request actually gets to the rd device, it assumes that since it must have missed in the buffer cache, it's a new block that was never accessed before, so it just returns 00s.

When a ramdisk is mounted as root, the image is loaded into the ramdisk at boot-time via direct device IO, which creates buffer cache entries, but not page cache entries.

File IO reads then check the page cache, which keeps its data in the buffer cache, but is searched in a separate hash. If the data wasn't written via file IO, it will miss in the page cache search and issue a read request directly to the device. The ramdisk thinks it missed in the buffer cache, so it returns 00s. This results in 2 separate buffers for the same block on the ramdisk, only one of which has a page cache entry. Because of the way the ramdisk works, the second copy is wrong (filled with 00s).

This problem could affect any other device that mixes direct device IO and file IO as well, but the consequences are more subtle. If a buffer cache (but not a page cache) entry exists for a particular block of a device, then a file IO read (or even worse, a write) takes place to that same block, the read will miss in the page cache and create new buffers. For any device other than ramdisk, the new buffers will be filled from the physical device itself, so the data at least has a good chance of being valid. You still wind up with 2 separate buffers claiming to represent the same block on the device, not a good thing. Aside from the inefficiency (both space and speed), if those 2 buffers ever get to hold different versions of the data (which I assume would happen if a write takes place to that block), you've got big big trouble.

Some possible solutions: The obvious answer to the ramdisk problem would seem to be having it store its data in the page cache, but that would probably result in throwing out many of the advantages of the current ramdisk implementation (which is simple, efficient, and clean), and also wouldn't do anything about the broader problem. I don't like this approach because I think the problem is with the page cache implementation, not with the ramdisk implementation.

On the broader problem, I don't see any alternative to either checking the buffer cache on page cache misses, or fundamentally changing either the buffer cache or page cache implementations. If the buffer cache is checked on page cache misses, what to do if a block hits in the buffer cache, but not the page cache, is a fuzzier issue. Should the old buffer be flushed and invalidated, then a new buffer created and filled from the device? That seems unnecessarily inefficient, and would still leave the current ramdisk broken. Should the new buffer be filled from the old buffer, then the old buffer invalidated? Or should the page cache entry be created to use the old buffer without moving the data (which is not nearly as simple as it sounds)? Or something I haven't thought of?

Ok, I just have a preliminary patch that try to fix the potential data corruption that can happens in 2.3.15-pre1 and previous 2.3.x kernels (and that will automagically fix the ramdisk driver without changing its internals).

The corruption bug (that has nothing to do with the ramdisk driver) is the use of truncate_inode_page() to shrink the icache. If an inode is not in-use and it's hashed in the icache, it can have dirty or protected pages allocated in its page cache.

So when we shrink the icache so we need to release also all the page-cache pages that belongs to such inode, we can't simply mark all the page-cache-overlapped-buffers as clean in flushpage. Otherwise we'll lose data-writes and this will lead to data corruption on disk.

Previously (in 2.2.x) it was possible to use truncate_inode_pages without differences (both for shrink the icache and for truncate(2)/unlink), since the page cache was only there for reads, and both writes and protected buffers was placed in the buffer cache. This is not possible anymore since now the pagecache has dirty or protected data in it.

NOTE: with my patch applyed the blockdevice writes are still not synchronized with filesystem writes (this avoids us having to hash in the buffer-hashtable the page-cache-overlapped-buffers). So if you read from the blockdevice layer you shouldn't expect to read the last uptodate data and if you write to the blockdevice layer your writes can be lost. So just choose if to use a blockdevice in raw mode or with a filesystem on the top of it, before start using it ;).

But with the patch applyed it should be guaranteeed that if you unmount a ramdisk, and _then_ you read the ramdisk from the blockdevice layer, you'll read the right data (the page-cache will be correctly converted to regular buffers and not to orphaned-lost buffers).

Please test the ramdisk driver heavily with only this patch applyed against 2.3.15-pre1. Try to generate ramdisk images using the ramdisk itself. Make sure to always unmount before accessing the ramdisk via /dev/ram*. If you get in troubles give me a way to reproduce please ;).

i've been very impressed with the Sony MiniDisc file system -- it's a simplified file system that transparently manages data on MO MiniDiscs.

the name space is flat -- you can have 1 to 255 separate "tracks", and i've seen this system used for monaural or up to 8 concurrent channels of recording, on 128M MO disks. i'm no expert, but i'd bet a faster, larger disk could be used to boost the number of concurrent channels. it's probably just a matter of how many channels can be multiplexed into a track's block stream.

a simple TOC-based block-allocation system is used. the TOC is stored in the disk writer's volatile memory, and written out when the disk is ejected; this eliminates the extra seeks involved with metadata updates. new blocks are allocated sequentially from free space at the end of the disk, or eventually from blocks freed by erasing tracks of previously recorded material. a text area of about 2K per disk is reserved for tagging each track with "title" data.

for managing a large disk, one might extend such a file system to handle more tracks, more blocks, and generate a periodic TOC update that writes the TOC into special areas allocated across the entire disk to minimize seek latency.

this kind of system would eliminate the need for indirect blocks or even extent-based allocation, and keep metadata updates very cheap. it would also make it easy to combine, divide, and otherwise edit the data in the tracks.

This is the user-to-kernel interface for the kernel system calls. It takes the system-call number, puts it into register eax as a function code and executes the software interrupt 80 hex. Upon return, it checks for a negative value, also in eax. If it is negative, it puts the negative of the return value (now positive) into global errno, and returns -1.

Note that this is a MACRO so substitution rules apply for 'type' and 'name' Type would be typically 'int' and 'name' would be system-call number.

Look at the code generated by this (by syscall0(int sync)):

sync:
        movl $36,%eax
        int $0x80
        testl %eax,%eax
        jge .L3
        negl %eax
        movl %eax,errno
        movl $-1,%eax
        ret

Perform a syscall with __NR_sync in eax. If the result is >=0 (success), then return that value. Else, set errno to -eax and return -1.

Other syscalls are similar, except some other values are loaded into registers (up to 5 or 6 values -- not eax or esp, probably not ebp).

_syscall0(foo_t, foo) expands into approximately:

foo_t foo(void)
{
        long result;

        /* make the call */
        result=call_interrupt_vector_0x80(__NR_foo);

        if(result>=0) return (foo_t) result;

        ITYM errno=result;

        return -1;
}

If you take a look at the changes, it's not too difficult to see why the "stable" kernels are not as stable as one might like. Lots of changes get in that are not strictly bug fixes. The most direct problem is the one you point out: that the "stable" kernels are unstable. However, there are other problems with a policy of back-porting new features to "stable" kernels. It reduces the incentive to get the current development kernel closed, thereby slowing the development cycle. I think this is a big part of the reason why 2.2 arrived more than two years later than 2.0. In other words, if new features were not added to "stable" kernels, then Linus would not be under so much pressure to continue accepting patches to the developmental kernels.

Enforcing a policy of accepting only bug fixes into the "stable" kernels would have three effects:

1. the "stable" kernels would become more stable, probably much more stable,
2. the pressure on Linus (and others) would shift from keeping the development kernel going longer to getting it closed sooner, which would shorten the development cycle (I think this faster development cycle would result in many features getting into the stable kernels sooner rather than later), and
3. would make Linux much more palatable to enterprise IT departments.

People needing new features right away could either patch the stable kernel themselves or, in the case of popular features, use Alan's ac patches.

1. findsuper is a small utility that finds blocks with the ext2 superblock signature, and prints out location and some info. It is in the non-installed part of the e2progs distribution.
2. rescuept is a utility that recognizes ext2 superblocks, FAT partitions, swap partitions, and extended partition tables; it prints out information that can be used with fdisk or sfdisk to reconstruct the partition table. It is in the non-installed part of the util-linux distribution.
3. fixdisktable (http://bmrc.berkeley.edu/people/chaffee/fat32.html) is a utility that handles ext2, FAT, NTFS, ufs, BSD disklabels (but not yet v1 Linux swap partitions); it actually will rewrite the partition table, if you give it permission.
4. gpart (http://home.pages.de/~michab/gpart/) is a utility that handles ext2, FAT, Linux swap, HPFS, NTFS, FreeBSD and Solaris/x86 disklabels, minix, reiser fs; it prints a proposed contents for the primary partition table, and is well-documented.

There's a rather huge patch-set out there now, taking the 2.3.x series to 2.3.15.

This has a lot of the merge code I've been sent over the last two weeks, but I will invariably have missed some, if for no other reason than simply that I got absolutely _flooded_ by people sending me patches.

One of the more interesting things was the SMP pipe cleanup sent by Richard, but try as I might it was never really stable under load on x86 - not with the plain semaphores in 2.3.14, and not with the patches Andrea had either. I assume Richard tested it on an alpha with the much more well-thought-out atomic operation that the alpha provides.

I ended up rewriting the x86 semaphore code (and some of Richards pipe code too, for that matter, to get rid of some races in waking things up), and it doesn't show the problems I saw before, but hey, maybe I just exchanged one set of problems for another set that I can't trigger any more. Give me feedback, please.

Other features that don't impact everybody, but are rather major:

• ATM support merged in
• firewalling is gone (again), replaced by an even more generic netfilter facility.
• general networking merges and updates
• Various driver updates (ISDN, ISA PnP, sound, fbcon, usb, intelliport, you name it)
• make system call return type "long" even if the system call only returns valid data in the lower order bits - we use the high bits for error handling, and some 64-bit architectures care (read: the Merced calling conventions want this because they don't automatically extend the return type - I bet it will be a new portability issue for other programs than just the kernel)

Have fun

Well, I certainly saw strange behaviour. The trylock code seemed to be the prime culprit - it tried to decrement the "waking" count, but it could end up doing it too late so that people had already seen a increment from a concurrent "up()".

I'm not saying the new code is bug-free, but it works for me where the old one did not - and your claim that it is obviously broken is also obviously wrong, see later..

You have one choice: fix things up. It already failed, there's no point in doing anything else.

We tried to be clever before. There was absolutely no data that it was ever a win, and there were lots of indications that it was buggy. Let's not make that mistake again. Don't optimize code that doesn't need optimization.

Btw, the case you optimize for is the case that is supposed to be _extremely_ rare even in the presense of contention. You optimize not just for the contention-case, you optimize for the specific case where the values are racing and changing on different CPU's at the same time. Do you _really_ think that it is worth it, considering that you make the semaphore behaviour more complex?

I really don't.

it could be quite useful for /proc/${pid}

It might even lead to a nice solution to the problem of debugging threaded applications. Here a lot of the problems with gdb getting things right have to do with reparenting and how thread libraries implement things internally, signals, what have you.

When you can do selects on /proc/${pid}/debug or whatever, a lot of the "what should we do if xxx" questions then have answers.

it makes sense to have __init in front of init_cmpci() but it seems rather suspicious to have it for a module since the code for throwing away .init* stuff is only called from free_initmem() on boot and does not seem to be used on loading modules?

On the other hand, if it *is* needed for init_module() then plenty of other places must be modified to have __init. So, in all cases, some changes are required.

This is a second attempt at an improved PLIP driver. It does the following:

• Allow using PLIP with parallel ports without IRQ.
• Move the PLIP driver to use the generic parallel port API, rather then using direct I/O access.
• Use MAC addresses on PLIP ethernet packets all the time (this also fixes a problem that was reported here a few weeks ago, of tcpdump not handling plip correctly). I'm afraid that I don't fully understand what dev->rebuild_header() is all about, and I couldn't get the old version of the code to do anything useful, so I changed the way of doing things. Any hints here would be appreciated, though I find the current solution quite satisfactory, as it works and has no outstanding overhead.

The patch as follows below compiles and works with 2.2.10. It should work with any other 2.2.x. It also compiles (and work) with 2.3.12, and probably with other development kernels, with a slight change.

Please let me know if this needs any more changes. I can see no reason not to include this in the kernel.

The patch can (also) be found at http://thor.prohosting.com/~zimerman.

Well, a very nice fellow at IBM Web Library sent me an electronic copy of the IBM Local Area Network Technical Reference. It's kind of old (Dec. 1990) but it describes wire-level NetBEUI, info I haven't found anywhere else. Features have been added since then, but I think this'll do to get a basic implementation going.

They are copyrighted, so I seriously doubt I could simply make the docs available, but the first thing I intend to do is summarize it into a simple spec. I don't have a clue how to implement a LAN protocol in Linux, but I'll see what I can do.

I remember NetBEUI coming up occasionally here. If you have any interest, skills or info, feel free to contact me. I'll announce when I have useful info up.

NetBEUI is waning in popularity, but it is a fast protocol and, like IPX, useful in legacy environments. Like DLC, it's commonly used for network printers.

The ipvs folks are doing IP level load balancing with inverse masquerade--ie

                        internet
                          |
                        [MASQ]
        +------+--------+-------+-------+------+
       www1   www2    www3    www4     www5   www6

incoming SYN frames create masquerade sessions mapping user->www to user->www1 www2, etc according to load and rules.

In one mode they set things up differently as follows

                internet
                   |
    +-----+-----+-----+-----------+----------+---------+
    |     |     |     |           |          |         |
 MASQ    www   www   www         www        www       www
         www1  www2  www3        www4       www5      www6

Each host is www and a unique name. The masq host arps for www, then tunnels the packet to a www[1-n] but without rewrite. The reply doesnt touch the masq so is a lot faster.

However they need to stop www1->wwwn arping for their www tunnel address

I would like to report success in modifying the tulip.c version 91g to work with linux kernel 2.3.14 and above, I have tested it on my home network by transferring linux-2.3.12.tar.bz2 from one system on my network to my development system which is using the tulip driver. The first time I tried to transfer the kernel it stalled (the tulip dropped off the network) at 7.5Mb transferred (at over 700kbps) I did ifconfig eth0 down and ifconfig eth0 up right after .. telnetted some, sent some email ftpd some more and tried to ftp the linux-2.3.12.tar.bz2 again .. this time success, again the speed is over 700kbps.

So it works for me :-)

Since the driver is over 100kb (39kb gzipped) I'm not attaching it here, If you want to try it, it can be optained through ftp:

I have the tulip.c, a gzipped version and a patch against the tulip v91g from

up there. I have made it clear in the header of the file and the startup banner that I'm the one who modified the file, so please don't send any nasty bugreports to Donald or the linux-tulip list without verifying that the bug exsists in the original tulip driver as well.

Anyone who has sweated to build a 2.88 CD boot image from syslinux and the skinniest fat kernel possible will understand why I'm posting this. There is only a minor impact on the kernel, but overall it could make quite a difference to the way Linux installations are done.

I was reading through the El Torito specification a few months ago in order to debug something and noticed that the excellent mkisofs doesn't support some very interesting features of it. Neither does any other CD image mastering program I know of. As a result every x86 OS installation facility is somewhat crippled. Linux isn't as crippled as NT from this perspective, but still, booting from CD is nothing like home.

I am referring to the PDF of the hacked-up spec that we all have to use for CD booting, http://www.nikko.simplenet.com/goldentime/cdrom7.zip. This could turn into a major marketing feature for Linux as well as relieving those poor people who have to write 2-stage installation programs.

First some observations on the annoying habits of everyone's install procedure:

• It's really _really_ unintelligent. You boot off the CD, get several questions into the install and then it asks you what install media you want to use, then to insert the correct CD(!) and then chains to the second-stage install. This has always annoyed me. There are some good reasons for this behaviour related to the way El Torito gives you an emulated floppy. Please note that I haven't written an installation program, in fact the source of nearly every installation program I've ever seen makes me feel rather ill. Mandrake and some others are changing that bit by bit, thankyou :-)
• It's very limited because you are cramming everything into 2.88Mb. This is also a bit silly, you have a great big CD ROM and the best hardware-detecting OS in existence so why not just allow the data and code for the installation program to reside on CD and take up whatever room it wants to?

So what does El Torito let us do?

1. You can have multiple boot images

   A quick and simple alternative solution to what most people currently use that would help a bit is to have an initial CD image on a 2.88 floppy with little more than the kernel and some kind of mixed human and/or automatic logic to choose which of an arbitary number of other floppy boot images to boot as a secondary bootstrap. This might be the quick and dirty option for getting around current space problems. As far as I can tell you can chain around between images all day within the spec. I think the SuSE installer does this to some extent when they offer either an installation or an emergency boot floppy. But I haven't dived into it to see for sure.

2. You can have no-emulation booting.

   This is the really interesting bit. Quote from the PDF in section "5.3 No Emulation Booting":

   --- start quote -------------
   When the Media Type is set to zero the BIOS does not use the CD to emulate a disk. The boot operation loads the requested number of sectors directly to the specified segment. When loading is complete the BIOS will jump to segment:0. The associated piece of software can be a "loader" (which provides its own CD interface), or it can be a stand alone program. The El Torito specification allows for the loading of FFFF sectors (This would allow the BIOS to fill the entire low 640k memory area with data). Once the system jumps to segment:0, the program can retrieve its boot information by issuing INT 13, Function 4B, AL=01. After the boot process has been initiated the INT 13 Extensions (functions 41-48) will access the CD using 800 byte sectors and the LBA address provided to INT 13 is an absolute sector number. This gives any program running in no emulation mode the ability to locate the boot catalog, and any other information on the CD, without providing a device driver.
   ---- end quote ----------

   In other words you can get a CD image to boot from a kernel jump-point. This isn't like a hard drive bootstrap, this actually loads up to 640k of data into memory and then sets the IP to an entry point. The kernel can then work out how to mount /. There would be no commandline parameters so I would think there would have to be a compile-time option to say "root fs on CD ROM".

   Provided we can get clever with the CD ROM drivers this is exactly what we need; the net effect would be that the installer developers could treat the CD exactly like a hard drive, with access to as much data as they liked. Booting Unix from read-only filesystems was solved years ago.

   OK, What Next?

   Someone has to sit down with an ISO image and do some sector editing and build one of these. Once we know what works and what doesn't then we can hack mkisofs. I'd probably be tempted to hack mkisofs first, it's fairly obvious what needs to change.

   I don't have any reason to do this apart from hack value. If someone else does I'm sure they'll get to it long before I've even looked at it. It might make sense for one of the distributions to fund some experiements.

   To head off some obvious suggestions:

   • I sent this to eric@andante.jic.com, the author of mkisofs. Bounced.
   • There was a thread called "Merging EXT2 and El Torito" on linux-kernel over a year ago, and it was nothing to do with this.

This sounds really interesting! The big question is how many systems actually faithfully implement this part of the El Torito spec. Sorry for being paranoid, but the reality is that if it's not used commonly, there is an all-too-unfortunately high probability that some cheasy Taiwan-special (or even made in America) hardware won't support it correctly. <Insert standard grumbling about the cheap-sh*t Wintel hardware industry.>

Unfortunately, the only way I know for sure to check would be to make some ISO images available and ask people to test it and see whether it works.

Definitely. Note that you can't just bootstrap a bzImage this way; you still need a boot loader to go before the kernel. As I've already mentioned I intend for lbcon to support this booting mode (with full access to the ISO 9660 filesystem), but lbcon isn't ready for prime time just yet.

*Furthermore*, no-emulation does have a limit on the size of the image (640K), which means it may not be suitable to simply attach a stub loader to the image anyhow (after all, you might as well just make a disk image if you're going to do that.)

Hard disk emulation is known to be broken on many BIOSes, and as such isn't really an option.

After a very long delay, caused by my being terminally busy at MIT, and then changing jobs, I've finally gotten an update to the serial driver which supports PCI patches. It can be found at:

This driver supports for the Oxford Semiconductor 16C950 UART, and good collection of PCI boards (basically everything for which people have sent me patches, or for which I've been able to get my hands on the PCI serial cards).

I've tested this driver with ConnectTech, Sealevel, and GTek serial boards, and it has support for the SPCom 200 and Keyspan boards as well. I haven't tested the latter since I don't have the boards, though.

The sources are designed to work on either 2.2 or 2.3 kernels, and as shipped it comes with a Makefile which allows you to compile the serial driver as a stand-alone module outside the kernel tree. Alternatively, it should be pretty obvious how to copy the relevant source files (serial.c, serialP.h, serial_reg.h, serial.h, etc.) into the right places into the kernel, which will build the new serial driver as part of 2.2 or 2.3 kernel.

I'd like to send an update of this driver to Linus fairly soon for inclusion into the 2.3 mainline, so please send me any comments you might have. In particular, if you have some other PCI serial boards other than the ones supported by this driver, please send me the vendor, device, subvendor, and subdevice id numbers, how the PCI board interfaces to the system (mapped I/O memory, I/O ports) and what clock is used to drive the UART (i.e., what base_baud setting is needed; some drivers use a faster clock crystal which allows the port to run at speeds greater than 115200 bps).