Kernel Traffic #308 For 28�Apr�2005

In order to port kdump to x86_64, we need to have the memmap= kernel command line option available. This is so that the dump-capture kernel can be booted with a custom memory map.

The attached patch adds the memmap= functionality to the x86_64 kernel. It is against 2.6.12-rc2-mm3. I have done some amount of testing and it is working fine.

You should add a __setup somewhere, otherwise the kernel will complain about unknown arguments or generate a memmap variable in inits environment.

Comma parsing would be nice.

Otherwise it looks ok.

My Linux SATA software/hardware status reports have just been updated. To see where libata (SATA) support stands for a particular piece of hardware, or a particular feature, go to

http://linux.yyz.us/sata/

I've still got several patches from EMC (Brett) and IBM (Albert) to go through, as well as a few scattered ones from random authors.

I'm still working in BitKeeper for the time being.

Perhaps the hardest part of becoming a kernel developer is submitting your first major feature. There are technical and social hurdles to overcome and the process can be daunting to someone who is new to the community.

Thus, I'm proposing an informal project to get experienced developers to mentor new developers and coach them on the best ways to get their code ready for submission.

Developers will submit a description of their project and its current state as well as pointer to the code to the kernel-mentors mailing list. Mentors will pick for themselves which projects and developers they'd like to work with and offer their assistance.

The mentor will help the developer get their code accepted by:

• reviewing the code and suggesting how to improve it further
• acquainting the developer with best practices for code submission
• letting the developer know what to expect in the submission process

For their part, new developers will be expected to use the feedback they're given productively and eventually get their code merged!

The project list is at [email protected] with a web interface at:

http://selenic.com/mailman/listinfo/kernel-mentors

If you're interested in helping out, come join us.

Nice, it looks like the merge of this tree, and my usb tree worked just fine.

So, what does this now mean? Is your kernel.org git tree now going to be the "real" kernel tree that you will be working off of now? Should we crank up the nightly snapshots and emails to the -commits list?

Can I rely on the fact that these patches are now in your tree and I can forget about them? :)

Just wondering how comfortable you feel with your git tree so far.

Hold off for one more day. I'm very comfortable with how well git has worked out so far, and yes, mentally I consider this "the tree", but the fact is, git isn't exactly easy on "normal users".

I think my merge stuff and Pasky's scripts are getting there, but I want to make sure that we have a version of Pasky's scripts that use the new "read-tree -m" optimizations to make tracking a tree faster, and I'd like to have them _tested_ a bit first.

In other words, I want it to be at the point where people can do

git pull <repo-address>

and it will "just work", at least for people who don't have any local changes in their tree. None of this "check out all the files again" crap.

But how about a plan that we go "live" tomorrow - assuming nobody finds any problems before that, of course.

Done, pushed out. Can you verify that the end result looks sane to you? I just cheched that the diffstat looks similar (mine claims just 108 lines changed in aoecmd.c - possibly due to different diff formats).

And yes, my new merge thing seems to have kept the index-cache much better up-to-date, allowing an optimized checkout-cache -f -a to work and only get the new files.

Pasky? Can you check my latest git stuff, notably read-tree.c and the changes to git-pull-script?

I've made git merge to use read-tree -m, HTH.

I will probably not buy git-export, though. (That is, it is merged, but I won't make git frontend for it.) My "git export" already does something different, but more importantly, "git patch" of mine already does effectively the same thing as you do, just for a single patch; so I will probably just extend it to do it for an (a,b] range of patches.

http://selenic.com/mercurial/

April 19, 2005

I've spent the past couple weeks working on a completely new proof-of-concept SCM. The goals:

• to initially be as simple (and thereby hackable) as possible
• to be as scalable as possible
• to be memory, disk, and bandwidth efficient
• to be able to do "clone/branch and pull/sync" style development

It's still very early on, but I think I'm doing surprisingly well. Now that I've got something that actually does some interesting things if you poke at it right, I figure it's time to throw it out there. Here's what I've got so far:

• O(1) file revision storage and retrieval with efficient delta compression
• efficient append-only layout for rsync and http range protocols
• bare bones commit, checkout, stat, history
• working "clone/branch" and "pull/sync" functionality
• functional enough to be self-hosting (though the repository format is still in flux)
• all in less than 600 lines of Python

When I say "pull/sync" works, that means I can do:

hg merge other-repo

and it will pull all "changesets/deltas" that are in other-dir that I don't have, merge them into the changeset history graph, and do the same for all files changed for those deltas. It will call out to a user-specified merge tool like tkdiff for a proper 3-way merge with the nearest common ancestor in the case of conflicts.

Right now, "cloning/branching" is simply a matter of "cp -al" or "rsync" (mercurial knows how to break hardlinks if needed).

Some benchmarks from my laptop:

• prepare for commit of Linux 2.6.10: ~1s
• commit Linux 2.6.10: 27s
• checkout Linux 2.6.10: 45s
• full tree stat for added/changed/deleted files: <1s
• local hardlink clone: 1.5s
• empty merge between full trees: <.1s
• trivial 3-way merge with changes to Makefile: ~1s

Much thought has gone into what the best asymptotic performance can be for the various things an SCM has to do and the core algorithms and data structures here should scale relatively painlessly to arbitrary numbers of changesets, files, and file revisions.

What remains to be done:

• a halfway-usable command line tool
• remote (network) repository support
• diff generation
• changelog entry editing
• various manual interventions for merge
• handle rename
• handle rollback
• all sorts of other error handling
• all sorts of cleanup, packaging, documentation, testing...

Sample usage:

 export HGMERGE=tkmerge   # set a 3-way merge tool
 mkdir foo
 hg create                # create a repository in .hg/
 echo foo > Makefile
 hg add Makefile          # add a file to the current changeset
 hg commit                # commit files currently marked for add or delete
 hg history               # show all changesets
 hg index Makefile        # show change
 touch Makefile
 hg stat                  # find changed files
 cd ..; cp -al foo branch  # make a branch
 hg merge ../branch-foo    # sync changesets from a branch

This is my continuing attempt to make an SCM suitable for kernel hacking. It supports a distribution model similar to BK and Monotone but is orders of magnitude simpler than both (about 1k lines of code).

http://selenic.com/mercurial/

New in this version:

• much improved command line tool
• installation instructions
• commit log editing
• experimental network pull support

Some example usage:

Setting up a Mercurial project:

 $ cd linux/
 $ hg init         # creates .hg
 $ hg status       # show differences between repo and working dir
 $ hg addremove    # add all unknown files and remove all missing files
 $ hg commit       # commit all changes, edit changelog entry

Mercurial commands:

 $ hg history      # show changesets
 $ hg log Makefile # show commits per file
 $ hg checkout     # check out the tip revision
 $ hg checkout <hash> # check out a specified changeset
 $ hg add foo      # add a new file for the next commit
 $ hg remove bar   # mark a file as removed

Branching and merging:

 $ cd ..
 $ cp -al linux linux-work   # create a new hardlink branch
 $ cd linux-work
 $ <make changes>
 $ hg commit
 $ cd ../linux
 $ hg merge ../linux-work    # pull changesets from linux-work

Network support (highly experimental):

 # export your .hg directory as a directory on your webserver
 foo$ ln -s .hg ~/public_html/hg-linux

 # merge changes from a remote machine
 bar$ hg merge http://foo/~user/hg-linux

 This is just a proof of concept of grabbing byte ranges, and is not
 expected to perform well yet.

I can't quite see how to manage multiple "heads" in git. I notice that in your tree on kernel.org that .git/HEAD is a symlink to heads/master ... perhaps that is a clue.

I'd like to have at least two, or perhaps even three "HEADS" active in my tree at all times. One would correspond to my old "release" tree ... pointing to changes that I think are ready to go into the Linus tree. A second would be the "testing" tree ... ready for Andrew to pull into "-mm", but not ready for the base. The third (which might only exist in my local tree) would be for changes that I'm playing around with.

I can see how git can easily do this ... but I don't know how to set up my public tree so that you and Andrew can pull from the right HEAD.

go into your "master" working directory, and do:

git fork release ~/release
git fork testing ~/testing

Then in ~/release or ~/testing respectively, you have working trees for the respective branches.

Currently, git pull will _never_ care about your internal heads structure in the remote repository. It will just take HEAD at the given rsync URI, and update the remote branch's head in your repository to that commit ID. This actually seems to be one of the very common pitfalls for people.

The way to work around that is to setup separate rsync URIs for each of the trees. ;-) I think I will make git-pasky (Cogito) accept also URIs in form

rsync://host/path!branchname

which will allow you to select the particular branch in the given repository, defaulting to the "master" branch.

Would that work for you?

I personally like the "one repository, one head" approach, and if you want multiple heads you just do multiple repositories (and you can then mix just the object database - set your SHA1_FILE_DIRECTORY environment variable to point to the shared object database, and you're good to go).

But yes, if you want to mix heads in the same repo, you can do so, but it's a bit dangerous to switch between them, since you'll have to blow any dirty state away, or you end up having checked-out state that is inconsistent with the particular head you're working on.

Switching a head is pretty easy from a _technical_ perspective: make sure your tree is clean (ie fully checked in), and switch the .git/HEAD symlink to point to a new head. Then just do

read-tree .git/HEAD
checkout-cache -f -a

and you're done. Assuming most checked-out state matches, it should be basically instantaneous.

Oh, and you may want to check that yoy didn't have any files that are now stale, using "show-files --others" to show what files are _not_ being tracked in the head you just switched to.

I think Pasky has helper functions for doing this, but since I think multiple heads is really too confusing for mere mortals like me, I've not looked at it.

I want to have one "shared objects database" which I keep locally and mirror publicly at kernel.org/pub/scm/...

I will have several "repositories" locally for various grades of patches, each of which use SHA1_FILE_DIRECTORY to point to my single repository. So I never have to worry about getting all the git commands to switch context ... I just use "cd ../testing", and "cd ../release".

But now I need a way to indicate to consumers of the public shared object data base which HEAD to use.

Perhaps I should just say "merge 821376bf15e692941f9235f13a14987009fd0b10 from rsync://rsync.kernel.org/pub/scm/linux/kernel/git/aegl/linux-2.6.git"?

That works for interacting with you, because you only pull from me when I tell you there is something there to pull.

But Andrew had a cron job or somthing to keep polling every day. So he needs to see what the HEAD is.

I think you could just define a head name, and say something like:

rsync://rsync.kernel.org/pub/scm/linux/kernel/git/aegl/linux-2.6.git/HEAD.for-linus

and we're all done. Give andrew a different filename, and you're done. The above syntax is trivial for me to automate.

This is more or less forced by the fact that the 2.6.12-rc3 is now git based. So as of now I won't be updating linux-scsi.bkbits.net

Hopefully I've set up broadly similar functionality on www.parisc- linux.org (with thanks to Dann Frazier and Paul Bame, the parisc-linux web admins).

To view the currently available SCSI trees, go to

http://www.parisc-linux.org/cgi-bin/gitweb.pl

This is using Kay Sievers' scripts and should give you broadly similar browsing capabilities to bkbits.

I'm still pushing the diffs and the change logs for all the trees into

http://www.parisc-linux.org/~jejb/scsi_diffs

which has almost the exact same content as the old BK based ones used to have. Since git is a very mobile target at the moment, I suspect the diffs and the changelogs will be of most use to people.

P.S. if you have any general git questions, please, I don't want to hear them. Ask on the git mailing list: [email protected] (preferably after having searched the archives http://marc.theaimsgroup.com/?l=git first).