g-octave 0.3 was released today and come with some cool features:

• Modular support for all the package managers available for Gentoo Linux
• Optional logging for a file
• Update of installed packages
• Override configuration options from the file '/etc/g-octave.cfg' with environment variables

The announcement is here:

http://article.gmane.org/gmane.linux.gentoo.science/1339

Please test it and share your thoughts with us :)

Thanks!

All important modules are ported. Some need a few tweaks but generally Dracut provides the same functionality Genkernel initramfs generator does.

Here's summary (new things are bold):

Module Status Required pkgs 00dash OK app-shells/dash 01fips FIXED¹ app-crypt/hmaccalc 10redhat-i18n FIXED: 10i18n 10i18n NEW 10rpmversion OK 40network OK >=dhcp-4.0
&& sys-apps/iproute2
&& net-misc/bridge-utils 45ifcfg OK 50plymouth OK >=sys-boot/plymouth-0.8.3² 60xen OK app-emulation/xen 90btrfs OK sys-fs/btrfs-progs 90crypt ENHANCED³ sys-fs/cryptsetup 90dm FIXED sys-fs/device-mapper
|| sys-fs/lvm2 90dmraid FIXED sys-fs/dmraid
&& sys-fs/multipath-tools 90dmsquash-live OK sys-apps/eject
&& checkisomd5 for check functionality
(not in portage or couldn't find) 90kernel-modules FIXED sys-apps/module-init-tools 90lvm OK sys-fs/lvm2 90mdraid OK sys-fs/mdadm 90multipath FIXED sys-fs/multipath-tools 95rootfs-block OK 95terminfo OK 95udev-rules OK sys-fs/udev 95uswsusp FIXED sys-power/suspend 98syslog OK app-admin/syslog-ng
|| app-admin/rsyslog 99base OK 95dasd SKIPPED⁴ 95dasd_mod SKIPPED⁴ 95debug ENHANCED sys-process/procps &&
dev-util/strace &&
net-fs/nfs-utils &&
sys-apps/net-tools &&
net-nds/rpcbind &&
net-misc/openssh && 95fcoe NO TOOLS lldpad
&& fcoe-utils
&& vconfig (vlan-tools) 95iscsi OK sys-block/open-iscsi[utils] 95nbd OK sys-block/nbd 95nfs OK net-fs/nfs-utils 95resume OK⁵ 95zfcp SKIPPED⁴ 95znet SKIPPED⁴ 98selinux OK

1. Fixed, should be OK, but not tested because setting it up is PITA and not especially important.
2. Not yet in Portage tree but ebuild from Martin Benz (see #274065) works without a problem with my fork of Dracut. Martin reported tiny problem with dracut-006. I've posted workaround as dracut-006-r1. Reported problem is already properly solved in official Dracut's git repo.
3. Added support for keys on external devices but it's not fully finished. Needs some more work, but it's not so important for now (since it worksforme ;-)).
4. Do it later, maybe. Have to set up special environment for them, e.g. dasd is for S/390, so some emulator…
5. Tested with µswsusp only.

Moreover I've refectored the ebuild, added new USE flags and updated dependencies. There's just kernel config check left. Ebuild works with my repo. When new version of Dracut will be released (with things I've submitted) just small details will have to be changed.

Now I'm switching to Genkernel integration.

References:

• activity at LKML: http://www.spinics.net/lists/linux-initramfs/
• my repo: http://github.com/aidecoe/dracut (branches gsoc2010 and gentoo)

Hi folks,

here is my weekly report #7.

GSoC report: g-Octave (Improve Octave/Matlab support) July 6

• Small fixes on the g-octave.eclass
• Changed default db_mirror to use the files on the new server.

July 7

• No work on the project. Last week of classes. Overloaded with homework.

July 8

• More homework to do. Nothing done for the project.

July 9

• Long travel to my mother's town. Nothing done again.

July 10

• added support to the override of configuration values using environment variables.
• Implemented the automatic management of the g-octave overlay for portage.
• Created an module to store the installed packages. Useless. Will be removed.

July 11

• Added modular support to package managers (g_octave/package_manager.py).
• Implemented the Portage class.
• Implemented the Pkgcore class.
• Added support to package updates (-u option from the command line).
• Fixed some test case.

July 12

• A bunch of fixes for the support of multipĺe package managers works fine.
• Implemented the Paludis class.
• Some fixes to the package updates.
• Added the new options the configuration file template.
• Added a bunch of logging calls to the source code.
• Fixed some test case.

That's all for now.

Since my original post from two years ago didn’t reach yet all the users, and some of the developers as well, I would like to reiterate that you should not be enabling ccache unconditionally.

It seems like our own (Gentoo’s) documentation is still reporting that using ccache makes build “10 to 5 times faster”. I’ll call this statement for what it is: bullshit. The rebuild of the same package might have such a hit, but not the normal emerge process of a standard user with Gentoo. If anything at all, the use of ccache will slow your build down, and even add further failure cases and make it difficult to identify errors.

Now, since the approach last time might not have been clear enough, let me try a different one, by describing the steps it takes when you call it:

• it has to parse the commandline to make sure you’re calling it for a single compile, it won’t do any good if you’re using it to link, or to build multiple source files at once (you can, especially if you use -fwhole-program, but that’s for another day to write about), so in those cases, the command is passed through to the compiler itself;
• once it knows that it’s doing a single compile, it changes the call to the compiler so that instead it simply preprocess the file, and stores the result in a temporary area;
• now it’s time to hash the data, with md4 (the parent of MD5), that as the man page suggests is a strong hash; this has good reasons to be strong, but it also means that it takes some time to hash the content; we’re not talking about the source files themselves, that are usually very small and thus quick to hash, but rather of the preprocessed file, which includes all the headers used… a quick example on my system, by just including eight common header files, produces a 120KB output (with -O2 and _FORTIFY_SOURCE… it goes down to 93KB if -O0 is used); to that add the extra information that ccache has to save (check the man pages for those);
• now it has to search the filesystem, within its cache directory, if there is a file with the same md4; if there is, it gets either copied (or experimentally hardlinked, but let’s not go there for now), otherwise the preprocessed file is compiled and copied in the cache instead; in either case, it involves copying the object file from one side to the other.

Now, we can identify three main time-consuming operations: preprocessing, hashing and copying; all of them are executed whether this is a hit or a miss; if it’s a miss you add to that the actual build. How do they fare about the kind of resources used? Hashing, just like compiling, is a CPU-intensive operation; preprocessing is mixed (you got to read the header files from around the disk); copying is I/O-intensive. Given that nowadays most systems have multiple CPU and find themselves slowing down on I/O (the tinderbox taught me that the hard way), the copying of files around is going to slow down the build quite a bit. Even more so when the hit-to-miss ration is high. The tinderbox, when rebuilding the same failing packages over and over again (before I started masking the packages that failed at any given time), had a 40% hit-to-miss ratio and was slowed down by using ccache.

Now, as I already wrote, there is no reason to expect that the same exact code is going to be rebuilt so often on a normal Gentoo system… even if minor updates to the same package were to share most of the source code (without touching the internal header files), for ccache to work you’d have to leave untouched compiler, flags, and all the headers of all the dependent libraries… and this often includes the system header files from linux-headers. And even if all these conditions were to hold true, you’d have to have rebuilt object files for a total size smaller than the cache size, in-between, or the objects would have had expired. If you think that 2GB is a lot, think again, if you were to use -ggdb especially.

Okay now there are some cases where you might care about ccache because you are rebuilding the same package; that includes patch-testing and live ebuilds. In these cases you should not simply set FEATURES=ccache, but you can instead make use of the per-package environment files. You can then choose two options: you can do what Portage does (setting PATH so that the ccache wrappers are found before the compilers themselves) or you can simply re-set the CC variable, such as export CC="ccache gcc". Just set it in /etc/portage/env/$CATEGORY/$PN and you’re done.

Now it would be nice if our terrific Documentation team – instead of deciding once again (the last time was with respect to alsa-drivers) that they know better what the developers should support – would understand that stating in the handbook that ccache somehow magically makes normal updates “5 to 10 times faster” is foolish and should be avoided. Unfortunately upon my request the answer hasn’t been what you’d expect from logic.

Hello there !
Week 7 finally paid off, we have a C api for layman, based on Brian Dolbec's new Python layman API.
It can be downloaded from my git repository :

git clone http://soc.dev.gentoo.org/~cazou/c-layman

To test it, you'll need Brian's Python lib. You can get it from git too :

git clone http://dev.gentooexperimental.org/~dol-sen/git/layman.git

and switch to the API branch

git checkout api

(if this last one does not work, try git checkout -b api remotes/origin/api)

As API's are not stable yet, it is possible that it sometimes won't work at all, Try to always use the latest git revision for both libraries.

The method I decided to use is the interpreter one. Calling a python function from C becomes as easy as :

int bareConfigSetOptionValue(BareConfig* cfg, const char* opt, const char* val)
{
PyObject *obj = PyObject_CallMethod(cfg->object, "set_option", "(zz)", opt, val);
if (obj)
return 1;
else
return 0;
}

which is quite handy.

The C lib is still crappy with memory management and does not have a building system yet.
To use it, enter the 'src' directory and run :

gcc -W -Wall -L/usr/lib/python2.6/ -lpython2.6 -I/usr/include/python2.6/ -o tester --std=c99 -g *.c

Using python2.6 is important here.
At runtime, you must let python know where the layman api is :

PYTHONPATH=/where/you/put/layman.git/:$PYTHONPATH ./tester

It should update the overlay list and print that list.

Next week, I'll work on getting it stable, fix all memory leaks, document it properly, even within the code and rename some function like interpreterInitialize() --> laymanInitialize()

Comprookie clones a failing hard drive. Meet the new Gentoo Council members.

LINKS:

Gentoo Council 2010-2011

• Petteri Räty (betelgeuse)
• Tony Vroon (chainsaw)
• Brian Harring (ferringb)
• Mark Loeser (halcy0n)
• Jorge Manuel B. S. Vicetto (jmbsvicetto)
• Tomas Chvatal (scarabeus)
• Alex Alexander (wired)

http://www.gentoo.org/proj/en/council/

GParted Live
http://gparted.sourceforge.net/livecd.php

Clonezilla
http://clonezilla.org/

Podcast Headphones
http://linuxcrazy.com/downloads/headphones.jpg

Download

Today, I am gonna rant a little bit wrt the QA status of many packages residing in Gentoo portage tree.  Few of my fellow colleagues think that all that matters is the # of ebuilds in our portage tree. Adding more and more ebuilds in our tree is a good way to prove that Gentoo is an active and cutting-edge distro. Sorry, but you failed :-)

Looking through our QA bugs[1] it looks pretty obvious ( at least to me ) that the ebuild QA level has lower priority than it should be. The most common bugs are

• packages that don’t respect the user CC/C{XX}FLAGS/LDFLAGS
• wrong dependencies ( listed build dependencies as runtime dependencies )
• packages that bundle external libraries == security problems
• Badly written ebuilds. Missing || die statements, install precompiled binaries to /usr
• etc etc etc

Whilst I do understand that it is not possible to always spot these type of QA errors, this looks quite bad to the rest of the community. Honestly I would prefer a much lower ebuild number on portage but with QA level than having broken ebuilds everywhere just to claim that “Gentoo has the latest version of each package” available.

The lack of manpower problem is more than obvious, so do not be afraid to ask help from our user community. Let them become proxy-maintainers[2] for the packages that you have little or no interest  in maintain them anymore.  Teach some of them to write proper ebuilds and/or bring them to developer family by mentoring them. Furthermore, if you do not care for a package anymore, just say it out loud and let somebody else maintain it :)

[1]:  QA bugs

[2]: proxy-maintainer

I don't normally post about new Paludis releases, but this one brings a new client that many has been asking about.

From NEWS:

• The ‘cave’ client is now enabled by default. ‘cave’ is a modular console client that will eventually replace ‘paludis’. It is currently reasonably functional and well tested, but does not yet have all the features present in ‘paludis’, and is thus not yet considered a complete replacement.

See this post for more on cave.

Without further ado… the change log:

• Fixed: Catch keyboard interrupts (Ctrl+C) while running child processes properly
• Fixed: doctest error in tar overlay code (bug #327491)
  Thanks to David Abbott for reporting!

MPlayer just very recently got support for playback of unencrypted Blu-Ray discs using libbluray.  (Thanks to all the devs and testers! )  Apparently development for the library is being hosted on VLC's git servers now, something I had no idea about.  I thought the project was dead upstream.

I'm adding an ebuild for libbluray to the gentoo multimedia overlay if someone wants to access it.  It's something I plan on pushing into the mainline tree soon enough, once it's properly finished.

If you are building MPlayer from SVN, it will automatically detect the new library, and build against it.  You can use the -9999 ebuild in the portage tree.

To playback some of your Blu-Ray content, you will first need to extract it to your harddrive.  I use MakeMKV, also in the multimedia overlay, to accomplish that.

Here's a simple way using the CLI to dump the contents:

$ makemkvcon backup --decrypt disc:/mnt/bluray/ <location to dump content>

The syntax for playback is:

$ mplayer br:// -bluray-device <path to dumped content>

By default, it will play the longest playlist (I think).  If you can get the list of playlists available, you can pass that as an optional parameter to br:// (fex: list_titles /home/steve/bluray/src; mplayer br://5 -bluray-device /home/steve/bluray/src).

libbluray also ships with a few example programs that do basic stuff like listing the titles (list_titles), dumping information about the playlists (mpls_dump), and a few more (sound_dump, index_dump, mobj_dump, libbluray_test, bdsplice, clpi_dump).

Have fun with it.

In this post, I'm going to spend some bytes telling you why PHP-5.3(.2) is awesome. The goal is to get you wanting to type echo "=dev-lang/php-5.3.2" >> /etc/portage/package.keywords followed by emerge php after reading this.

In my current little survey, 6 out of 7 who have voted so far, said that their code is ready for PHP-5.3. So they probably know about the goodies already and hopefully have already tried Gentoo's all new and shiny php-5.3.2 ebuild or even the php-5.3.3 RCs Ole maintains in our overlay.

For the rest of you, I'll highlight some points of the PHP-5.3 migration list:

Late Static Binding

Here's what the PHP website says: [...] can be used to reference the called class in a context of static inheritance [...] So far, so unclear. But it continues:
[...] static:: will no longer be resolved using the class where the method is defined but it will rather be computed using runtime information.
This is most useful in a class hierarchy, where your (abstract) base class does most of the work, but needs bits of information from the child class. Using static::, you can call back to methods of your child class from the base class without your base class having to know about the particular child class at all! This saves you numerous instanceof tests.

I use this in a MVC-Framework, where a BaseController holds a generic way to generate pagination data and forward it to the view. It also collects cookie and database information to save and restore user's preferences. This forced the cookie and database query methods into the ChildControllers, as the particular cookie or database column name was specified there. With PHP-5.3 I can now access those bits via static:: calls in the base class! No need for code duplication anymore.

Closures

If PHP-5.3 had contained no new features but this one, I'd still be happy. Closures are a powerful concept, but as the Late Static Binding above, can take a little bit of time to get into. In short, you can do everything you could do with create_function but instead of hiding all your code in a big string (loosing all the syntax highlighting and error checking of your editor), you now have perfectly valid PHP code inside a function you can declare about anywhere you could use a statement. You can assign your function like so:

$f = function($v) { return $v*$v; }
$f(2); // 4

Or you can use it directly, like so:

$squares = array_map(function($v){ return $v*$v; }, range(1,10));

This even saved me some precious horizontal space compared to array_map(create_function('$v', 'return $v*$v;'), range(1,10));

For additional coolness, you can let the closure use data from the parent scope (the context it is defined in, not the one it is run in), something you could achieve using create_function and embedding the extern variable in the string - it's so ugly I won't repeat it here. But look at the cleanliness of the closure approach (taken from the PHP manual):

$callback =
function ($quantity, $product) use ($tax, &$total)
{
$pricePerItem = constant(__CLASS__ . "::PRICE_" .
strtoupper($product));
$total += ($pricePerItem * $quantity) * ($tax + 1.0);
};

array_walk($this->products, $callback);

You definitely want to give closures a try!

Ternary default operator

Folks have lobbied long and hard for this. In short, it's your well-known ternary operator (bool ? if-true : if-false) shortened to return the evaluated ("bool" in my case) statement if you supply nothing for the "if-true" part but "bool" actually evaluated to true. Here's an example:

return 'gentoo is cool' ?: 'who wants to read this, anyway?';

This will return "gentoo is cool", as a non-zero-length string evaluates to true and nothing was given to be return in this case.

New date/time functions

Not much to see here: you can now use date_add(), date_diff() and date_sub() for which I've been waiting for ever since I needed to display dates and calculate expiry times. strtotime() is good for a lot of things, but sometimes you need more.

Misc

Some new stuff I still need to test out:

• The new mysqlnd library now returns correctly typed data instead of strings only for a query result


• SPL gained a host of basic data structures like Heap, PriorityQueue or Stack
  
• Phar archive support is now shipped with PHP
  

As expected, some things stopped to work in PHP-5.3 - they are documented in the list of Backward Incompatible Changes. The most notable changes for my code were various array functions (like usort) not accepting anything besides arrays anymore and call_user_method() now issuing E_DEPRECATED notices. Also note that the POSIX regular expressions are now deprecated in favor of Perl regexes.

That's it for now, I hope you're on your way to emerge php (if you haven't done so already)!

Simplified, Layman’s main job is to make calls to version control utilities. A larger portion of that invocation code has been rewritten: no more os.system(), no more shell interpretation in between. Here’s the change log since 1.3.4:

• Handle command line calls “layman” and “layman foo” better. Former now shows a usage summary while latter now reports an error.  Both of them kept quiet before.
• Replace os.system() by subprocess.Popen()
• Close stdin of child processes when run in quiet mode to avoid running into infinite and blindly interactive sessions.

Please report bugs for it.  Thank you!

Greetings, following is the weekly update for gpypi2 project. It's main purpose is to generate ebuilds from Python Package Index. Quick links to project info:

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit
• wiki/user stories for scrum development

Last two weeks were crazy at university. Now I shall have time to work on gpypi2 full-time until the end of program.

Install gpypi2 $ sudo pip install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip or $ sudo easy_install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip Create an ebuild $ sudo gpypi2 create flask * Generating ebuild: Flask 0.4 * Your ebuild is here: /usr/local/portage/dev-python/flask/flask-0.4.ebuild * Dependency needed: Werkzeug * Dependency needed: Jinja2 * Generating ebuild: Werkzeug 0.6.2 * Your ebuild is here: /usr/local/portage/dev-python/werkzeug/werkzeug-0.6.2.ebuild * Generating ebuild: Jinja2 2.5 * Your ebuild is here: /usr/local/portage/dev-python/jinja2/jinja2-2.5.ebuild * Dependency needed: Babel * Generating ebuild: Babel 0.9.5 * Your ebuild is here: /usr/local/portage/dev-python/babel/babel-0.9.5.ebuild NOTE: ebuilds will probably not function yet, features are yet to be completed/added. Report bugs to issue tracker. Previous week (21st June - 5th July) Task: Use SrcUriNamer gpypi2.enamer.SrcUriNamer is awaiting for quite 2 weeks now to be finally used. It's basically factory that generates SRC_URI and HOMEPAGE according to mirror provider. A lot of additional refactoring will be needed to Enamer to use this utility instead of current guessing/parsing. Not done. With Jesus we decided to postpone this for one of the last tasks, since it's not that big benefit to gpypi2 and there are more important details to be implemented. It's also a big blocker for good unittest coverage, but that shall be sorted out. Task: Fix bugs reported by users Lot of bugs are going to pop up; fix bugs, write tests and add documentation. None reported :( Task: Increase test coverage I'm still not satisfied with test coverage. I'll try to increase it a bit more. It's 10 days week of exams on my university, soon I will be able to put in more effort in implementing features. Unittest coverage is about 75%. Won't get further than that until SrcUriNamer is properly implemented and tested. Few additional tests can be made at PortageUtils, but that's a minor issue.

Here is a rundown of additional changes to code:

• slight improvements to documentation
• option to choose PyPi base URL
• changed behaviour to use XML-RPC and SETUPTOOLS/SIMPLEPAGE method for retrieving SRC_URI
• updated licences dictionary mapping
• fixed echo command to work properly (without cleanup atm)

Upcoming week (22nd June - 28th June) Task: Write configuration skeleton Currently, gpypi2 code only takes configuration from command line. I'd like to extend that to config file (.ini), get data directly from setup.py and interactive questionaire. In order for that to function properly, configuration needs to be done in right sequence. Following week I will write down the design and implement initial working prototype. Task: sync command - populate overlay with successful ebuilds from pypi When I will get bored or stuck with task #1, I'll start working on command that will generate ebuilds from all packages in pypi to an overlay. Cheers, Domen

Most of modules are checked. Some of them needed fixes, some only needed dependency updates in ebuild. Checking involved going through install and check scripts which operate on host system, analyzing if result image looks as expected, running test if available. Sometimes problems are more or less deeply hidden. I had some with networking. Stable and ~arch dhclient gets DHCP info but doesn't assign address to interface (at least under qemu). Moreover Dracut relies on dhcp-4 and invocations are quite hardcoded. To make it work well I had to use hardmasked dhcp-4 package. I also spent quite much time (and wonder if it's not completely wasted time…) debugging fips module. Probably found the reason. Dracut installs binaries with its library dependencies. Needed sha512hmac depends on libnspr4.so.8 (ldd says so), but in fact, as strace says, it searches only for libnspr4.so (on host system there's symlink libnspr4.so → libnspr4.so.8). Dracut relies on ldd output and doesn't create reverse symlinks, so we have to enhance inst_binary function… but anyway it's strange case, especially as it has never occurred before.

Shortly and clearly. Here's what's done (new things are bolded):

Module Status Required pkgs 00dash OK app-shells/dash 01fips SMALL PROBLEMS app-crypt/hmaccalc 10redhat-i18n FIXED: 10i18n 10i18n NEW 10rpmversion OK 40network OK >=dhcp-4.0 sys-apps/iproute2
&& net-misc/bridge-utils net-misc/iputils
&& sys-auth/nss-mdns sys-libs/glibc 45ifcfg OK 50plymouth NO TOOLS YET 60xen OK app-emulation/xen 90btrfs OK sys-fs/btrfs-progs 90crypt TO ENHANCE sys-fs/cryptsetup 90dm FIXED sys-fs/device-mapper || sys-fs/lvm2 90dmraid FIXED sys-fs/dmraid sys-fs/multipath-tools 90dmsquash-live OK sys-apps/eject
&& checkisomd5 for check functionality
(not in portage or couldn't find) 90kernel-modules FIXED sys-apps/module-init-tools 90lvm OK sys-fs/lvm2 90mdraid OK sys-fs/mdadm 90multipath FIXED sys-fs/multipath-tools 95rootfs-block OK 95terminfo OK 95udev-rules OK sys-fs/udev 95uswsusp FIXED sys-power/suspend 98syslog OK app-admin/syslog-ng || app-admin/rsyslog 99base OK sys-apps/coreutils
&& sys-apps/util-linux
&& sys-apps/module-init-tools sys-apps/sed
&& sys-apps/sysvinit

There are some modules left and I'm working on them:

• 95dasd
• 95dasd_mod
• 95fcoe
• 95iscsi
• 95resume
• 95zfcp
• 95znet
• 98selinux

Somehow checked and rather working:

• 95nfs
• 95nbd
• 95debug

Moreover I've updated Ramereth's Dracut ebuild to release 006 and he's going to put it into portage tree today. There's also ebuild using my git repo in my git repo at branch gentoo in gentoo directory. :-)

References:

• some activity at LKML: http://www.spinics.net/lists/linux-initramfs/
• my repo: http://github.com/aidecoe/dracut (branches gsoc2010 and gentoo)

Hi there,
This week has been the one were I realized that most of the remaining bugs in Kuroo can't be solved without rewriting some parts of the emerge/portage part of the code.
Instead of doing that, I decided to focus on the C portage library, which will eventually solve those problems.
So, if you don't remember, a C portage API will be written, based on a new python API, so that non-Python applications can easily use portage.
There must be a way to run python code from C and interpret the results in C. And there are 2 ways of that :

• Running a python script which uses the portage API and gives the result on STDOUT, then redirect this output to the library by a socket.

• Directly embed Python in C with the C Python interface directly provided by Python.

The first solution is the one used in PackageKit. The reason is most likely because the portage API changes a lot and it is easier to modify a Python script than a C program using the C Python interface.
The goal of having a new portage Python API is to get to a stable API for portage. The stability of this API is a good reason for using the second option.
The second solution is also less demanding in CPU (no need to fork, use a socket, redirect output and parse it)

My guess is that we will use the C Python interface. I actually wrote some code to test all that and you can access it from my git repository :

git clone http://soc.dev.gentoo.org/~cazou/c-portage c-portage

There is no building tools yet but the compile command is at the start of the source files (which are 2 different programs).

There is also the kuroo4 live ebuild online. You can access it by :

git clone http://soc.dev.gentoo.org/~cazou/ebuilds ebuilds

This repository will also contain ebuilds for the C portage API.

Hi folks,

here is my weekly report #6.

GSoC report: g-Octave (Improve Octave/Matlab support) June 30

• More changes on g-octave to be able to work with SVN stuff.
• Finished the stuff to create new package databases.

July 1

• Introduced the support of the installation from SVN to the module g_octave.ebuild.
• Fixed some test cases.
• Changed g-octave to be able to work with a package database installed by portage. (octave-forge databases now "live" on the SRC_URI of g-octave's ebuild)
• Changed g-octave to only fetch a package database directly from the mirrors if the user is with the live version of g-octave installed.
• Fixed the documentation for the coming release.
• More minor fixes.
• Released g-octave 0.2.

July 2

• Fixed the ebuilds to g-octave 0.2 and to add the pysvn dependency.
• Bug fixes: missing source tarballs (optim and ode), and missing $OCT_PKGDIR for the fresh octave installations. (reported by Paulo Matias, thanks!)
• Released g-octave 0.2.1.

July 3

• Added support to logging actions to a file. not used in the source code yet.

July 4

• Only small bug fixes in the eclass g-octave.eclass.

July 5

• Started moving the g-octave stuff from a VPS to the Gentoo infrastructure (Moved the source control to Git and the octave-forge's source tarballs to http://soc.dev.gentoo.org/~rafaelmartins). More details about this in a next post.

That's all for now.

If you want to find out which capabilities a binary needs, you have several options:

1. You use strace and determine them from the output
2. You use capable_probe

The first method is quite simple, but you have to deduct the caps from the error messages, which can be a bit difficult.
The second method is more convenient and simple, so I’m going to describe this method further.
capable_probe is a loadable kernel module developed by Serge Hallyn [0]. It uses the kprobe kernel mechanism [1] to insert a Jprobe.  And what does it do exactly?

When this kernel module is inserted, any calls to cap_capable() are replaced by a call to the cr_capable() function. This function prints the name of the program that requires capabilities and the capability being checked. It then continues executing the actual cap_capable() call through the call to jprobe_return(). [0]

The signature of cr_capable has to be the same as the signature of cap_capable. The signature changed from
int cap_capable (struct task_struct *tsk, int cap)
to
int cr_capable (struct task_struct *tsk, const struct cred *cred, int cap, int audit).
So to get capable_probe to work you need to change the signature or use the one from my repository [2].

There is a good README by Chris Friedhoff [3], which describes how to install and use it.

Important: Don’t forget to unload capable_probe after you determined the caps, as it will spam your /var/log/messages extremely.

To see capable_probe in action here is how to determine the caps for passwd:

• Remove suid
  chown -s /bin/passwd
• tail -f /var/log/messages | grep passwd
• modprobe capable_probe
• Try to change your password
• This is what you get from passwd:
  superkfr@totoro ~ $ passwd
  Changing password for superkfr.
  (current) UNIX password:
  Password:
  Retype new password:
  passwd: Authentication token lock busy
  passwd: password unchanged
  Not very informative.
• modprobe -r capable_probe
• Let’s see what we got from capable_probe:
  Jul 5 15:20:54 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 15:20:54 totoro kernel: cr_capable: asking for capability 2 for passwd
  Jul 5 15:20:54 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 15:20:54 totoro kernel: cr_capable: asking for capability 2 for passwd
  Jul 5 15:20:54 totoro kernel: cr_capable: asking for capability 24 for passwd
  
• What does this all mean?
  The numbers stand for different caps, you can look them up in /usr/include/linux/capability.h.

  /* Override all DAC access, including ACL execute access if
  [_POSIX_ACL] is defined. Excluding DAC access covered by
  CAP_LINUX_IMMUTABLE. */

  #define CAP_DAC_OVERRIDE 1

  /* Overrides all DAC restrictions regarding read and search on files
  and directories, including ACL restrictions if [_POSIX_ACL] is
  defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE. */

  #define CAP_DAC_READ_SEARCH 2
  ...

  /* Override resource limits. Set resource limits. */
  /* Override quota limits. */
  /* Override reserved space on ext2 filesystem */
  /* Modify data journaling mode on ext3 filesystem (uses journaling
  resources) */
  /* NOTE: ext2 honors fsuid when checking for resource overrides, so
  you can override using fsuid too */
  /* Override size restrictions on IPC message queues */
  /* Allow more than 64hz interrupts from the real-time clock */
  /* Override max number of consoles on console allocation */
  /* Override max number of keymaps */

  #define CAP_SYS_RESOURCE 24

• Now, which caps seem plausible? passwd needs to read and write /etc/shadow so it needs CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH is a subset, is does not have to be set additionally. There is no indication, that passwd could need CAP_SYS_RESOURCE. So we try it out with CAP_DAC_OVERRIDE.
• sudo setcap cap_dac_override=ep /bin/passwd
  sudo modprobe capable_probe
  passwd
  And it fails again.
• Jul 5 16:02:50 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:02:50 totoro kernel: cr_capable: asking for capability 2 for passwd
  Jul 5 16:02:50 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:02:50 totoro kernel: cr_capable: asking for capability 24 for passwd
  Jul 5 16:03:01 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:03:01 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:03:01 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:03:01 totoro kernel: cr_capable: asking for capability 0 for passwd
  Jul 5 16:03:01 totoro kernel: cr_capable: asking for capability 1 for passwd
  It also asks for CAP_CHOWN(0) now, that seems also plausible, so grant it and repeat.
• It fails again, what is still missing?
  Jul 5 16:03:56 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:03:56 totoro kernel: cr_capable: asking for capability 2 for passwd
  Jul 5 16:03:56 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:03:56 totoro kernel: cr_capable: asking for capability 24 for passwd
  Jul 5 16:04:06 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:04:06 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:04:06 totoro kernel: cr_capable: asking for capability 1 for passwd
  Jul 5 16:04:06 totoro kernel: cr_capable: asking for capability 0 for passwd
  Jul 5 16:04:06 totoro kernel: cr_capable: asking for capability 0 for passwd
  Jul 5 16:04:06 totoro kernel: cr_capable: asking for capability 3 for passwd
  Jul 5 16:04:06 totoro kernel: cr_capable: asking for capability 1 for passwd
  It also wants CAP_FOWNER(3)
• Add that to the set and it works.
• Don’t forget:
  modprobe -r capable_probe

This example is a little more complicated than the standard ping-example, but I wanted to do a different one. Hope that helps :).

[0] http://www.ibm.com/developerworks/library/l-posixcap.html
[1] You need CONFIG_KPROBES=y in your kernel
[2] http://github.com/constanze/GSoC2010_Gentoo_Capabilities/tree/master/capable_probe/
[3] http://www.friedhoff.org/posixfilecaps.html

A couple months ago, Facebook announced announced that their chat system would also be available through XMPP. So it is possible to just configure it like any other Jabber server on a N900. And then to merge Facebook contacts with other contacts and it the address book will store it as a field of type « X-JABBER ».  Since the PR1.2 version of Maemo5, Nokia has added a specific profile for Facebook to the N900 software. But instead of calling the address book field « X-JABBER », they created a « X-FACEBOOK » field (since Facebook Chat is not XMPP in the backend, it can not talk to other Jabber servers). So if you remove your old Jabber account and re-add it as a Facebook account and use Marco’s contact merger to merge back the contacts. You’re still stuck with a bunch of extra « X-JABBER » fields in your contacts. So to clean them up, I wrote a really simple program to remove them. It is based on Marco’s jid-to-email program. You can get the source code or a precompiled binary. You can just put the binary on your N900 and run it from the Terminal as « ~/MyDocs/remove-old-facebook » (if you put it in the N900’s documents root). Be warned that is is not a good example like Marco’s code, I stripped all error handling and it never frees anything. If you want a good example, look at Marco’s code.

I have to thank Arfrerver for making me notice this with the bug about Ruby 1.9 he reported.

The GNU project released autoconf 2.66 two days ago. Very few notable changes are present in it, just like a few were listed before, so I didn’t go out of my way to test it beforehand. My bad! Indeed there is one big nasty change with it for which I’d say to all of you to put off the update until I write it so. Hopefully it won’t get unmasked in Gentoo for a while either.

There are two main problems with this release; the first is due to the implementation of a stricter macro to ensure the parameters given to it is not variable over executions:

• The macro AS_LITERAL_IF is slightly more conservative; text containing shell quotes are no longer treated as literals. Furthermore, a new macro, AS_LITERAL_WORD_IF, adds an additional level of checking that no whitespace occurs in literals.
  

well, whatever the idea about this was, it seems to have broken the AC_CHECK_SIZEOF macro: if you pass it [void*] as parameter, it’ll report it not being a literal (while it is) causing the following error:

flame@yamato test % cat configure.ac AC_INIT([foo], [0]) AC_CHECK_SIZEOF([void*]) AC_OUTPUT flame@yamato test % autoconf configure.ac:3: error: AC_CHECK_SIZEOF: requires literal arguments ../../lib/autoconf/types.m4:765: AC_CHECK_SIZEOF is expanded from... configure.ac:3: the top level autom4te-2.66: /usr/bin/m4 failed with exit status: 1

This would be bad enough. But the nastier surprise I got when running autoreconf over the feng sources, the build system of which I wrote myself, and if I may say so, is very well engineered:

flame@yamato feng % autoreconf -fis configure:6275: error: possibly undefined macro: AS_MESSAGE_LOG_FDdnl If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. autoreconf-2.66: /usr/bin/autoconf-2.66 failed with exit status: 1

The problem here is almost obvious, and it’s related to the dnl entry at end of the macro name; the dnl keyword is used as (advanced) comment delimiter in autoconf scripts, meaning “Discard up to New Line” and is often used to keep on multiple lines commands that should be kept togever, like \ is in many languages. A quick check at the configure files brings in this:

as_fn_error $? "Package requirements (glib-2.0 >= 2.16 gthread-2.0) were not met: $GLIB_PKG_ERRORS Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables GLIB_CFLAGS and GLIB_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details." "$LINENO" AS_MESSAGE_LOG_FDdnl

You can easily see that the problem here is with the pkg-config macros (pkg.m4). Funnily enough there is no change related to the errors reporting that is listed in the autoconf news file so I wasn’t expecting this. The problem is further down the path of pkg-config files but it’s not important to fully debug it right now, it’s actually quite easy to fix, in pkg-config itself, but here’s the catch.

Since the pkg.m4 macro file is way too often bundled with the upstream packaging, and its presence overrides the copy from the system, even fixing pkg-config will not fix all the software that carries outdated copies of the macro file.

This is almost the same problem with libtool 1 vs libtool 2 macro files with the difference that this is going to be much much more common. If you’re a package maintainer, you can do something already before this even hits the users: remove the pkg.m4 file during the src_prepare() phase; you’re already depending on pkg-config in the ebuild for it to work at build-time, and since we don’t split the macro file from the command itself, you can simply rely on its presence on the system.

In the mean time, I’m not sure if I want to start testing with it just yet or if we should be waiting for 2.67…

In spite of my exams-period beginning next week, I managed to do something useful for GSoC :).

What did I do this week:
In order to do some better testing I wanted to patch some ebuilds, as I said last week. The question there is, how do you find out which caps the file needs to function? I consulted Friedhoffs page [0] again and downloaded capable_probe. This neat loadable kernel module detects when a program asks for caps and prints it in /var/log/messages. All you need is CONFIG_KPROBES=y in your kernel.
I tried it out and unfortunately it didn’t work. I remembered vaguely, that the caps-format had been changed recently, but I was busy with learning so I didn’t bother to look for the problem (but I came back later :)).
Friedhoff has some example caps-sets on his page and there are also some on [1], so I used these and patched shadow, pam and util-linux.
The patched binaries are:

• passwd - CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FOWNER
• chsh - CAP_CHOWN, CAP_DAC_READ_SEARCH, CAP_FSETID, CAP_SETUID
• chfn - CAP_CHOWN, CAP_DAC_READ_SEARCH, CAP_FSETID, CAP_SETUID
• chage - CAP_DAC_READ_SEARCH
• expiry - CAP_DAC_OVERRIDE, CAP_SETGID
• gpasswd - CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_SETUID
• newgrp - CAP_DAC_OVERRIDE, CAP_SETGID
• su - CAP_SETGID, CAP_SETUID (this only seems to work, when unix_chkpwd has its cap)
• mount - CAP_DAC_OVERRIDE, CAP_SYS_ADMIN
• umount - CAP_DAC_OVERRIDE, CAP_SYS_ADMIN, (CAP_CHOWN)
• unix_chkpwd - CAP_DAC_OVERRIDE

After testing them, all worked except umount. So to investigate what was missing I turned back to capable_probe. After a bit of digging through kernel docs and sources, the problem was quite clear: The signature of cap_capable had changed (the method which was used to place the Jprobe-hook on). After adapting the signature it worked!

Using capable_probe I detected that umount was asking for CAP_CHOWN, which was not in its set. Because umount failed when trying to move mtab.tmp back to mtab, it seemed reasonable, that CAP_CHOWN could help. And it did :).

All in all I’m very happy today and will write a separate post on capable_probe. I added the patched version to my git-repo [2].

I would be even happier :D if someone would try my patched ebuilds and tell me if all is okay or if there is something missing.

[0] http://www.friedhoff.org/posixfilecaps.html
[1] http://wiki.archlinux.org/index.php/Using_File_Capabilities_Instead_Of_Setuid
[2] http://github.com/constanze/GSoC2010_Gentoo_Capabilities/tree/master/capable_probe/