• Setting up my old Mac Mini G4 as a development machine

    I've spent quite a few time last week setting up my old Mac Mini G4 — a PPC 1.2GHz with 1GB of RAM running NetBSD/macppc current — as a "workstation" for the development of Kyua and other tools for NetBSD. Yes, this machine is very slow, but that's the whole point of the exercise I'm going to narrate below.I recently got approval from the NetBSD core team to import Kyua into the NetBSD source tree and replace ATF with it... which is great, but when thinking about it objectively, I am reluctant to unnecessarily "punish" all NetBSD users by increasing the build time of the system significantly. Don't get me wrong: Kyua itself runs efficiently on old machines, but building the code — particularly the hundreds of tests that come with it — takes just too long. This slowdown is not too noticeable on a relatively-modern machine, but it's unacceptable on not-so-old machines.Of course, I could proceed with the import right now (with the code disabled by default) and make it leaner later, but this would cause quite a first bad impression on our users. So it's better to delay the import a little bit until, at least, I have had a chance to simplify some core parts of the code. Mind you, this simplification work is already in progress and quite advanced: it consists on modularizing (as separate processes) some critical parts of the code and writing these with a much simpler style and in plain C.But back to the original point of my post.The first thing to mention about this experience is that with some effort and long waits, I've got a pretty decent setup for development even on this old machine. From time to time, I miss having a real Unix desktop at hand for development (no OS X, you are not one of those). The GUI behaves relatively well with a 1920x1200 display, running Fluxbox, traditional xterms, Mutt for GMail access and a bunch of other applications.Unfortunately, too many things feel really sluggish. A few specific examples:Firefox 16 is barely usable. I'm not sure there are many alternatives to decent web browsing for such an old non-Intel platform. Dillo is blazing fast and allows me to access online documentation and mailing list archives (just enough for development), but it's pretty much useless for any other purpose given the "Web 2.0".Any operation that involves pkgsrc takes ages. Even when building the simplest packages, one can notice the system crawl through all of the pkgsrc infrastructure. Sometimes this is the fault of a bad algorithm; other times it's just sh(1) being the wrong tool for something as complex as pkgsrc internals.Things like basic code editing in Emacs 24 are slow at responding to typing. Disabling font lock mode makes it feel fast again, but it's just surprising to see that even color coding is slow.I still remember my old and trusty machine from 10 years ago (a Pentium II 233 MHz): with a similar setup, it was significantly snappier. Yes, software has evolved and these packages now have many more features... but really, does editing a text file have to be sluggish?Leaving aside sluggishness, there is also the problem of instability. NetBSD/macppc is a tier 2 port, and things don't work as well as one would like. I personally would enjoy bringing this port to tier 1... but I currently lack the time (and basic knowledge of the architecture) to do so :-/Anyway, the result of this exercise: the new code I'm writing to modularize Kyua builds damn fast in this machine, and keeping it this way is the whole point of having such an old machine as a build environment. So I'll try to keep using it to develop these new components of Kyua. [Continue reading]

  • I don't really like C++

    Somebody recently tweeted me this message:As a strong C++ dev and googler (hopefully with some #golang exposure), what's your opinion on @rob_pike post? (goo.gl/xlMi4)The answer deserves much more than my original reply included, so here it goes.First of all, I found Rob's article quite interesting. Basically, the authors of Go never expected Go to be more widely adopted by Python users than C++ users. In fact, their original goal was to create a replacement for C++ as a systems programming language. The rationale for this is that C++ users like the verbosity and flexibility of the language, with all of its special cases, while Python users like simplicity and switch to Go when they look for the performance bump. This is all reasonable but there is one detail I don't associate with: Rob claims that whoever is excited by the new C++11 features will not move to Go, because liking new C++ features implies that one likes all the flexibility of C++ and will not enjoy Go's simplicity.Before I explain why, let's look at my personal scenario: I use C++ for personal projects and Python at work. I enjoy using C++ because of its expressivity and the cool hacks that can be implemented (although that generally goes against readability), and I really like Python due to its simplicity and its extensive standard library. But I dislike C++'s corner cases, lack of real modules, verbosity, huge binaries, slow compilation times... and I dislike Python's non-static typing.So why do I use C++?  Most of my personal projects, if not all, are in some way related to NetBSD. NetBSD has support in the base system for C, C++, POSIX shell and, recently, Lua. If I want my projects to be bundled into the NetBSD base system, I need to use one of these languages; that's clear. Of these languages, shell is not scalable for large projects and I don't know Lua enough to implement large pieces of code (but from what I have learned, Lua has its own problems). Writing C is a waste of time due to its lack of support for basic high level data structures (things like, you know, strings and dictionaries) — and these are a must for any medium-large project. And C++, despite all of the problems mentioned above, is a good compromise.In other words: the reason I chose C++ for things like ATF and Kyua is only because it simplified my work and it was supported by the NetBSD base system. It's relatively easy to write safe code (the RAII pattern is lovely) in C++ and there is an abundant amount of rich data structures in the base library. (This doesn't mean I will stick to C++ forever in Kyua; in fact, I have some ideas in mind to reduce its usage!)And finally, this is where I can counter Rob's statement: I am excited about the new C++11 features but not because I really like C++. I will enjoy the new features because they simplify a language that is the only real alternative I have in this context.Confession: In fact, I don't really like C++. I wish there was a better alternative within the NetBSD base system for large projects, but there is none. [Continue reading]

  • Testing NetBSD's share/mk

    For a long time, a pet peeve of mine has been the lack of tests for the build infrastructure files of NetBSD: i.e. those bsd.*.mk files that live under /usr/share/mk/ and on which the whole source tree depends.One could argue that writing tests for these files is not strictly necessary because the successful build of NetBSD is the real final test of whether the files work or not. That's partly true, but unfortunately is not the whole story:Other projects depend on these build files and thus rely on their behavior. This can either be by just relying on NetBSD having these files installed, or by using the separate mk-files package (available e.g. in Fedora). The functionality of the build infrastructure should not regress, or it would break these third-party projects. (Think about these files as having a public API outside of NetBSD.)Even if the build of NetBSD is the real test of the functionality of the build infrastructure, the build infrastructure supports dozens of options and tweaks to change its behavior. One would need to rebuild NetBSD tens, if not hundreds of times, each with a different combination of build options, to ensure the files work.Lastly, when a new functionality is added to the build infrastructure, it is because some component of the source tree needs such functionality. It may latter happen that this component disappears or stops needing such functionality. The functionality becomes unused by the source tree, and thus can regress unexpectedly (breaking third-party packages or introducing bugs, as mentioned earlier).With this in mind, it's clear that we should have some tests for the share/mk files. Unfortunately, that's easier said than done: the files in share/mk are extremely complex and expose hundreds, if not thousands, of different behaviors each with its own subtleties. Adding tests for these is hard. The fact that this code that was never designed to be unit-tested doesn't help either.Regardless, I have just submitted some "placeholder" tests to the tree. The major point of these new test programs is to lower the barrier of entry to writing tests for share/mk and, therefore, maybe get other people to write some tests. To predicate with the example, I have populated these skeleton test programs with a couple of test cases each, although as you will see they are very trivial tests.And, to conclude, why have I done this now? Well: I'm working on integrating Kyua into the source tree and, to make this happen, I need to do a couple of changes to the build infrastructure files. The changes are tricky, so I want to write tests to have some assurance that my modifications work and that, specially, they do not regress over time. Having the placeholder test programs in place makes this much easier, and the real functionality changes easier to review. [Continue reading]

  • Introducing shtk

    Have you ever wanted to have a collection of ready-to-use modules for shell scripts? I have, particularly because I keep reimplementing the same functions over and over and over and over again whenever I write non-trivial shell scripts, and I'm tired of doing so.That's why I have just abstracted all the common code in the aforementioned tools and put it into a new package called the "Shell Toolkit", or shtk for short. Yeah, this name sounds very pretentious but, really, I don't intend this to be anything big. The only thing I want to do is simplify my life when implementing shell scripts, and hope that other people might find the modules useful. So far, I have taken the generic (and common!) code from sysbuild and sysupgrade, reconciled a few tiny divergences, and moved it into this new shtk package.In reality, writing something like shtk is sin-borderline. I really should not be using shell scripting for the kind of tools I am implementing (they deserve better data structures and better error checking than what shell provides, for example). However, shell scripting is incredible convenient to get reasonably-good implementations of such tools with minimal effort, and is the only scripting language available in NetBSD's base system. (Yes, yes, there is Lua, but my limited knowledge of Lua would not let me write these tools in any decent manner nor in any reasonable time.)So, what's in shtk? There are a few functions to deal with command lines (error/warning reporting and such things), some trivial stuff to deal with lists, a bunch of code to interact with cvs and, what I like the most, a module to implement configuration files with some kind of key/value validation.At the moment, shtk can only be found in pkgsrc under pkgsrc/devel/shtk and I don't currently have any plans to make it more widely available. If there are enough people interested in that with real needs, I could reconsider, but the maintenance effort would be non-trivial.To showcase the features of shtk, I have updated the sysbuild and sysupgrade packages to depend on this new toolkit while at the same time dropping all this duplicate supporting code. It's a good thing that I wrote exhaustive tests for all possible code paths, because the migration from the built-in modules to shtk was riddled with subtleties that would have impacted end users otherwise.Now... time to really consider taking the task of rewriting pkg_comp in a more maintainable style so that I can add the features I have wished for for many years (like OS X support). [Continue reading]

  • Introducing sysupgrade for NetBSD

    Over the last two weeks, you might have had fun rolling your own NetBSD binary releases with sysbuild. But what fun is that if you have no trivial way of upgrading your existing NetBSD installation to a newer version?Upgrading NetBSD to a newer version from distribution sets generally looks like the following;Fetch new distribution sets (or roll your own).Upgrade the kernel.Unpack the distribution sets over the root directory, without fat-fingering the command and unpacking etc.tgz along the way.Use etcupdate to merge new changes to configuration files.Use postinstall to validate the upgraded system.Simple? Yes. Easy? No. The above procedure is obscure to anyone new to NetBSD. (Actually, if you tell anybody that the way to upgrade your machine is by unpacking tarballs over / will stare at you thinking you are kidding. It's 2012.) "Jokes" aside, what is worse is that the procedure is quite monotonous and, therefore, it is very easy for the administrator to make a trivial mistake along the way and screw up a running system. (Been there, done that... multiple times.)Machines are made to automate trivial and repetitive tasks like the above, and they are actually very good at that. Over the years, I have performed NetBSD updates manually and later written crappy, unreliable scripts to do the upgrades for me. These scripts have never been reusable and they haven't dealt with error conditions gracefully. Furthermore, because these scripts live in my home directory, I have to remember to carry them around every time I set up a new NetBSD box.It was about time I sat down and rewrote my custom scripts into something more "decent". Something with documentation, with a configuration file, and with tests.So today, and for all the reason above, I am introducing sysupgrade.sysupgrade is a script that automates (trivializes) the whole process of upgrading an existing NetBSD installation to a newer release, be it the currently-running system or a non-live system. sysupgrade does so by following the steps outlined above and is coordinated by a configuration file. You can find the tool in pkgsrc/sysutils/sysupgrade, next to sysbuild. The bundled sysupgrade(8) and sysupgrade.conf(5) manual pages, and the default sysupgrade.conf configuration file should get you started and hopefully answer most of your questions.For the impatient, the following command would upgrade your machine to the specified version target:$ sysupgrade auto     ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-<X.Y.Z>/$(uname -m)At the moment, please consider sysupgrade to be experimental. It works well for me on my various machines (running both NetBSD 6.0 BETA and -current), and I have been using the upgrade procedure outlined above for years without issues. However, as with any shiny-new software, be careful. If you use NetBSD on a virtual machine, take a snapshot before running this tool; I don't think your machine is going to blow up, but better safe than sorry!Enjoy, and feedback very welcome!PS: There is lots of room for improvement. The TODO file in the package directory includes some of the ideas I'd like to work on later. [Continue reading]

  • Introducing sysbuild for NetBSD

    NetBSD's build system is close to awesome: after checking a source tree out from CVS on virtually any Unix-like operating sytem, building a full NetBSD release for any of the supported platforms is as simple as running the build.sh script with the right arguments.There are, however, a few things that would deserve automation in this process, but that are not in build.sh's domain to solve. These are:Fetching and keeping the source tree up to date: interacting with CVS is still the responsibility of the user.Simplifying the call to build.sh: this script takes a ton of arguments, and running it by hand quickly becomes "annoying". It would be nice if all the desired values to its arguments where stored elsewhere and picked up automatically. (mk.conf may not always be the right choice.)Plugging into your crontab(5) for easy periodic rebuilds of NetBSD, with proper reporting of failures. Upgrading a live system from source is a major supported mechanism (particularly for NetBSD-current), so having a way to keep fresh release files around is welcome.Performing such daily rebuilds of NetBSD as a dedicated unprivileged user.To anybody familiar with NetBSD, it is obvious that all the above items are "simple enough" to solve one by one; however, doing them by hand gets old very quickly — machines exist to make our life easier, don't they? I know I am not the only one that has custom local wrapper scripts around build.sh, and I also know that my scripts were ad-hoc, ugly and non-reusable. (Also, of course, anybody new to NetBSD will certainly not find any of the above trivial, but that's secondary.)To this end, I have written a script that automates all the aforementioned steps in a single tool, driven by configuration files that specify what to do. Enter sysbuild. My setupThese days, my main NetBSD development box is a virtual machine. For a couple of years, I have been carrying around this virtual machine across machines and slowly tuning its configuration to suit my needs. The way I work is the following:I keep a read-write copy of the sources under my home directory, which is the one I use for my development work. This copy is accompanied by a script in my home directory that, given a machine type, updates the sources and rebuilds NetBSD for that machine type, using a known directory layout within my home directory.I also keep a read-only copy of the sources under /usr/src, checked out from anoncvs. This copy is used by the dedicated "builder" system user to perform dedicated rebuilds of NetBSD. The "builder" user also has a custom script (but different than the other one!) that updates /usr/src, rebuilds NetBSD for the machine types I am interested in, records full logs to files and places the build results into a shared network drive. Lastly, the dedicated "builder" user has a cron job that runs the previous script overnight.The purpose of this dual setup is to always have a fresh release build somewhere in the system, built from pristine sources, while at the same time allowing me to do my development work in a tree that I could break at any time.Unfortunately, the size of the virtual disk of my development box has become too small for my current needs, and in order to fix this I prefer to start afresh. The thought of having to set up this whole scheme all over again, by hand, is what triggered the creation of sysbuild. Getting startedsysbuild lives in pkgsrc under the pkgsrc/sysutils/sysbuild directory. This package provides the main script, tests, sample configuration files and the extensive manual page. Installing the script is as easy as installing any other package, and I would recommend reading its documentation now.Once installed, sysbuild should be ready to use out of the box assuming that you have write access to /usr/src. Simply typing sysbuild build from anywhere in the system will cause the source tree to be updated and a release for your current platform to be built.If you wish to customize the behavior of sysbuild, copy /usr/pkg/etc/sysbuild/default.conf to ~/.sysbuild/default.conf and edit the latter. The tool will pick it up on the next execution and use your new configuration settings. You can, of course, manage a variety of configurations in case you want to do different builds (say -current and 6.x). Setting up a daily cron jobThe pkgsrc/sysutils/sysbuild-user package is a convenience bundle that manages a "sysbuild" unprivileged user in the system and installs a sample crontab to perform the desired periodic rebuilds of NetBSD. This crontab uses the sysbuild4cron script, which is a very simple utility to run sysbuild, store its output in a file, and send an error report any time the execution fails.Simply installing the package will cause the creation of the "sysbuild" user and the configuration of this sample cron job. Follow the instructions provided by the package in its MESSAGE file to tune the behavior of any of these. Concluding triviaYes, these scripts are extremely tied to my particular workflow and use cases, although I don't think my needs are that special. However, I have attempted to come up with a generic-enough script and configuration file to allow the addition of new features.You may notice that sysbuild's version is 2.0 and not 1.0. While preparing the script for addition to pkgsrc, cvs add barfed that I was re-adding previously-deleted files. Uh hum, what a surprise! Upon reading the CVS log of the package's Makefile, I found that 10 years ago I already wrote this exact same script and that two years after it got deleted because it stopped working. I had completely forgotten about this! However, seeing pkg_comp's messy code (which was written around the same time), I imagine that the previous implementation of this idea was messy; I don't dare to look at the previous code.Enjoy! [Continue reading]

  • Getting rid of @public.talk.google.com GTalk contacts

    If you use a native IM client to connect to Google Talk and also have a Google+ account, you probably have noticed by now that your contacts list is polluted by tons of addresses of the form annoyinghash@public.talk.google.com. Attempting to talk to these people from the native IM client does not work (maybe it does in some specific situations? I don't know.), so these contacts only add noise and annoyance.It turns out that these "spurious" contacts are the people that you have in your Google+ circles, and I suppose allows such people to talk to you when you are logged into Google+. Me, I don't necessarily want to allow all of these people in my chat list.After a bit of searching around, I have found that it is possible to remove them completely from your Google Talk list. Because all the answers I could find were incomplete and/or vague replies to forum questions, I'm writing the procedure down here for posterity. Do as follows:Log into your Google+ account.Sign into the builtin chat client on the right-hand side panel.Open the drop-down menu next to your name within the chat client and select Privacy settings.In the Choose who can chat with you option, select Custom.Unselect all the circle names in the checkboxes that appear.Click save.Witness all those useless contacts vanish from your IM clients! (Both native and web based.) [Continue reading]

  • Kyua gets its own blog

    For the last couple of weeks, I have been pondering the creation of a Kyua-specific blog. And, after a lot of consideration, I have finally taken the plunge. Say hello to Engineering Kyua!From now on, all Kyua-related posts (as well as ATF posts) will go to the new blog. I recommend you to subscribe to Engineering Kyua's Atom feed right now to not miss a beat!  If you care enough about Kyua, that is...I may still post Kyua-related stuff in here once in a while, but you should assume that all news and, in particular, weekly status reports will be sent to the new blog."Why?" Well, The Julipedia is supposed to be (and always has) my personal blog. Looking back at all the recent posts, they almost univocally are about Kyua and there is no personal content in them. In respect for the readers of this blog (who may not care about Kyua at all) and in order to attempt to give Kyua a more definite identity, it makes sense to move the posts to their own blog.Also, by having a blog dedicated to Kyua, I will not feel uncomfortable about publishing weekly status reports again. I previously felt that they were adding too much noise to this blog, and is the main reason behind why I stopped posting them at some point. Weekly reports have their value, mostly to keep myself focused and to allow outsiders to know what the project is up to (particularly in a world of DVCSs, where code changes may be kept private for weeks at a time).And you may wonder: "will you continue to post content here?" Sure I will, but I need ideas (suggestions welcome)! Today's social ecosystem makes it difficult for me to decide whether a post belongs in a blog, in Google+, in Twitter... and updating them all at once to provide the same content is pointless.Here is my take: for most of the irrelevant stuff that one may want to share at a personal level (photos, videos, arbitrary thoughts), social networks seem to provide a better platform. The blog seems a place more suited for short essays that should be indexable and be accessible by users across the web; for example, these include how-tos, technical explanations for a particular concept, or opinion articles. And, finally, Twitter seems like the place to throw pointers to longer articles elsewhere and very short opinion comments. I think this summarizes pretty well what my current "practices" around these systems follow. And, as you can deduce, this also explains (as you have experienced) why the blog gets fewer content than ever because most things are better suited for a social network. [Continue reading]

  • Exposing a configuration tree through Lua

    In the previous post, I discussed the type-safe tree data structure that is now in the Kyua codebase, aimed at representing the configuration of the program. In this post, we'll see how this data structure ties to the parsing of the configuration file.One goal in the design of the configuration file was to make its contents a simple key/value association (i.e. assigning values to predetermined configuration variables). Of course, the fact that the configuration file is just a Lua script means that additional constructions (conditionals, functions, etc.) can be used to compute these values before assignment, but in the end all we want to have is a collection of values for known keys. The tree data structure does exactly the latter: maintain the mapping of keys to values, and ensuring that only a set of "valid" keys can be set. But, as a data structure, it does not contain any of the "logic" involved in computing those values: that is the job of the script.Now, consider that we have the possible following syntaxes in the configuration file:simple_variable = "the value"complex.nested.variable = "some other value"These assignments map, exactly, to a tree::set() function call: the name of the key is passed as the first argument to tree::set() and the value is passed as the second argument. (Let's omit types for simplicity.) What we want to do is modify the Lua environment so that these assignments are possible, and that when such assignments happen, the internal tree object gets updated with the new values.In order to achieve this, the configuration library modifies the Lua environment as follows:The newindex metatable method of _G is overridden so that an assignment causes a direct call to the set method of the referenced key. The key name is readily available in the newindex arguments, so no further magic is needed. This handles the case of "a = b" (top-level variables).The index metatable method of _G is overridden so that, if the indexed element is not found, a new table is generated and injected into _G. This new table has a metatable of its own that performs the same operations as the newindex and index herein described. This handles the case of "a.b = c", as this trick causes the intermediate tables (in this case "a") to be transparently created.Each of the tables created by index has a "key" metatable field that contains the fully qualified key of the node the table corresponds to. This is necessary to be able to construct the full key to pass to the set method.There is further magic to ensure that values pre-populated in the tree (aka default values) can be queried from within Lua, and that variables can be set more than once. These details are uninteresting though.At the moment, we deny setting variables that have not been pre-defined in the tree structure, which means that if the user wants to define auxiliary variables or functions, these must be declared local to prevent calling into the _G hooks. This is quite nice, but we may need to change this later on if we want to export the standard Lua modules to the configuration files. [Continue reading]

  • Type-safe, dynamic tree data type

    The core component of the new configuration library in Kyua is the utils::config::tree class: a type-safe, dynamic tree data type. This class provides a mapping of string keys to arbitrary types: all the nodes of the tree have a textual name, and they can either be inner nodes (no value attached to them), or leaf nodes (an arbitrary type attached as a value to them). The keys represent traversals through such tree, and do this by separating the node names with dots (things of the form root.inner1.innerN.leaf).The tree class is the in-memory representation of a configuration file, and is the data structure passed around methods and algorithms to tune their behavior. It replaces the previous config static structure.The following highlights describe the tree class:Keys are (and thus the tree layout is) pre-registered. One side-effect of moving away from a static C++ structure as the representation of the configuration to a dynamic structure such as a tree is that the compiler cannot longer validate the name of the configuration settings when they are queried. In the past, doing something like config.architecture would only compile if architecture was a valid structure defined... but now, code like config["architecture"] cannot be validated during the build.In order to overcome this limitation, trees must have their keys pre-defined. Pre-defining the keys declares their type within the tree.  Accesses to unknown keys results in an error right away, and accesses to pre-defined keys must always happen with their pre-recorded types.Note that pre-defined nodes can, or cannot, hold a value. The concept of "being set" is different than "being defined".Some nodes can be dynamic. Sometimes we do not know what particular keys are valid within a context. For example, the test_suites subtree of the configuration can contain arbitrary test suite names and properties within it, and there is no way for Kyua (at the moment) to know what keys are valid or not.As a result, the tree class allows defining a particular node as "dynamic", at which point accesses to any undefined keys below that node result in the creation of the node.Type safety. Every node has a type attached to it. The base configuration library provides common types such as bool_node, int_node and string_node, but the consumer can define its own node types to hold any other kind of data type. (It'd be possible, for example, to define a map_node to hold a full map as a tree leaf.)The "tricky" (and cool) part of type safety in this context is to avoid exposing type casts to the caller: the caller always knows what type corresponds to every key (because, remember, the caller had to predefine them!), so it knows what type to expect from every node. The tree class achieves this by using template methods, which just query the generic internal nodes and cast them out (after validation) to the requested type.Plain string representations. The end user has to be able to provide overrides to configuration properties through the command line... and the command line is untyped: everything is a string. The tree library, therefore, needs a mechanism to internalize strings (after validation) and convert them to the particular node types. Similarly, it is interesting to have a way to export the contents of a tree to strings so that they can be shown to the user.With that said, let's see a couple of examples. First, a simple one. Let's create a tree with a couple of fictitious nodes (one a string, one an integer), set some values and then query such values:config::tree tree;// Predefine the valid keys.tree.define< config::string_node >("kyua.architecture");tree.define< config::int_node >("kyua.timeout");// Populate the tree with some sample values.tree.set< config::string_node >("kyua.architecture", "powerpc");tree.set< config::int_node >("kyua.timeout", 300);// Query the sample values.const std::string architecture = tree.lookup< config::string_node >("kyua.architecture");const int timeout = tree.lookup< config::int_node >("kyua.timeout");Yep, that's it. Note how the code just knows about keys and their types, but does not have to mess around with type casts nor tree nodes. And, if there is any typo in the property names or if there is a type mismatch between the property and its requested node type, the code will fail early. This, coupled with extensive unit tests, ensures that configuration keys are always queried consistently.Note that we'd also have set the keys above as follows:tree.set_string("kyua.architecture", "powerpc");tree.set_string("kyua.timeout", "300");... which would result in the validation of "300" as a proper integer, conversion of it to a native integer, and storing the resulting number as the integer node it corresponds to. This is useful, again, when reading configuration overrides from the command line as types are not known in that context yet we want to store their values in the same data structure as the values read from the configuration file.Let's now see another very simple example showcasing dynamic nodes (which is a real-life example from the current Kyua configuration file):config::tree tree;// Predefine a subtree as dynamic.tree.define_dynamic("test_suites");// Populate the subtree with fictitious values.tree.set< config::string_node >("test_suites.NetBSD.ffs", "ext2fs");tree.set< config::int_node >("test_suites.NetBSD.iterations", 5);// And the querying would happen exactly as above with lookup().Indeed, it'd be very cool if this tree type followed more standard STL conventions (iterators, for example). But I didn't really think about this when I started writing this class and, to be honest, I don't need this functionality.Now, if you paid close attention to the above, you can start smelling the relation of this structure to the syntax of configuration files. I'll tell you how this ties together with Lua in a later post. (Which may also explain why I chose this particular representation.) [Continue reading]

  • Rethinking Kyua's configuration system

    In the previous blog post, I described the problems that the implementation of the Kyua configuration file parsing and in-memory representation posed. I also hinted that some new code was coming and, after weeks of work, I'm happy to say that it has just landed in the tree!I really want to get to explaining the nitty-gritty details of the implementation, but I'll keep these for later. Let's focus first on what the goals for the new configuration module were, as these drove a lot of the implementation details:Key/value pairs representation: The previous configuration system did this already, and it is a pretty good form for a configuration file because it is a simple, understandable and widespread format. Note that I have not said anything yet about the types of the values.Tree-like representation: The previous configuration schema grouped test-suite specific properties under a "test_suites" map while it left internal run-time properties in the global namespace. The former is perfect and the latter was done just for simplicity. I want to move towards a tree of properties to give context to each of them so that they can be grouped semantically (e.g. kyua.report.*, kyua.runtime.*, etc.). The new code has not changed the structure of the properties yet (to remain compatible with previous files), but it adds very simple support to change this in the shortcoming future.Single-place parsing and validation: A configuration file is an external representation of a set of properties. This data is read (parsed) once and converted into an in-memory representation. All validation of the values of the properties must happen at this stage, and not when the properties are queried. The reason is that validation of external values must be consistent and has to happen in a controlled location (so that errors can all be reported at the same time).I have seen code in other projects where the configuration file is stored in memory as a set of key/value string pairs and parsing to other types (such as integers, etc.) is delayed until the values are used. The result is that, if a property is queried more than once, the validation will be implemented in different forms, each with its own bugs, which will result in dangerous inconsistencies.Type safety: This is probably the trickiest bit. Every configuration node must be stored in the type that makes most sense for its value. For example: a timeout in seconds is an integer, so the in-memory representation must be an integer. Or another example: the type describing the "unprivileged user" is a data structure that maps to a system user, yet the configuration file just specifies either a username or a UID.Keeping strict type validation in the code is interesting because it helps to ensure that parsing and validation happen in just a single place: whenever the configuration file is read, every property will have to be converted to its in-memory type, and this means that the validation can only happen at that particular time. Once the data is in memory, we can and have to assume that it is valid. Additionally, strict types ensure that the code querying such properties uses the values as intended, without having to do additional magic to map them to other types.Extensibility: Parsing a configuration file is a very generic concept, yet the previous code made the mistake of tying this logic with the specific details of Kyua configuration files. A goal of the new code has been to write a library that parses configuration files, and allows the Kyua-specific code to define the schema of the configuration file separately. (No, the library is not shipped separately at this point; it's placed in its own utils::config module.)With all this code in place, there are a bunch of things that can now be easily implemented. Consider the following:Properties to define the timeout of test cases depending on their size (long-standing issue 5).Properties to tune the UI behavior: width of the screen, whether to use color or not (no, there is no color support yet), etc.Properties to configure how reports look like "by default": if you generate reports of any form frequently, it is very likely that you will want them to look the same every time and hence you will want to define the report settings once in the configuration file.Hooks: one of the reasons for using Lua-based configuration files was to allow providing extra customization abilities to the user. Kyua could theoretically call back into Lua code to perform particular actions, and such actions could be explicitly stated by the user in the form of Lua functions. Neither the current configuration code nor Kyua has support for hooks, but the new implementation makes it rather easy to add them.And that's all for today. Now that you know what the current code is trying to achieve and why, we will be able to look at how the implementation does all this in the next posts. [Continue reading]

  • Kyua's configuration system showing its age

    A couple of years ago, when Kyua was still a newborn, I wrote a very ad-hoc solution for the parsing and representation of its configuration files. The requirements for the configuration were minimal, as there were very few parameters to be exposed to the user. The implementation was quick and simple to allow further progress on other more-important parts of the project. (Yep, quick is an euphemism for dirty: the implementation of the "configuration class" has to special-case properties everywhere to deal with their particular types... just as the Lua script has to do too.)As I just mentioned in the previous paragraph, the set of parameters exposed through the configuration file were minimal. Let's recap what these are:Run-time variables: architecture and platform, which are two strings identifying the system; and unprivileged_user, which (if defined) is the name of the user under which to run unprivileged tests as. It is important to mention that the unprivileged_user is internally represented by a data type that includes several properties about a system user, and that it ensures that the data it contains is valid at all times. The fact that every property holds a specific type is an important design requirement.Test suite variables: every test suite can accept arbitrary configuration variables. Actually, these are defined by the test programs themselves. All of these properties are strings (and cannot be anything else because ATF test programs have no way of indicating the type of the configuration variables they accept/expect).Because of the reduced set of configurable properties, I opted to implement the configuration of the program as a simple data structure with one field per property, and a map of properties to represent the arbitrary test suite variables. The "parser" to populate this structure consists on a Lua module that loads these properties from a Lua script. The module hooks into the Lua metatables to permit things like "test_suites.NetBSD.timeout=20" to work without having to predeclare the intermediate tables.Unfortunately, as I keep adding more and more functionality to Kyua, I encounter additional places where a tunable would be appreciated by the end user (e.g. "disallow automatic line wrapping"). Exposing such tunable through a command-line flag would be a possibility, but some of these need to be permanent in order to be useful. It is clear that these properties have to be placed in the configuration file, and attempting to add them to the current codebase shows that the current abstractions in Kyua are not flexible enough.So, why am I saying all this? Well: during the last few weeks, I have been working on a new configuration module for Kyua. The goals have been simple:Have a generic configuration module that parses configuration files only, without any semantics about Kyua (e.g. what variables are valid or not). This ensures that the implementation is extensible and at the right level of abstraction.Be able to get rid of the ad-hoc parsing of configuration files.Allow defining properties in a strictly-typed tree structure. Think about being able to group properties by function, e.g. "kyua.host.architecture"; this is more or less what we have today for test-suite properties but the implementation is a special-case again and cannot be applied to other tunables.And... I am pleased to say that this code is about to get merged into the tree just in time for Kyua 0.4. In the next few posts, I will explain what the particular design constraints of this new configuration system were and outline a little bit its implementation. I think it's a pretty cool hack that mixes C++ data structures and Lua scripts in a "transparent" manner, albeit you may think it's too  complex. The key part is that, as this new configuration module is not specific to Kyua, you might want to borrow the code/ideas for your own use! [Continue reading]

  • Kyua gets nicer console messages

    For the last couple of weeks, particularly during a bunch of long flights, I have been improving the command-line user interface of Kyua by implementing controlled line wrappings on screen boundaries: messages that are too long to fit on the screen are preprocessed and split into multiple lines at word boundaries. This affects informational messages, error messages and, specially, the output of the built-in help command.I originally got this idea from Monotone and later implemented it into ATF but, when writing Kyua's code, I decided to postpone its implementation until a later stage. Reusing the code from ATF was not "nice" because the API of the formatting code was quite nasty, and reimplementing this feature during the initial stages of Kyua felt like a waste of time.However, because controlled line wrapping is crucial to having readable built-in help messages, I have had to do this eventually and the time finally came.The ATF approachWhy did I say that the ATF code for line wrapping was quite nasty? The main reason is that the printing of messages was incredibly tied to their wrapping. All the code in ATF that prints a message to the screen has to deal with the line wrapping itself, which involves dealing with too many presentation details.For example, consider the help routine that prints the table of options and their descriptions. This routine has to calculate the width of the longest option first and then, for every option, output its name, output some padding, and output the description properly refilled so that subsequent lines are properly arranged with respect to the previous one. While this may not sound too bad in writing, it actually is in code.Furthermore, because all this formatting logic is spread out throughout the code, there is no way to perform decent unit testing. The unit testing did some basic tests on input text, but could not validate that more complex constructions were working right.The Kyua approachIn the Kyua codebase, I decided to take a more declarative and functional approach. Smaller, pure building blocks that can be combined to achieve more complex constructions and that can be easily tested for correctness individually or in combination.I started by implementing a simple function that takes a paragraph and reformats it to any given length. This simple function alone gives full flexibility to the caller to decide how to later merge this reformatted text with other text: e.g. place a line prefix or bundle such text inside a table cell.The next step was to implement tables. Code wishing to print, e.g. the collection of options/commands along their descriptions only cares about declaring a table of two columns and N rows; why should it bother about properly lining up the two columns and printing them? It doesn't, hence the table approach. With tables, the caller can just decide which particular column needs to be wrapped if the table does not fit on the screen, and allow the formatting code to do this. Plus, having this higher level constructs means that we can eventually print the textual reports in a nicer, tabulated way (not done yet).And the last step was to mix all these higher level constructs into the console frontend class. This class (the ui) knows how to query the width of the terminal and knows how to fit certain kinds of text and/or tables within such width. For example, error messages are not tables: they are messages prefixed with the command name; only the message has to be reformatted if it does not fit while the rest of the text has to flow after the command name. Or, for tables, the maximum width of the terminal determines how wide the table can be and thus how much one of its columns has to be refilled.Getting this whole thing right working has proven to be extremely tricky and I'm sure there are still quite a few rough edges to be discovered. That said, it has been fun enough :-)  But, after this experience, I certainly don't want to imagine the pain that the writers of HTML/CSS renderers have endured... this text-based table-rendering is trivial compared to what web browsers do! [Continue reading]

  • Kyua generates its first public HTML report

    Lately, three long trips (5 hours in a bus, and 6 and 10 hours in two planes) have allowed me to work on the long-promised HTML reporting feature of Kyua. The result of these three trips is, effectively, the ability to generate HTML reports for specific test actions!The current results are extremely rudimentary (they lack tons of would-be-useful information) and not that aesthetically pleasing. However, the database already records enough information to make these reports more useful and pretty, so "all that is left" is coming up with the necessary code to extract such information in an efficient way and spending time creating a visually-nicer appearance. None of these are as trivial as they sound, but I prefer to work one step at a time (i.e. coming up first with a very rough draft and improve it later) rather than keeping the feature private until it is "perfect".Without further ado, you can take a look at the report of the execution of the NetBSD test suite. This output comes from my NetBSD-current virtual machine, and I've set up a couple of cron jobs to keep it up to date. (If the "action X" in the title does not increase periodically, you will know that something is broken on my side. I found that the VM already crashed, so I don't know for how long it will run now after a restart...) [Continue reading]

  • Projects migrated to Git

    I finally took the plunge. Yesterday night, I migrated the Kyua and Lutok repositories from Subversion to Git. And this morning I migrated ATF from Monotone and custom hosting to Git and Google Code; oh, and this took way longer than expected.Migration of Kyua and LutokMigrating these two projects was straightforward. After preparing a fresh local Git repository following the instructions posted yesterday, pushing to Google Code is a simple matter:$ git remote add googlecode https://code.google.com/p/your-project$ git push googlecode --all$ git push googlecode --tagsOne of the nice things I discovered while doing this is that a Google Code project supports multiple repositories when the VCS system is set to Git or Mercurial. By default, the system creates the default and wiki repositories, but you can add more at will. This is understandable given that, in Subversion, you have the ability to check out individual directories of a project whereas you cannot do that in the other supported VCSs: you actually need different repositories to group different parts of the project.I performed the full migration under a Linux box so that I could avail of the most recent Git version along with a proven binary package. The migration went alright, but I encountered a little problem when attempting a fresh checkout from NetBSD: git under NetBSD will not work correctly against SSL servers because it lacks the necessary CA certificates. The solution is to install the security/mozilla-rootcerts package and follow the instructions printed during installation; why this does not happen automatically escapes my mind.Migration of ATFI had been having doubts about migrating ATF itself, although if Kyua was moved to Git, it was a prerequisite to move ATF as well.  Certainly I could convert the repository to Git, but where could I host it afterwards?  Creating a new Google Code project just for this seemed too much of a hassle. My illumination came when I found out, as above, that Google Code supports an arbitrary amount of repositories in a particular project when converting it to Git.So, for ATF, I just ran mtn git_export with appropriate flags, created a new atf repository on the Kyua site, and pushed the contents there. Along the way, I also decided to kill the home-grown ATF web site and replace it by a single page containing all the relevant information. At this point, ATF and Kyua are supposed to work together quite tightly (in the sense that ATF is just a "subcomponent" of Kyua), so coupling the two projects on the same site makes sense.Now, let's drink the kool aid. [Continue reading]

  • Converting a Subversion repository to Git

    As discussed over a week ago, I have been pondering the idea of migrating my projects from Subversion to Git. One of the prerequisites of such a migration is the preparation of a process to cleanly migrate the revision history from the old system to the new one. Of course, such process should attempt to preserve the revision history as close to reality as possible (regardless of what some other big projects have done by just throwing away their history; shrug).The projects I am interested in migrating from Subversion to Git all live in Google Project Hosting. This is irrelevant for all but one detail I will mention later. However, the project hosting's site is the obvious place to look for help and, as expected, there is a little document titled Convert your project from Subversion to Git that provides a good start.Summarizing, there are two well-suited tools to convert a Subversion repository to Git:The built-in git svn command: works pretty well and has a good amount of customization features. The only little problem is that git svn does not recognize Subversion tags as such and therefore converts them to Git branches instead of tags.The third-party svn2git: while this is actually built on top of git svn, it properly handles Subversion tags and converts them to native Git tags.If you do not have tags in your Subversion repository or do not care about them being properly converted to Git tags, you can go with git svn. However, I am interested in properly recording tags, so I chose svn2git for my needs.Now, using svn2git by itself as instructed in the referenced document will happily convert your Subversion repository to Git. You should not have to care about anything else... unless you want to polish a few details mentioned below.Set upIn the rest of this article, I will place all the scripts for the conversion in a ~/convert/ directory. When the script is run, it will create a ./git/ subdirectory that will end up containing the new Git repository, and a ./control/ directory that will maintain temporary control files.Disclaimer: Be aware that all the code below is quite ugly and probably bug-ridden. It is not meant to be pretty as I only have two use cases for it. Also please note that I am, by no means, a Git expert, so the process below may be way more convoluted than is actually necessary.Oh, and this does not deal at all with the wiki changes. Google Code stores such information in the same Subversion repository as your code. However, when migrating to Git, you get two different repositories. Adapting the code below to deal with the wiki component of your project should be easy enough, although I'm not sure I care enough about the few contents I have in the wikis to go through the hassle.Warning: The instructions below completely mess up tags if you happen to have any in your repository. I only discovered this after following the procedure below and pushing HEAD and the tags, which resulted in a different set of revisions pushed for every tag. When I actually did this, I ended up running the steps below, then discarding the tags, and creating the tags again by hand pointing to the appropriate revisions.Setting up authorsThe first step in using svn2git is defining a mapping between Subversion authors to Git authors. This is easy enough, but there is one caveat that affects projects hosted in Google Code: the root revision of your Subversion repository is not actually authored by you; its author is (no author) so you should account for that. Something like this in your ~/.svn2git/authors file will tae care of this teeny tiny detail:your-address@example.net = Your Name (no author) = Your Name However, as we will see below, we will be skipping the first revision of the repository altogether so this is actually not really necessary. I just felt like mentioning it for completeness, given that I really didn't expect (no author) to be properly recognized in this context.References to old Subversion revision idsUnless you have been extremely terse in your commit history and in your bug tracker, you will have plenty of cross-references pointing to Subversion revision identifiers. For example, if you are fixing a bug introduced a month ago in r123, you may as well point that out in the commit message of r321 for reference purposes. I guess you can see the problem now: if we migrate the history "as is", all these references become meaningless because the new Git history has no traces of the old revision identifiers.The -m option to svn2git will annotate every revision with a git-svn-id line that contains the relevant information. However, such line is quite ugly because it is not really meant for human consumption: the information in such line is used by git-svn to allow pulls and pushes from a master Subversion repository.What I want to do is reword the git-svn-id line to turn it into a real sentence rather than some internal control code. We can achieve this with git rebase in interactive mode: mark all revisions as "reword" and then go revision by revision fixing its message. A pain that can be automated: if an editor can do the necessary changes, we can create a fake "editor" script that performs the same modifications.How? Store this as ~/convert/editor1.sh:#! /bin/shif grep git-svn-id "${1}"; then # We are editing a commit message. new="This revision was r\1 in Subversion." sed -r -i -e "s,git-svn-id[^@]+@([0-9]+).*$,${new}," "${1}"else # We are editing the git-rebase interactive control file. sed -i -e 's,^pick,reword,' "${1}"fiWith this script, we can simply do EDITOR=~/convert/editor1.sh git rebase -i base-revision and get every revision tagged... but this will blow up if your repository contains empty revisions, which takes us to the next concern.Drop empty revisionsAs you probably know, Subversion supports attaching metadata to directories and files in the form of properties. These properties cannot be represented in Git, so, if you have any Subversion commits that touched properties alone, svn2git will happily convert those as empty Git commits.There is nothing wrong with this, but things like git rebase will choke on these empty commits over and over again... and it gets quite annoying. Furthermore, these empty commits serve no purpose in Git because the changes they performed in Subversion make no sense in Git land. It is easier to just kill them all from the history.The git rebase command above will abort on every empty revision it encounters. We can, at that point, record their identifiers for later deletion. However, recording the revision identifier will not work because, as we are doing a rebase, the identifier will have changed once we are done. Instead, and because I have been careful to write detailed commit messages, we can rely on the first line of the message (aka the subject) to identify every message. Rerun the rebase as follows, storing the list of empty commits in ../control/empty:first=$(git log | grep '^commit' | tail -n 1 | cut -d ' ' -f 2)EDITOR="${convert}/editor1.sh" git rebase --interactive "${first}" || truetouch ../control/emptywhile [ -f .git/MERGE_MSG ]; do head -n 1 .git/COMMIT_EDITMSG >>../control/empty EDITOR="${convert}/editor1.sh" git commit --allow-empty EDITOR="${convert}/editor1.sh" git rebase --continue || truedoneWith this list in mind, we create another ~/convert/editor2.sh script to remove the empty revisions once we know them:#! /bin/shecho "Empty revisions to be deleted:"cat ../control/empty | while read line; do grep "${line}" "${1}" sed -i -e "/^pick ${line}$/s,^pick,fixup," "${1}"doneAmend the root revisionThe root revision in a Google Code subversion repository is empty: the system creates it to initialize the "directory layout" of your repository, but it serves no purpose. We can skip this root revision by passing the --revision=2 flag to svn2git.However, no matter what we do, the git rebase above to tag Git revisions with their corresponding Subversion identifiers, will happily skip our first real revision and leave it untagged. We have to manually go and fix this, which is actually quite tricky. Luckily, this reply in stackoverflow provides the solution.Putting it all togetherAlright then. If all the above was dense and cryptic code-wise, it is the time to put it all together in a script that performs all the steps for us. Assuming you already have ~/convert/editor1.sh and ~/convert/editor2.sh in place, now create ~/convert/driver.sh as follows:#! /bin/shset -e -x[ ${#} -eq 1 ] || exit 1convert=$(dirname ${0})project=${1}rm -rf git controlmkdir git controlcd git# Start at revision 2 to skip the initial empty revision.svn2git -v --revision=2 -m "http://${project}.googlecode.com/svn"# Tag git revisions with the original Subversion revision id.first=$(git log | grep '^commit' | tail -n 1 | cut -d ' ' -f 2)EDITOR="${convert}/editor1.sh" git rebase --interactive "${first}" || truetouch ../control/emptywhile [ -f .git/MERGE_MSG ]; do head -n 1 .git/COMMIT_EDITMSG >>../control/empty EDITOR="${convert}/editor1.sh" git commit --allow-empty EDITOR="${convert}/editor1.sh" git rebase --continue || truedone# Drop empty revisions recorded in the previous step.# The list is in the 'empty' file and is read by editor2.sh.EDITOR="${convert}/editor2.sh" git rebase --interactive "${first}" || true# Tag the root revision with the original Subversion revision id.git tag root $(git rev-list HEAD | tail -1)git checkout -b new-root rootEDITOR="${convert}/editor1.sh" git commit --amendgit checkout @{-1}git rebase --onto new-root rootgit branch -d new-rootgit tag -d rootcd -rm -rf controlYuck. Well, it works, and it works nicely. Converting a Subversion repository to Git will all the constraints above is now as easy as: ~/convert/driver.sh your-project-name! [Continue reading]

  • Kyua 0.3 released!

    Dear readers,I am pleased to announce that Kyua 0.3 is available!The major feature in this release is the introduction of the "test results store"; i.e. a SQLite3-based database that records all test execution activity so that reports can be gathered in a central machine and reports generated out of it. This is still very experimental and the generated reports are quite rudimentary, but is a first step towards this direction.The Kyua 0.3 release page provides links to the download as well as the list of major changes in this new release.It has been a really long time since the 0.2 release. I wanted to postpone 0.3 until HTML reports were ready, but I have not had the time to implement this for a while already. Instead of postponing the release even further, I have decided to create it right now and then take the time to create nice reports separately. Additionally, I am planning on doing some repository changes and wanted to do them without an imminent release along the way.The package in pkgsrc has been updated and now I'm working on bringing binary packages to Fedora.Enjoy! [Continue reading]

  • Encrypted disk images in NetBSD

    When I joined the NetBSD Board of Directors, I was trusted with access to private information and one of the prerequisites for downloading such data was to be able to store it safely: i.e. inside an encrypted volume. I initially went the easy route by putting the data in my laptop, which already uses FileVault 2 to encrypt the whole disk. But soon after, I decided that it would be nicer to put this data in my NetBSD home server, which is the machine that is perpetually connected to IRC and thus the one I use to log into the weekly meetings.At that point, I was faced with the "challenge" to set up an encrypted volume under NetBSD. I knew of the existence of cgd(4): cryptographic disk driver but had never used it. It really is not that hard, but there are a bunch of manual steps involved so I've been meaning to document them for almost a year already.Because my home server is a NetBSD/macppc and reinstalling the system is quite a nightmare, I did not want to go through the hassle of repartitioning the disk just to have an encrypted partition of 1GB. The choice was to create a disk image instead, store it in /secure.img and mount it on /secure. Let's see how to do this.The first step is to create the disk image itself and "loop-mount" it (in Linux terms). We do this the traditional way, by creating an empty file and setting up a vnd(4): vnode disk driver device to access it:# dd if=/dev/zero of=/secure.img bs=1m count=1024# vnconfig -c /dev/vnd2 /secure.imgThe next step is to configure a new cgd(4) device. This block-level device is a layer placed on top of a regular storage device, and is in charge of encrypting the data to the lower-level device transparently. cgdconfig(8), the tool used to configure cgd(4) devices, stores control information on a device basis under /etc/cgd/. The control information can be generated as follows:# cgdconfig -g -o /etc/cgd/vnd2c aes-cbc 192This creates the /etc/cgd/vnd2c configuration file, which you can choose to inspect now. With the file in place, configuring the cryptographic device for the first time is easy:# cgdconfig cgd0 /dev/vnd2cThis command will inspect the underlying /dev/vnd2c device looking for a signature indicating that the volume is valid and already encrypted. Because the volume is still empty, the tool will proceed to ask us for an encryption key to initialize the volume. We enter our key, wait a couple of seconds, and /dev/cgd0 will be ready to be used as a regular disk device. With that in mind, we can proceed to create a new file system and mount it on the desired location:# newfs -O 2 /dev/rcgd0c# mkdir /secure# mount /dev/cgd0c /secureTo simplify access to the device for your users, I did something like this:# user=your-user-name# mkdir /secure/${user}# chown ${user}:users /secure/${user}# chmod 700 /secure/${user}# ln -s /home/${user}/secure /secure/${user}And we are done. Now, depending on the situation, we could choose to get /secure automatically mounted during boot, but that would involve having to type the decryption key. This is OK for a laptop, but not for a headless home server. Because I don't really need to have this secure device mounted all the time, I have the following script in /root/secure.sh that I can use to mount and unmount the device at will:#! /bin/shset -e -xcase "${1:-mount}" inmount) vnconfig -c /dev/vnd2 /secure.img cgdconfig cgd0 /dev/vnd2c fsck -y /dev/cgd0c mount /dev/cgd0c /secure ;;unmount) umount /secure cgdconfig -u cgd0 vnconfig -u /dev/vnd2 ;;esacBe aware that cgdconfig(1) has many more options! Take a look at the manual page and choose the ones that best suit your use case.Random idea: instead of using a vnd(4) device, plug in a USB stick and use that instead for your secure data!  (I actually do this too to back up sensitive information like private keys.) [Continue reading]

  • NetBSD 6.0 BETA tagged

    Dear users of NetBSD,I am pleased to announce that we (well, the release engineering team!) have just tagged the netbsd-6 branch in the CVS repository and thus opened the gate for testing of NetBSD 6.0_BETA. New binary snapshots should start appearing in the daily FTP archive soon. You can, of course, perform a cvs update -r netbsd-6 on your existing source tree and roll your own binaries (as I'm already doing on my home server).Please help us make NetBSD 6.0 the best release ever! As you may already know, it will provide tons of new features compared to the ancient 5.x release series that will make your user experience much better. The branch just needs a bit of love to shake out any critical bugs that may be left and to ensure that all those annoying cosmetic issues are corrected. So, if you find any of those, do not hesitate to file a problem report (PR).Thank you! [Continue reading]

  • Kyua: Weekly status report

    A couple of things happened this week:Spent quite a few time researching the idea of moving away from Monotone and Subversion to Git. I haven't made a decision yet, but I'm pretty convinced this is the right way to go. It will simplify development significantly, it will allow me to code offline (have a bunch of really long flights coming), and it will lower the entry barrier to Kyua by making all components use the same, mainstream VCS.Implemented filtering by result type of the test case results in the textual reports.I think it is time to prepare a 0.3 release. I wanted to wait until we had HTML reports in place, but this will require significant effort and I have been postponing the implementation for too long already. As it is now, the current codebase provides major changes since the ancient 0.2 release and it is worth a release. Then, we can create packages for NetBSD and Fedora instead of continuing to add new features, which should be a good step in giving further visibility to the project. Finally, we can reserve HTML reporting as the major feature for 0.4 :-) [Continue reading]

  • Switching projects to Git

    The purpose of this post is to tell you the story of the Version Control System (VCS) choices I have made while maintaining my open source projects ATF, Kyua and Lutok. It also details where my thoughts are headed to these days.This is not a description of centralized vs. distributed VCSs, and it does not intend to be one. This does not intend to compare Monotone to Git either, although you'll probably feel like it while reading the text. Note that I have fully known the advantages of DVCSs over centralized systems for many years, but for some reason or another I have been "forced" to use centralized systems on and off. The Subversion hiccup explained below is... well... regrettable, but it's all part of the story!Hope you enjoy the read.Looking back at Monotone (and ATF)I still remember the moment I discovered Monotone in 2004: simply put, it blew my mind. It was clear to me that Distributed Version Control Systems (DVCSs) were going to be the future, and I eagerly adopted Monotone for my own projects. A year later, Git appeared and it took all the praise for DVCSs: developers all around started migrating en masse to Git, leaving behind other (D)VCSs. Many of these developers then went on to make Git usable (it certainly wasn't at first) and well-documented. (Note: I really dislike Git's origins... but I won't get into details; it has been many years since that happened.)One of the projects in which I chose to use Monotone was ATF. That might have been a good choice at the time despite being very biased, but it has caused problems over time. These have been:Difficulty to get Monotone installed: While most Linux distributions come with a Monotone binary package these days, it was not the case years ago. But even nowadays if all Linux distributions have binary packages, the main consumers of ATF are NetBSD users, and their only choice is to build their own binaries. This generates discomfort because there is a lot of FUD surrounding C++ and Boost.High entry barrier to potential contributors: It is a fact that Monotone is not popular, which means that nobody is familiar with it. Monotone's CLI is very similar to CVS, and I'd say the knowledge transition for basic usage is trivial, but the process of cloning a remote project was really convoluted until "recently". The lack of binary packages, combined with complex instructions on just how to fetch the sources of a project only help in scaring people away.Missing features: Despite years have passed, Monotone still lacks some important features that impact its usability. For example, to my knowledge, it's still not possible to do work-directory merges and, while the interactive merges offered by the tool seem like a cool idea, they are not really practical as you get no chance to validate the merge. It is also not possible, for example, to reference the parent commit of any given commit without looking at the parent's ID. (Yeah, yeah, in a DAG there may be more than one parent, but that's not the common case.) Or know what a push/pull operation is going to change on both sides of the connection. And key management and trust has been broken since day one and is still not fixed. Etc, etc, etc.No hosting: None of the major project hosting sites support Monotone. While there are some playground hosting sites, they are toys. I have also maintained my own servers sometimes, but it's certainly inconvenient and annoying.No tools support: Pretty much no major development tools support Monotone as a VCS backend. Consider Ohloh, your favorite bug tracking system or your editor/IDE. (I attempted to install Trac with some alpha plugin to add Monotone support and it was a huge mess.)No more active development: This is the drop that spills the cup. The developers of Monotone that created the foundations of the project left years ago. While the rest of the developers did a good job in coming up with a 1.0 release by March 2011, nothing else has happened since then. To me, it looks like a dead project at this point :-(Despite all this, I have been maintaining ATF in its Monotone repository, but I have felt the pain points above for years.Furthermore, the few times some end user has approached ATF to offer some contribution, he has had tons of trouble getting a fresh checkout of the repository and given up. So staying with Monotone hurts the project more than it helps.The adoption of Subversion (in Kyua)To fix this mess, when I created the Kyua project two years ago, I decided to use Subversion instead of a DVCS. I knew upfront that it was a clear regression from a functionality point of view, but I was going to live with it. The rationale for this decision was to make the entry barrier to Kyua much lower by using off-the-shelf project hosting. And, because NetBSD developers use CVS (shrugh), choosing Subversion was a reasonable choice because of the workflow similarities to CVS and thus, supposedly, the low entry barrier.Sincerely, the choice of Subversion has not fixed anything, and it has introduced its own trouble. Let's see why:ATF continues to be hosted in a Monotone repository, and Kyua depends on ATF. You can spot the problem, can't you? It's a nightmare to check out all the dependencies of Kyua, using different tools, just to get the thing working.As of today, Git is as popular, if not more, than Subversion. All the major operating systems have binary packages for Git and/or bundle Git in their base installation (hello, OS X!). Installing Git on NetBSD is arguably easier (at least faster!) than Subversion. Developers are used to Git. Or let me fix that: developers love Git.Subversion gets on the way more than it helps; it really does once you have experienced what other VCSs have to offer. I currently maintain independent checkouts of the repository (appropriately named 1, 2 and 3) so that I can develop different patches on each before committing the changes. This gets old really quickly. Not to mention when I have to fly for hours, as being stuck without an internet connection and plain-old Subversion... is suboptimal. Disconnected operation is key.The fact that Subversion is slowing down development, and the fact that it really does not help in getting new contributors more than Git would, make me feel it is time to say Subversion goodbye.The migration to GitAt this point, I am seriously considering switching all of ATF, Lutok and Kyua to Git. No Mercurial, no Bazaar, no Fossil, no anything else. Git.I am still not decided, and at this point all I am doing is toying around the migration process of the existing Monotone and Subversion repositories to Git while preserving as much of the history as possible. (It's not that hard, but there are a couple of details I want to sort out first.)But why Git?First and foremost, because it is the most popular DVCS. I really want to have the advantages of disconnected development back. (I have tried git-svn and svk and they don't make the cut.)At work, I have been using Git for a while to cope with the "deficiencies" of the centralized VCS of choice. We use the squashing functionality intensively, and I find this invaluable to constantly and shamelessly commit incomplete/broken pieces of code that no-one will ever see. Not everything deserves being in the recorded history!Related to the above, I've grown accustomed to keeping unnamed, private branches in my local copy of the repository. These branches needn't match the public repository. In Monotone, you had this functionality in the form of "multiple heads for a given branch", but this approach is not as flexible as named private branches.Monotone is able to export a repository to Git, so the transition is easy for ATF. I have actually been doing this periodically so that Ohloh can gather stats for ATF.Lutok and ATF are hosted in Google Code, and this hosting platform now supports Git out of the box.No Mercurial? Mercurial looks a lot like Monotone, and it is indeed very tempting. However, the dependency on Python is not that appropriate in the NetBSD context. Git, without its documentation, builds very quickly and is lightweight enough. Plus, if I have to change my habits, I would rather go with Git given that the other open source projects I am interested in use Git.No Bazaar? No, not that popular. And the fact that this is based on GNU arch makes me cringe.No Fossil? This tool looks awesome and provides much more than DVCS functionality: think about distributed wiki and bug tracking; cool, huh? It also appears to be a strong contender in the current discussions of what system should NetBSD choose to replace CVS. However, it is a one-man effort, much like Monotone was. And few people are familiar with it, so Fossil wouldn't solve the issue of lowering the entry barrier. Choosing Fossil would mean repeating the same mistake as choosing Monotone.So, while Git has its own deficiencies — e.g. I still don't like the fact that it is unable to record file moves (heuristics are not the same) — it seems like a very good choice. The truth is, it will ease development by a factor of a million (OK, maybe not that much) and, because the only person (right?) that currently cares about the upstream sources for any of these projects is me, nobody should be affected by the change.The decision may seem a bit arbitrary given that the points above don't provide too much rationale to compare Git against the other alternatives. But if I want to migrate, I have to make a choice and this is the one that seems most reasonable.Comments? Encouragements? Criticisms? [Continue reading]

  • A brief look into Fedora's packaging infrastructure

    As you probably know, I have been a long-time "evangelist" of pkgsrc. I started contributing to this packaging system when I first tried NetBSD in 2001 by sending new packages for small tools, and I later became a very active contributor while maintaining the GNOME packages. My active involvement came to an end a few years ago when I switched to OS X, but I still maintain a few packages and use pkgsrc in my multiple machines: a couple of NetBSD systems and 3 OS X machines.Anyway. pkgsrc is obviously not everything in this world, and if I realistically want other people to use my software, there have to be binary packages for more mainstream systems. Let's face it: nobody in their sane mind is going to come over to my project pages, download the source package, mess around with dependencies that do not have binary packages either, and install the results. Supposedly, I would need just one such person, which by coincidence would also be a packager of a mainstream distribution, to go through all these hoops and create the corresponding binary packages. Ah yes, what I said: not gonna happen anytime soon.Sooooo... I spent part of past week learning (again) how to create binary packages for Fedora and, to bring this into practice, I prepared an RPM of lutok and pushed it to Fedora rawhide and Fedora 16. All in all, it has been a very pleasant experience, and the whole point of this post is to provide a small summary of the things I have noticed. Because I know pretty well how pkgsrc behaves and what its major selling points are, I am going to provide some pkgsrc-related comments in the text below.Please note that the text below lacks many details and that it may claim some facts that are not completely accurate. I'm still a novice in Fedora development land.First, let's start describing the basic components of a package definition:spec file: The spec file of a package is RPM's "source description" of how to build a package (pkgsrc's Makefile) and also includes all the package's metadata (pkgsrc's PLIST, DESCR, etc.). This does not include patches nor the original sources. I must confess that having all the details in a single file is very convenient.File lists: Contrary to (common?) misconception, spec files can and do have an explicit list of the files to be included in the package (same as pkgsrc's PLIST). This list can include wildcards to make package maintenance easier (e.g. you can avoid having to list all files generated by Doxygen and just include the directory name, which will just do the right thing). No matter what you do, the build system will ensure that all files generated by the package are part of the file list to ensure that the package is complete.SRPMs: Think about this as a tarball of all the files you need to build a particular package, including the spec file, the source tarball and any additional patches. These files are very convenient to move the source package around (e.g. to publish the package for review or to copy it to a different machine for rebuilding) and also to upload the package to Koji's build system (see below).Subpackages: Oh my... what a relief compared to pkgsrc's approach to this. Creating multiple independent packages from a single spec file is trivial, to the point where providing subpackages is encouraged rather than being a hassle. For what is worth, I have always liked the idea of splitting development files from main packages (in the case of libraries), which in many cases helps in trimming down dependencies. pkgsrc fails miserably here: if you have ever attempted to split a package into subpackages to control the dependencies, you know what a pain the process is... and the results are a collection of unreadable Makefiles.Now let's talk a bit about guidelines and access control:Policies: I was going to write about documentation in this point, but what I really wanted to talk about are policies. There are several policies governing packaging rules, and the important thing is that they are actually documented (rather than being tribal knowledge). The other nice thing is that their documentation is excellent; just take a moment to skim through the Packaging Guidelines page and you will see what I mean. The packaging committee is in charge of editing these policies whenever necessary.Review process: Any new package must go through a peer review process. Having grown accustomed to Google's policy of having every single change to the source tree peer-reviewed, I can't stress how valuable this is. It may seem like a burden to newcomers, but really, it is definitely worth it. The review process is quite exhaustive, and from what I have seen so far, the reviewers tend to be nice and reasonable. As an example, take a look at lutok's review.Repository and ACLs: The source files that describe a package (mainly a spec file and a sources file) are stored in a Git repository (I believe there is a different repository for every package, but I may be wrong). This is nothing unusual, but the nice thing is that each package has its own read/write ACLs. New maintainers have access to their own packages only, which means that the barrier of entry can be lowered while resting assured that such contributors cannot harm the rest of the packages until they have gained enough trust. Of course, there are a set of trusted developers  that can submit changes to any and every package."But you said packaging infrastructure in the title!", you say. I know, I know, and this is what I wanted to talk most about, so here it goes:Common tools: Other than the well-known rpm and yum utilities, developers have access to rpmbuild and fedpkg. rpmbuild would be rpm's counterpart, in the sense that it is the lowest level of automation and exposes many details to the developer. fedpkg, on the other hand, is a nice wrapper around the whole packaging process (involving git, mock builds, etc.).Koji: Koji is Fedora's build system, ready to build packages for you on demand from a simple command-line or web interface. Koji can be used to test the build of packages during the development process on architectures that the developer does not have (the so-called "scratch builds"). However, Koji is mainly used to generate the final binary packages that are pushed into the distribution. Once the packager imports a new source package into the repository, he triggers the build of binary packages to include them later into the distribution.Bodhi: Bodhi is Fedora's update publishing system. When a packager creates a new version of a particular package and wishes to push such update to a formal release (say, Fedora 16), the update is first posted in Bodhi. Then, there are a set of scripts, rules and peer reviews that either approve the update for publication on the branch or not.Let's now talk a bit about pkgsrc's waived strengths and how they compare to Fedora's approach:Mass fixes: In pkgsrc, whenever a developer wants to change the infrastructure, he can do the change himself and later adjust all existing packages to conform to the modification. In Fedora, because some particular developers have write access to all packages, it seems certainly possible to apply a major fix and/or rototill to all packages in the same manner as is done in pkgsrc. Such developer could also trigger a rebuilt of all affected packages using a specific branch for testing purposes and later ensure that the modified packages still work.Isolated builds: buildlink3 is an awesome pkgsrc technology that isolates the build of a particular package from the rest of the system by means of symlinks and wrapper scripts. However, pkgsrc is not alone. Mock is Fedora's alternative to this: Mock provides a mechanism to build packages in a chroot environment to generate deterministic packages. The tools used to generate the "blessed" binary packages for a distribution (aka Koji) use this system to ensure the packages are sane.Bulk builds: This is a term very familiar to pkgsrc developers, so I'm just mentioning it en-passing because this is also doable in RPM-land. While package maintainers are responsible for building the binary packages of the software they maintain (through Koji), authorized users (e.g. release engineering) can trigger rebuilds of any or all packages.And, lastly, let's raise the few criticisms I have up to this point:Lack of abstractions: spec files seem rather arcane compared to pkgsrc Makefiles when it comes to generalizing packaging concepts. What I mean by this is that spec files seem to duplicate lots of logic that would better be abstracted in the infrastructure itself. For example: if a package installs libraries, it is its responsibility to call ldconfig during installation and deinstallation. I have seen that some things that used to be needed in spec files a few years ago are now optional because they have moved into the infrastructure, but I believe there is much more that could be done. (RHEL backwards compatibility hurts here.) pkgsrc deals with these situations automatically depending on the platform, and extending the pkgsrc infrastructure to support more "corner cases" is easier.No multi-OS support: One of the major selling points of pkgsrc is that it is cross-platform: it runs under multiple variants of BSD, Linux, OS X, and other obscure systems. It is true that RPM also works on all these systems, but Fedora's packaging system (auxiliary build tools, policies, etc.) does not. There is not much more to say about this given that this is an obvious design choice of the developers.To conclude: please keep in mind that the above is not intended to describe Fedora's system as a better packaging system than pkgsrc. There are some good and bad things in each, and what you use will depend on your use case or operating system. What motivated me to write this post were just a few small things like Koji, Bodhi and subpackages, but I ended up writing much more to provide context and a more detailed comparison with pkgsrc. Now draw your own conclusions! ;-) [Continue reading]

  • Kyua: Weekly status report

    Created an RPM package for Lutok for inclusion in Fedora.Created a preliminary RPM spec for ATF for Fedora. Now in discussions with the FPC to figure out how to install the tests on a Fedora system, as /usr/tests may not be appropriate.No activity on Kyua itself though, unfortunately. [Continue reading]

  • Kyua: Weekly status report

    This comes two days late... but anyway, I felt like posting it now instead of waiting until next Sunday/Monday.The activity past week focused mostly on implementing support for the require.memory test-case metadata property recently introduced into ATF. This was non-trivial due to the need to write some tricky Autoconf code to make this "slightly portable". Seriously: It's scary to see how hard it is to perform, in a portable manner, an operation as simple as "query the amount of physical memory"... but oh well, such are the native Unix APIs...I later spent the weekend on Lutok, preparing and publishing the project's first release and writing my first RPM library spec for it. This was a prerequisite for the upcoming 0.3 release of Kyua and thus deserved special attention! [Continue reading]

  • Lutok 0.1 available

    A few months ago, I introduced the Lutok project, a simple C++ API for Lua. To recap: the major goal of this API, which does not mimic the Lua C API bit by bit, is to enforce correct coding practices on the client side. This is done by (ab)using the RAII programming pattern to automatically free resources when not needed and to ensure that the Lua stack is correctly managed. The library also adheres to common C++ programming idioms and exposes exceptions for error management and uses the pimpl idiom to completely hide the Lua C API from clients of Lutok (unless you use the c_gate backdoor!).Today, I am pleased to announce that the first formal release of Lutok, obviously named 0.1, is available for download! You can obtain this release by visiting the lutok-0.1.tar.gz download page.Also, in preparation for this release, I have spent the weekend writing some little example programs to demonstrate the usage of Lutok, and also some installation instructions.Hope you find this release useful and please do send me any comments you may have!Hint: Yes, releasing Lutok 0.1 was a prerequisite for Kyua 0.3. So stay tuned ;-) [Continue reading]

  • Kyua: Weekly status report

    Released ATF 0.15 and imported it into NetBSD.Added support for integer/float printf-like modifiers to the utils::format module. These will be required to beautify size and time quantities in the reports and error messages.I spent way more time than I wanted on this. At first, I attempted to use std::snprintf to parse and process the format modifiers for integers and floats so that I could avoid implementing a custom parser for them. While this sounds like a cool idea (yay, code reuse!), it resulted in a ugly, nasty and horrible mess. In the end, I just ended up implementing custom parsing of the formatters, which was way easier and "good enough" for Kyua's needs.Started work on backporting ATF's new require.memory property into Kyua. This needs having a way to parse and format byte quantities in user-friendly forms (e.g. 1k, 2m, etc.)... hence the previous work on utils::format!Set up a Google+ Page for Kyua. I have no idea what to use it for yet. Maybe the status reports should go in there. Ideas? [Continue reading]

  • Kyua: Weekly status report

    Finally some progress!Backported the require.memory changes in NetBSD to the ATF upstream code, and extended them to support OS X as well.Backported local pkgsrc patches to ATF into the upstream code.Started to prepare ATF 0.15 by doing test runs of NetBSD/i386 and NetBSD/amd64 and by building the code in various Linux distributions. Several build bugs fixed along the way.Spent a long while trying to figure out how the Fedora package maintainer procedure has changed since 3 years ago to create packages for ATF, Lutok and Kyua. Not very successful yet unfortunately.Nothing on the Kyua front, but getting a new release of ATF out of the door has higher priority now! [Continue reading]

  • Kyua: Weekly status report

    It's that time of the week again (although delayed by over a day). Unfortunately, no progress either during past week. Being on semi-vacations doesn't leave much "free time"... However, I traveled back home yesterday and getting back to my daily routine should give some spare time! [Continue reading]

  • Kyua: Weekly status report

    Happy new year!No activity past week as I was completely away from the computer. [Continue reading]