• Kyua: Weekly status report

    Some cool stuff this week, albeit not large-scale:Implemented the --variable flag in the test command. This flag, which can be specified multiple times, allows a user to override any configuration variable (be it a built-in or a test-suite variable) from the command line. This is actually the same as atf-run's -v flag, but with a clear separation between built-in configuration settings and test-suite specific settings.Added support for several environment variables to allow users (and tests) to override built-in paths. I can't imagine right now any legitimate use for these variables, but hardcoded values are bad in general, atf-run provided these same variables, and these variables are very handy for testing purposes.Added support for the new require.files test-case metadata property to both ATF and Kyua. This new property allows tests to specify a set of files that they require in order to run, and is useful for those tests that can't run before make install is executed.The functionality planned for the 0.1 release is now pretty much complete. There is still a few rough edges to clean, some documentation to write, and some little features to implement/fix. See the open bugs for 0.1 to get an idea of the remaining tasks. [Continue reading]

  • Kyua: Weekly status report

    This week:Cleaned up the internal code of the "list" command and added a few unit tests.Added integration tests for the "test" command that focus mostly on the behavior of the "test" command itself. There is still a need for full integration tests that validate the execution of the test cases themselves and their cleanup, and these will be tricky to write.Changed atf-c, atf-c++ and atf-sh to show a warning when a test program is run by hand. Users should really be using atf-run to execute the tests, or otherwise things like isolation or timeouts will not work (and they'll conclude that atf is broken!). [Continue reading]

  • Joining the Board of Directors of The NetBSD Foundation

    Dear readers,It is a great pleasure for me to announce that I have just joined the Board of Directors of The NetBSD Foundation.If you are curious about how this all happened, here it goes: as described in the election procedure, someone who I don't know nominated me back in November of 2010 to become part of the new board composition. After the Nomination Committee made their way through the long list of nominees, interviews and deliberation, they proposed a slate for the new members of the board. This slate included two people who were renewing their previous term (tron@ and reed@) and two new members (spz@ and jmmv@) to replace the two members stepping down (agc@ and david@). The final approval of the proposed slate happened in April 2011 and, yesterday, it became official.Looking back, I can't believe it has been already ~10 years since I first started using NetBSD. Ah, those were the times of 1.5. My responsibilities within the project have shifted a lot during this time, ranging from the maintenance of GNOME and several web site tasks, the development of the testing framework (which is still ongoing today), and, from today, to my new duties within the board.I'm looking forward to working on this new area of the project and I hope to meet the requirements of such position. The two members being replaced will be missed, and keeping up to the high bar they left behind will be tough. That said, I hope spz@ and I will be able to meet the expectations.If you have any ideas or concerns regarding the direction of The NetBSD Project, don't hesitate to send them to the board@! [Continue reading]

  • Kyua: Weekly status report, BSDCan 2011 edition

    I spent past week in Ottawa, Canada, attending the BSDCan 2011 conference. The conference was composed of lots of interesting content and hosted many influential and insightful BSD developers. While the NetBSD presence was very reduced, I could have some valuable talks with both NetBSD and FreeBSD developers.Anyway. As part of BSDCan 2011, I gave a talk titled "Automated testing in NetBSD: past, present and future". The talk focused on explaining what led to the development of ATF in the context of NetBSD, what related technologies exist in NetBSD (rump, anita and dashboards), what ATF's shortcomings are and how Kyua plans to resolve them. (Video coming soon, I hope.)The talk was later followed by several questions and off-session conversations about testing in general in BSDs. Among these, I gathered a few random ideas / feelings:The POSIX 1003.3 standard defines the particular results a test can emit (see the corresponding DejaGnu documentation). Both ATF and Kyua already implement all the results defined in the standard, but they use different names and extend the standard with many extra results. Given that the standard does not define useful concepts like "expected failures", an idea that came up is to provide a flag to force POSIX compliance at the cost of being less useful. Why? Just for the sake of saying that Kyua conforms to this standard.The audience seemed to like the idea of a "tests results store" quite a bit, and the sound of SQLite for the implementation was not bullied. This is something I'm eager to work on, but not before I publish a 0.1 release.I highlighted the possibility of allowing Kyua to run "foreign" test programs so that we could integrate the results into the database. This could be useful to run tests for which we (*BSD) have no control (e.g. gcc) in an integrated manner. The idea was not bullied by anyone either.FreeBSD has already been looking at ATF / Kyua and they are open to collaboration.OpenBSD won't import any new C++ code, and adding C-based tests to the tree while relegating the C++ runtime to the ports is not an option. Somehow I expected this.Junos (the FreeBSD-based operating system from Jupiter Networks) recently imported ATF and they are happy with it so far. Yay!Would be nice to have a feature to run tests remotely after, maybe, deploying a single particular test and its dependencies. This is gonna be tricky and not in my current immediate plans.Other than that, I had little time to do some coding:Fixed a problem in which both ATF and Kyua were not correctly resetting the timezone of the executed tests. I only found this because, after arriving in Canada, some Kyua tests would start to fail. (Yes, the fix is in both code bases!)Added some support to capture deadly signals that terminate Kyua so that Kyua can print an informational message stating that something went wrong and which log file contains more information. See r121.That's it folks! Thanks to those attending the conference and, in particular, to those that came to my talk :-) [Continue reading]

  • Kyua: Weekly status report

    Unfortunately, no progress whatsoever this week :-( Too busy at work and preparing my upcoming trips. Time to fly to BSDCan 2011 tomorrow. [Continue reading]

  • Kyua: Weekly status report

    Unfortunately, I have had no time for coding this week. The only things I could do were:Fixed a few build problems on NetBSD introduced during past week's changes.Built Kyua and ran a few tests on NetBSD/macppc (just for the joy of it).Coding has been eclipsed by the preparation of my presentation for BSDCan 2011; at this point, this has priority over any code changes. I'd argue that preparing the presentation is also part of the project, so some time has been invested ;-)More next week, hopefully, but I don't expect being able to do any big code improvements until after BSDCan. [Continue reading]