• Kyua: Weekly status report

    My goal for this past week was to change the database layer to be able to store full definitions of the test cases, and to later be able to load these while scanning an action. This is to allow the report command to provide all kinds of information about the executed tests, not just their names and their results. However, adding this functionality has proven to be more complex than I wished because the current types to represent test programs and test cases are kinda broken: that the abstractions chosen a while ago do not seem to be appropriate, and this is complicating further changes.Due to this, I ended up doing some cleanups. First, I reimplemented the way in which test programs that fail to list their test cases are represented. And second, I got rid of the useless test_case_id class, which exposed even further problems in the data types that represent test cases.It's now time to sit and think if the current representations of test programs and test cases make sense and, if not, how to better redo them. Not going to be easy, but I hope to have some time for this cleanup during this upcoming week. [Continue reading]

  • Kyua: Weekly status report

    I only have one thing to report this week, but oh boy it's big: the report command finally reports the results of a run of a test suite! Yes, you heard well: Kyua is, finally, able to keep a record of all the previous executions of test suites and allows you to extract reports of any of them a posteriori.At the moment, the report is just some disorganized plain-text.  For example:$ kyua test[... wait for the tests to run ...]Committed action 82$ kyua report===> Skipped tests[... too long to show ...]===> Expected failuresintegration/cmd_report_test:output__console__change_file  ->    expected_failure: --output not implemented yet:    atf-check failed; see the output of the test for details===> Failed testsstore/transaction_test:put_test_case__ok  ->    failed: Line 663: stmt.step() not met===> SummaryAction: 82Test cases: 934 total, 15 skipped, 1 expected failures,0 broken, 1 failedI'm now working on changing the database schema to be able to really store all the data about test cases, because at the moment I'm only storing their names. Once all the original data is stored, the report command will have lots more information to work with, and then will be the time to start improving the format of the reports. As mentioned earlier, having interactive HTML dashboards is high in the priority list, and a very important goal of Kyua altogether.Stay tuned :-) [Continue reading]

  • Kyua: Weekly status report

    Kyua has finally gained a report subcommand, aimed at processing the output data of an action (stored in the database) and generating a user-friendly report in a variety of formats. This is still extremely incomplete, so don't get your hopes too high yet ;-) The current version of the report command takes an action and all it does is dump its runtime context (run directory, environment variables, etc.). Consider it just a proof of concept.I have now started work on loading the data of test case results for a particular action, and once that is done, the report command will start yielding really useful data: i.e. it will actually tell you what happened during a particular execution of a test suite. The way I'm approaching the work these days is by building the skeleton code to implement the basic functionality first (which actually involves writing a lot of nasty code), with the goal of adding missing pieces later bit by bit.For example, at this moment I'm only targeting text-based outputs with a limited set of data. However, when that is done, adding extra data or different formats will be relatively easy. Generating HTML dashboards (without going through XML, as was the case of atf-report!) is definitely highly prioritized.By the way: I just realized it has already been one year since Kyua saw life. Wow, time flies. And only now we are approaching a point where killing the atf-run / atf-report pair is doable. I'm excited. [Continue reading]

  • Kyua: Weekly status report

    Many things have happened this week, but they can all be summarized in one single sentence: kyua test now records the results of the execution of a test suite into the SQLite database."Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results. It's interesting to realize that most of these features are just one SQL query away. If you install Kyua, you can already run a few tests and then use kyua db-exec to issue arbitrary SQL queries against the database; the schema (see store/schema.sql) might look a bit convoluted, but a bunch of NATURAL JOINs will yield the desired output.The feature requests that have the highest priority at this point are the ability to generate a report of the last tests run both as a text file and as an HTML dashboard, because having these features means we can finally kill the atf-run and atf-report pair. At this point I'm, once again, "stuck" while figuring out how to best organize the code to make all these things possible while still keeping a nice separation across the existing layers (cli, engine and store)... all without introducing much unnecessary complexity. But exciting times lie ahead! [Continue reading]