Foster’s Tests and Builds Automation System

After examining the python-bootstrap and build system for foster that we’ve been working on, in the context of the previous post about a potential port to C being in analysis, I’ve found a way to do all of this in a much simpler way.

First, a re-examining of the requirements.

We need a test system that is:

  • for environment variables, package or binary versions, package or binary presence, symlink targets, and all manners of other little flexible checks that could really be looking at anything and be done in different ways on different linux systems.
  • relatively abstracted away from the distribution running it without having to alter the implementation of the test system every time we find a new type of check.
  • Massively extendable in a simple way that can either result in atomic use cases that are easy to implement or absolutely gigantic and comprehensive uses across many purposes for more robust solutions.
  • Designed in a way that prevents configuration complexity from scaling with usage complexity (huge solutions are just as easy to configure as small solutions).
  • Rigidly simple in testing order.

Introducing: Examplar

“Not your father’s testing framework.”

First, some updates to the data model allowed the new design to take shape:

[code]
{
"units": [
{
"name": "bash",
"target": "./ubuntu/xenial/check_bash_version.run",
"output": "4.3.46(1)-release",
"rectifier": "./ubuntu/xenial/install_correct_bash.run",
"active": true,
"required": true,
"rectify": false
},
{
"name": "bash2",
"target": "./ubuntu/xenial/check_bash_version.run",
"output": "4.3.46(1)-release",
"rectifier": "./ubuntu/xenial/install_correct_bash.run",
"active": true,
"required": true,
"rectify": false
}
],
"plan": [
{ "name": "bash", "depends on": [ null ] },
{ "name": "bash2", "depends on": [ "bash" ] }
]
}

[/code]

To read the above, we have two JSON objects here.  One is units, which define/declare the tests themselves.  The other is plan which orders the execution of units to be tested and sets test interdependency (flow control, execution, implementation).

The tool will cycle through the plan object’s entries, executing units using their unique name attribute as a handle.

To do this, the tool will deserialize units, execute each corresponding unit’s target value when that unit’s active property is true and is next in the plan object’s test order.  The unit’s target attribute points to an executable.  That executable serves as the test.  If target returns a non-zero exit code or if target‘s output does not match the unit’s output attribute value then the test will fail.  If the unit’s required attribute is set to true it will stop the tests.  Provided these all happen well it will continue to the next unit test in the deserialized collection by order of what’s in plan.

If a test is dependent on other assumptions, say, in our use case we have “making sure GCC compiles” we want that to depend on “making sure GCC is present”, we’d have something like:

[code]
{
"units": [
{
"name": "gcc is present",
"target": "./ubuntu/xenial/check_gcc_present.run",
"output": "present",
"rectifier": "./ubuntu/xenial/install_gcc",
"active": true,
"required": true,
"rectify": true
},
{
"name": "gcc can compile",
"target": "./ubuntu/xenial/check_gcc_compiles.run",
"output": "can compile",
"rectifier": "",
"active": true,
"required": true,
"rectify": false
}
],
"plan": [
{ "name": "gcc is present", "depends on": [ null ] },
{ "name": "gcc can compile", "depends on": [ "gcc is present" ] }
]
}

[/code]

 

And what will happen is this:

  • This tool will load plan
  • locate a unit called “gcc is present” first from units
  • check if units["gcc is present"].active is true
  • execute that unit’s target value as a subprocess
  • fail on either non-zero exit of target or if target‘s output does not match that unit’s outputvalue (checks for “present” in stdout to subprocess call of ./ubuntu/xenial/check_gcc_present.run
  • if it failed, execute "gcc is present".rectifier which is an executable that is intended to repair the failed condition of the test.
  • It will then loop between the rectifier and target to allow the creation of multi-pass repairs for obscure conditions (like race conditions)
  • When "gcc is present".target finally returns 0 and echos “present” to STDOUT, the tool will print the pretty systemD style output of [ PASS ] "gcc is present" and move to the next unit in plan, which is “gcc can compile”.  If units["gcc can compile"].active is true ….
  • It will then run the script that attempts a gcc compile based on units["gcc can compile"].target, which should return 0 and echo “can compile” if all goes well.

Obviously the design was intended to accomodate even very, very complex use cases with quite straightforward configuration this way and you can even do some metascripting with this (though I won’t and I don’t know why you would).

The solution language is currently C++ since that’s going to be a pretty safe dependency assumption on the builder’s system (and since not having G++ installed will cause the tests we’re using in our use case to fail anyway).

Great.  Why do I care?

This design will be able to build all of Foster.  And not just that — it’ll force the linear nature of these builds to occur in proper order with limited assumptions about environment and toolchains — and without the pain of trying to enforce a consistent scripting pattern through what was turning into dozens of shell scripts or introducing heavy dependencies, while still being abstract enough to allow the introduction of mixed-language solutions for the gnarly bits.  It accomodates profiles for any distribution, and is reusable for things outside of foster — if you check out SURRO Industries’ personal blog it’ll even be able to serve as the engine for the Conditional Response Daemon (CRD) that was put on the backburner for SURRO Linux and Surro Linux’s Foster initiative.

That’s all for you though.  Why do *I* care?

This will be useful for unit tests of one’s codebase by QA Engineers in software engineering environments, Infrastructure Engineers validating the requirements are met of a server build operation, Build and Deployment Engineers (I’m thinking DevOps engineers) to build and deploy or who are ensuring that their deployment job or build job generated all the files they want and that those files ended up on the right servers — the list goes on, and on and on for what this can be used for as long as I keep the design abstracted from Foster and SURRO as a generic tool.  The target executables can even include curses menus or flow control prompts for extremely elegant and potentially very complex operations that could be designed by one party or department to be executed by another party or department (process design in your org now has a tool that keeps it separated from process implementation — this means you, the entrepreneur can tell another entity or department how to do things at the high level without confusion over what exact change was made during your RCA during a failed rollback or any of the millions of situations that can come up in an SDLC).

I hope that others find this useful, and consider donating to the SURRO Linux Development Fund to sponsor more development hours as designs for these types of tools are innumerable in the talks coming out of the new official greybeards channel on irc://irc.oftc.net/#surro-linux.

So when do I get to use it?

As soon as:

  1. I complete it, which is required as a next step to the Foster design and build out.
  2. It has an official name.  There are candidates waiting on the rest of the Council of Greybeards:
  • Marbles
  • Chauffeur
  • Examplar
  • Blinkers
  • Pebbles
  • Mane

It is requested of any Greybeards reading this to submit your selection of a name to the channel over the next 24 hours.  The default name is Examplar unless we have consensus on another name.

Note:

After discussion with DJ the units will be able to be placed in multiple files (or one big file) to be read in a cascading fashion from a directory referenced in a config.ini (think puppet manifests or apache.conf).  The plan will be its own file and there will remain only one plan per run.  The path of the plan file will be in that same directory or be a supplied path as a command line argument.