Examplar: A word on Tasks, Plans, and Dependencies

Why Examplar Works

So, this has come up a bit, and I wanted to have something written up somewhere that explains how these parts all fit together.

The Solved Problem

During the build of the initial environment, there are alot of considerations and hurdles that come up depending on what distribution you are on, and what you want built or automated.

Needed a Boostrap With No Runtime

In our case, we’re going to be compiling into a chroot with basically nothing but a compiler, glibc, a kernel, libstdc++ and your fingers.  We needed a tool that could, in a distro-independent manner, start automating first the initial creation of that chroot environment, then do the compilations necessary to populate it with those things, and then still have enough runtime to bootstrap into itself to continue the rest of the setup of that environment in a way that accommodated more and more complex tools being executed which depend on runtimes provided by previous automated tasks that had been performed by the tool — in a chain of successive scripts that have no defined complexity limit.

We also needed a way to run environment checks and audits on one track of scripting and work to perform repairs when those checks failed, or alternative scripts to run when automations failed — and current automation tools that are distro independent require massive runtimes that just are not realistic to build by hand in a chroot (ansible, chef, puppet).

Coding Task diversity and Order for a Distro Build had Unmanageable Complexity

In addition to this, it became clear that the nature of automations needed for building these environments was so diverse that abstracting them into units for planning and quality checks was just a monstrous task and we needed a tool that forced us to abstract tasks into work units before planning what needed to be done.  This would also allow us to version control the plan for the work needing performed itself as opposed to managing that process through very talented humans — and we needed the plan to be decoupled from the tasks needing performed so that the abstraction necessary to create a distribution could be focused on without too much low level on implementation.  We also needed the tool to have some level of recursion possible.

We needed the Plan as well as the Coding Tasks manageable by many Contributors

I realized that most of the guides out there were for single users to build single user systems.  I needed a way to allow a large contributor audience to contribute not just to the tasks that were being coded but to the plan that defined the execution order, and what would be executed, in a way that allowed profiles in the build system.

We needed something with a Flexible Use case

We needed a tool that could be repurposed outside of our initial use case to create a toolchain that was reusable to save time since we have a small crew that could become large if we were to beat the odds and create a successful project.

We wanted to introduce a new tool of this type to the Linux world, and we wanted it to be dead simple.

Due to many of the reasons mentioned above it was important to me at least to provide a tool to the Linux world that did not have such heavy runtime dependencies and domain-specific knowledge required to use, while still being able to accomplish most of the net capabilities these tools provide.  We wanted to provide a tool for the systems-heavy DevOps admin and their development-heavy environments.

We wanted modern configuration style

And JSON was it.

That’s why work started on Examplar.

A Pivotal Moment: The Design Evolution

However, during the design process I realized that I could make this so generic that it could do everything without requiring a complex specialized toolchain as long as it was only providing “workflow” capabilities for already existing tools and resources on the system, allowing the automation to grow with system capability.

This would save the eye-crossing complexity of massive makefiles or cmake configurations, while providing a tool that could bootstrap into itself and continue whenever the work was moving to a new environment (a situation that will happen a few times during the builds from “your current distro” to “your booted SURRO Linux Environment”.

 

* * *

 

How SURRO’s Examplar Works

First, some definitions:

Units

A Unit is an automation definition, written in JSON in a UNIT FILE.  Deeper into Examplar’s internals, Units and Tasks have slightly different functions, but for the purposes of users, the terms can be used interchangeably.  A Task is a task to be performed in a Plan, and a Unit is its definition.  A Unit is a JSON object that has:

  • A name, which is an identifier for the Unit used by people.
  • A target, which is the path to the automation script performing the work.  This provides a clean linear path for huge chains of scripts to be executed in order and tracked on return for additional logic in chaining.
  • A rectifier, which is the path to the automation script to be executed if the target call fails.
  • A rectify attribute, which tells Examplar whether or not to execute the rectifier in the case of failure when executing the target.
  • An active attribute,which tells Examplar whether or not the Unit can be used in a Plan.  This gives Unit developers a way to tell Plan developers not to use the Unit.
  • A required attribute which tells Examplar whether or not the Plan can continue if the Unit fails.  If the rectify attribute is set to true, this attribute is checked after a rectifier failure.  If not, this is checked after target failure.  In either case, if the rectifier or target do not return successfully, Examplar will halt the execution of the Plan if this is turned on for the unit being executed.  Otherwise it simply moves to the next Unit being executed.
Task

A Task is an action item in a Plan, just like in real life.  In the context of Examplar, a Task is a Unit that has been loaded and incorporated into a Plan in an actionable state.  Inactive Units can not be loaded into a Plan and thus can never be a Task.  The primary difference between a Task and a Unit is that a Unit is not actionable — it’s just a definition — while a Task a consumable, actionable automation.

Suite

A Suite is not visible to the user and this is only for informational purposes.  A Suite is a collection of all available Unit definitions loaded from one or more UNIT FILES.  Just as a Unit is the definition for a Task, a Suite is a collection of Units that define the Task components of a Plan.

A Suite is consumed by a Plan during the conversion of Units to Tasks, though this is not visible to the user — it just simply helps to understand the kind of abstraction taking place in the conceptual model of Examplar.

Plan

A Plan is the glue of all the components of Examplar and is deceptively simple.  A Plan loads a Suite for its Task definitions (Units), but the Tasks to actually execute are specified in the PLAN FILE.  The Tasks are executed in the order specified in the PLAN FILE.

FILES
CONFIG FILE

A config file at the time of writing this specifies a single JSON object with 3 attributes:

  • The UNIT FILE path.
  • The PLAN FILE path.
  • The configuration VERSION.
UNIT FILE

The UNIT FILE is a specification of where the Units are defined.  In future versions of Examplar this will actually be a directory and all UNIT FILES in that directory will be amalgamated to generate the Suite.  Currently this is only one file.

PLAN FILE

The PLAN FILE is a specification of the order that Tasks are executed, and their dependencies upon each other.  Dependency implementation is a touchy matter that is pending implementation, so, mileage may vary until release.

Configuration VERSION

The configuration version is checked to ensure that the configuration is consumable by that version of Examplar.  This will pave the way for reverse compatibility if the project moves in that direction.

 

* * *

 

What Examplar Looks Like

  1. Write a script that does a thing, or checks a thing, or builds a thing, even calls a thing that does a thing.
  2. Write a script that fixes conditions that you expect to prevent whatever you called previously from completing successfully.
  3. Create a UNIT FILE defining those two scripts.
  4. Create a PLAN FILE referencing the UNIT you created.
  5. Point your configuration file at the UNIT FILE and PLAN FILE.
  6. Run Examplar.  Eat Waffles if Necessary.

Experienced scripters will notice two things right off the bat.  For (2) you cant know all of the conditions, and if you could you can’t have a way to know which condition occurred.  This is accounted for in the design!

Write checks for each of your conditions as Unit targets.  Write heals for those conditions as Unit Rectifiers in their corresponding Unit.  Enable the Unit Rectify pattern.

Only you can predict the things that will go wrong, so you’ll want to approach it iteratively — which where previously was not possible is now gaurdrailed in a system that forced you to approach it as the creation of “code units” that “do a thing”, adding on as you encounter new ones needed.  This tool not only is possible to use in Agile environments — but it facilitates the Agile workflow in coding environments — and the fact that is has ZERO dependencies means it can run almost anywhere, including Windows systems in future versions, or embedded systems, or inside of a cloud container.

Can your current tool do that with this kind of flexibility or are you muscling Agile into your workflow by team willpower?