What are Units? What are Plans?

So you’ve got Examplar compiled and you’re ready to start automating the world.

If you’re thinking “how do I configure this thing”, this article is for you.

Definitions

Units

A Unit is an automation definition, written in JSON in a UNIT FILE.  Deeper into Examplar’s internals, Units and Tasks have slightly different functions, but for the purposes of users, the terms can be used interchangeably.  A Task is a task to be performed in a Plan, and a Unit is its definition.  A Unit is a JSON object that has:

  • A name, which is an identifier for the Unit used by people.
  • A target, which is the path to the automation script performing the work.  This provides a clean linear path for huge chains of scripts to be executed in order and tracked on return for additional logic in chaining.
  • A rectifier, which is the path to the automation script to be executed if the target call fails.
  • A rectify attribute, which tells Examplar whether or not to execute the rectifier in the case of failure when executing the target.
  • An active attribute,which tells Examplar whether or not the Unit can be used in a Plan.  This gives Unit developers a way to tell Plan developers not to use the Unit.
  • A required attribute which tells Examplar whether or not the Plan can continue if the Unit fails.  If the rectify attribute is set to true, this attribute is checked after a rectifier failure.  If not, this is checked after target failure.  In either case, if the rectifier or target do not return successfully, Examplar will halt the execution of the Plan if this is turned on for the unit being executed.  Otherwise it simply moves to the next Unit being executed.

Tasks

A Task is an action item in a Plan, just like in real life.  In the context of Examplar, a Task is a Unit that has been loaded and incorporated into a Plan in an actionable state.  Inactive Units can not be loaded into a Plan and thus can never be a Task.  The primary difference between a Task and a Unit is that a Unit is not actionable — it’s just a definition — while a Task a consumable, actionable automation.

Suite

A Suite is not visible to the user and this is only for informational purposes.  A Suite is a collection of all available Unit definitions loaded from one or more UNIT FILES.  Just as a Unit is the definition for a Task, a Suite is a collection of Units that define the Task components of a Plan.

A Suite is consumed by a Plan during the conversion of Units to Tasks, though this is not visible to the user — it just simply helps to understand the kind of abstraction taking place in the conceptual model of Examplar.

Plan

A Plan is the glue of all the components of Examplar and is deceptively simple.  A Plan loads a Suite for its Task definitions (Units), but the Tasks to actually execute are specified in the PLAN FILE.  The Tasks are executed in the order specified in the PLAN FILE.

FILES

CONFIG FILE

This is the one config file that Examplar uses.  The default path it looks is /etc/Examplar/config.json.

A config file at the time of writing this specifies a single JSON object with 3 attributes:

  • The UNIT FILE path or a path to a directory containing unit files.
  • The PLAN FILE path.
  • The configuration VERSION.

UNIT FILE

The UNIT FILE is a specification of where the Units are defined.  All UNIT FILES in that directory will be amalgamated to generate the Suite.  These types of files must end in `*.units` for their filename.

PLAN FILE

The PLAN FILE is a specification of the order that Tasks are executed, and their dependencies upon each other.  Dependency implementation is a touchy matter that is pending implementation, so, mileage may vary until release.

Configuration VERSION

The configuration version is checked to ensure that the configuration is consumable by that version of Examplar.  This will pave the way for reverse compatibility if the project moves in that direction.

SURRO Linux: M1 -> M2

SURRO Linux is now moving from milestone 1 to milestone 2.

In this phase we will build a bootable ISO where the rest of the build occurs for a lightweight environment from which SURRO will be installed.

Examplar will be the driving tool for these builds and configurations.

-CP

Beta Release – Examplar 1.5b

Examplar is now in Beta at version 1.5b.

I’ve removed all hardcoded paths, implemented all relevant commandline switches, usage docs, and more.

At this point the tool is to be used for all SURRO/Foster development.

This version by all manners looks and behaves stable, but, I’m not comfortable releasing as a stable version until it’s built a bootable ISO so that it can have that guarantee behind the release announcement.

Feel free to download it and use it for any unrelated projects.

I expect in a future version that we’ll want to have a switch to disable syslog logging for absolutely barren environments but I do not think that will be necessary at this time.

Since we’ve moved to beta, I will be offering public RPMs and spec files.

Most work from here related to development on Examplar itself will revolve around improving documentation on how to use it as this is a new tool in the Linux space so I expect alot of confusion at first.  It’s deceptively simple.

In the meantime the source can be downloaded here if you’d like to compile it yourself:

https://source.surroindustries.com/Cycle0/Examplar/repository/archive.tar.gz?ref=Examplar-1.5b

It is strongly recommended to package it before installing it due to the frequency with which we’ve been updating, but it really doesn’t need to be “installed”.

Here’s a script that will get you up and running. Make sure you have cmake installed on your system to be able to compile it. Cmake is a great tool for building C/C++ projects and more. You will also need a C++ compiler. I use G++.


mkdir ./Examplar
cd ./Examplar
wget https://source.surroindustries.com/Cycle0/Examplar/repository/archive.tar.gz?ref=Examplar-1.5b -O examplar-1.5b.tar.gz
tar xvf examplar-1.5b.tar.gz
cd Examplar-Examplar-1.5b-*
cmake .
make
./examplar -h

Thanks to all the wonderful people who have provided input on the direction of this project, and I look forward to what we do with this — and I know the SURRO Greybeards have been very patient while I tried to make the time to do this.  I can not say that making the time did not have consequences, but we made it and that’s what matters.

Examplar 1.3a Released

After a couple nights of grinding, we’ve got a major, major improvement from the v1.0a candidate.  Examplar v1.3a is released.

This version has major bug fixes and feature additions so it’s recommended to move to this version from 1.0a.

  • This version repairs a rather serious logic bug in the task execution flow.
  • There is also the new feature of now being able to split Units across multiple files in the same directory and pointing the config to the directory of .units files, which will give some more flexibility in unit definitions.
  • It now logs to syslog.
  • Reworked some of the exception layer to give more details to the user during failures.

I’m rather pleased with this version although I would like someone to come and clean up the code a little bit as there are a couple parts of it that are functional but ugly.

You can use whatever you’d like to monitor syslog — in my case I use systemd on my test machine, so I just do:

journalctl -t Examplar -f

And it will hold onto the syslog in realtime while I execute Examplar, giving me realtime logs.  It still dumps to STDOUT and STDERR. This should be relatively universal, but, like I said, use whatever you’d like to monitor syslog during Examplar runs.

At this point all original features are accounted for and it’s ready for some hands-on testing by building out Foster — but I think I’d like to hold this in Alpha before moving to a Beta or stable release.

-CP

Examplar v1.0a — Initial Alpha is Released

We made it to alpha.  This is where the fun starts.

In Alpha status, we’ve got all the functionality of the tool implemented but will still be adding in some polishing as we encounter any issues using it to build Foster.  There will be minor features added while in Alpha status:

  • Logging to file
  • Loading a directory of UNIT FILES instead of just one UNIT FILE.
  • Commandline switch processing.

Examplar is released under AGPL and requires only a C++ compiler and cmake to build, with zero dependencies.

Examplar: A word on Tasks, Plans, and Dependencies

Why Examplar Works

So, this has come up a bit, and I wanted to have something written up somewhere that explains how these parts all fit together.

The Solved Problem

During the build of the initial environment, there are alot of considerations and hurdles that come up depending on what distribution you are on, and what you want built or automated.

Needed a Boostrap With No Runtime

In our case, we’re going to be compiling into a chroot with basically nothing but a compiler, glibc, a kernel, libstdc++ and your fingers.  We needed a tool that could, in a distro-independent manner, start automating first the initial creation of that chroot environment, then do the compilations necessary to populate it with those things, and then still have enough runtime to bootstrap into itself to continue the rest of the setup of that environment in a way that accommodated more and more complex tools being executed which depend on runtimes provided by previous automated tasks that had been performed by the tool — in a chain of successive scripts that have no defined complexity limit.

We also needed a way to run environment checks and audits on one track of scripting and work to perform repairs when those checks failed, or alternative scripts to run when automations failed — and current automation tools that are distro independent require massive runtimes that just are not realistic to build by hand in a chroot (ansible, chef, puppet).

Coding Task diversity and Order for a Distro Build had Unmanageable Complexity

In addition to this, it became clear that the nature of automations needed for building these environments was so diverse that abstracting them into units for planning and quality checks was just a monstrous task and we needed a tool that forced us to abstract tasks into work units before planning what needed to be done.  This would also allow us to version control the plan for the work needing performed itself as opposed to managing that process through very talented humans — and we needed the plan to be decoupled from the tasks needing performed so that the abstraction necessary to create a distribution could be focused on without too much low level on implementation.  We also needed the tool to have some level of recursion possible.

We needed the Plan as well as the Coding Tasks manageable by many Contributors

I realized that most of the guides out there were for single users to build single user systems.  I needed a way to allow a large contributor audience to contribute not just to the tasks that were being coded but to the plan that defined the execution order, and what would be executed, in a way that allowed profiles in the build system.

We needed something with a Flexible Use case

We needed a tool that could be repurposed outside of our initial use case to create a toolchain that was reusable to save time since we have a small crew that could become large if we were to beat the odds and create a successful project.

We wanted to introduce a new tool of this type to the Linux world, and we wanted it to be dead simple.

Due to many of the reasons mentioned above it was important to me at least to provide a tool to the Linux world that did not have such heavy runtime dependencies and domain-specific knowledge required to use, while still being able to accomplish most of the net capabilities these tools provide.  We wanted to provide a tool for the systems-heavy DevOps admin and their development-heavy environments.

We wanted modern configuration style

And JSON was it.

That’s why work started on Examplar.

A Pivotal Moment: The Design Evolution

However, during the design process I realized that I could make this so generic that it could do everything without requiring a complex specialized toolchain as long as it was only providing “workflow” capabilities for already existing tools and resources on the system, allowing the automation to grow with system capability.

This would save the eye-crossing complexity of massive makefiles or cmake configurations, while providing a tool that could bootstrap into itself and continue whenever the work was moving to a new environment (a situation that will happen a few times during the builds from “your current distro” to “your booted SURRO Linux Environment”.

 

* * *

 

How SURRO’s Examplar Works

First, some definitions:

Units

A Unit is an automation definition, written in JSON in a UNIT FILE.  Deeper into Examplar’s internals, Units and Tasks have slightly different functions, but for the purposes of users, the terms can be used interchangeably.  A Task is a task to be performed in a Plan, and a Unit is its definition.  A Unit is a JSON object that has:

  • A name, which is an identifier for the Unit used by people.
  • A target, which is the path to the automation script performing the work.  This provides a clean linear path for huge chains of scripts to be executed in order and tracked on return for additional logic in chaining.
  • A rectifier, which is the path to the automation script to be executed if the target call fails.
  • A rectify attribute, which tells Examplar whether or not to execute the rectifier in the case of failure when executing the target.
  • An active attribute,which tells Examplar whether or not the Unit can be used in a Plan.  This gives Unit developers a way to tell Plan developers not to use the Unit.
  • A required attribute which tells Examplar whether or not the Plan can continue if the Unit fails.  If the rectify attribute is set to true, this attribute is checked after a rectifier failure.  If not, this is checked after target failure.  In either case, if the rectifier or target do not return successfully, Examplar will halt the execution of the Plan if this is turned on for the unit being executed.  Otherwise it simply moves to the next Unit being executed.
Task

A Task is an action item in a Plan, just like in real life.  In the context of Examplar, a Task is a Unit that has been loaded and incorporated into a Plan in an actionable state.  Inactive Units can not be loaded into a Plan and thus can never be a Task.  The primary difference between a Task and a Unit is that a Unit is not actionable — it’s just a definition — while a Task a consumable, actionable automation.

Suite

A Suite is not visible to the user and this is only for informational purposes.  A Suite is a collection of all available Unit definitions loaded from one or more UNIT FILES.  Just as a Unit is the definition for a Task, a Suite is a collection of Units that define the Task components of a Plan.

A Suite is consumed by a Plan during the conversion of Units to Tasks, though this is not visible to the user — it just simply helps to understand the kind of abstraction taking place in the conceptual model of Examplar.

Plan

A Plan is the glue of all the components of Examplar and is deceptively simple.  A Plan loads a Suite for its Task definitions (Units), but the Tasks to actually execute are specified in the PLAN FILE.  The Tasks are executed in the order specified in the PLAN FILE.

FILES
CONFIG FILE

A config file at the time of writing this specifies a single JSON object with 3 attributes:

  • The UNIT FILE path.
  • The PLAN FILE path.
  • The configuration VERSION.
UNIT FILE

The UNIT FILE is a specification of where the Units are defined.  In future versions of Examplar this will actually be a directory and all UNIT FILES in that directory will be amalgamated to generate the Suite.  Currently this is only one file.

PLAN FILE

The PLAN FILE is a specification of the order that Tasks are executed, and their dependencies upon each other.  Dependency implementation is a touchy matter that is pending implementation, so, mileage may vary until release.

Configuration VERSION

The configuration version is checked to ensure that the configuration is consumable by that version of Examplar.  This will pave the way for reverse compatibility if the project moves in that direction.

 

* * *

 

What Examplar Looks Like

  1. Write a script that does a thing, or checks a thing, or builds a thing, even calls a thing that does a thing.
  2. Write a script that fixes conditions that you expect to prevent whatever you called previously from completing successfully.
  3. Create a UNIT FILE defining those two scripts.
  4. Create a PLAN FILE referencing the UNIT you created.
  5. Point your configuration file at the UNIT FILE and PLAN FILE.
  6. Run Examplar.  Eat Waffles if Necessary.

Experienced scripters will notice two things right off the bat.  For (2) you cant know all of the conditions, and if you could you can’t have a way to know which condition occurred.  This is accounted for in the design!

Write checks for each of your conditions as Unit targets.  Write heals for those conditions as Unit Rectifiers in their corresponding Unit.  Enable the Unit Rectify pattern.

Only you can predict the things that will go wrong, so you’ll want to approach it iteratively — which where previously was not possible is now gaurdrailed in a system that forced you to approach it as the creation of “code units” that “do a thing”, adding on as you encounter new ones needed.  This tool not only is possible to use in Agile environments — but it facilitates the Agile workflow in coding environments — and the fact that is has ZERO dependencies means it can run almost anywhere, including Windows systems in future versions, or embedded systems, or inside of a cloud container.

Can your current tool do that with this kind of flexibility or are you muscling Agile into your workflow by team willpower?

Examplar is in Pre-Release

I am pleased to announce that Examplar subprocess execution has been reworked and the product now fully supports curses dialogs, ssh clients, and much more — giving you full I/O with any processes that are executed by Examplar.

There is one more feature to implement before the product can hit an alpha release, which is dependency management during the loading/execution of units.  This is mostly just managing the data model which is pretty fleshed out by this point due to the decision to bind design to source.

So excited about this.

Examplar Execution Flow is Mapped

Hey all.

Well, it’s certainly been a bit since the last update, but, I have diagrammed the execution flow for the Task class, finally, and fleshed out a (mostly) working implementation of the subprocess execution class consumed during that process.

Note:  Green is cheese for the mice.

Note 2: Not really.  Those are user-configurable items in the datastructure.

Decisions

  • At this point implementation is almost complete.  I’ve gone ahead and made an executive decision to include the design documents with the source in the Examplar source repository.
  • Any pull requests that do not reflect their changes in the design (where applicable) will be summarily rejected.
  • I have removed output checks because it’s unnecessary and complex.  If users desire that they’ll need to implement it in the code that Examplar calls.  It is designed to accommodate that, so there is no loss of functionality there.

TODO

  • There is a minor issue with I/O redirection that I’m working on (select/poll madness needed there).
  • It doesn’t support curses dialogs very well at all yet, I assume there’s additional communication going on regarding cursor position, refresh, and window size that needs to be accounted for.  I will not consider the product releaseable until curses dialogs are fully supported.  This also applies to interactive utilities like an ssh client.  This was actually partially working until an issue with exit code of child process capture was repaired that ended up breaking it in the process.  Everything comes back to the same class for subprocess execution.
  • Dependency check implementation.

Summary of Current State

Examplar is actually a functional product at this point and can be used for simple styles of scripting (i.e. tools that have output but dont require input to STDIN) but not for interactive processes.  The difference between a full, robust software suite and current state is a matter of fixing I/O redirection in a single class, which is in progress.

That should be relatively soon, but in the meantime you should be able to use it to build with your favorite python scripts, bash scripts, ruby scripts, C tools, or whatever and even begin building out things with it as long as those things don’t require keyboard entry from the user.

This is, of course, at the time of writing this, so for posterity’s sake the current source is here.

The repos are kind of the wild west until first release.

Musings

As I’m working through the test scenarios and use cases it’s occured to me how badly I needed this tool in some environments I’ve worked in and some builds I’ve done in the past — yet it’s so simple when you zoom out.  I have a feeling this has the potential for a larger impact than I initially intended, in terms of the quality of the automations that will be produced in this as opposed to just scripting or programming something out without rails in place.

All in all, it’s coming together to meet its originally intended purpose:  A scalable complex (or simple) automation or testing framework.  It could even be used for orchestration or configuration management.

I do have to laugh at the fact that the last 5% of the code took 90% of the project time.

New Staff and More

We’re in the process of onboarding a couple of new developers that heard about SURRO at a convention recently.  Stay tuned.

As for current priorities, I’ve been distracted with a non-surro related project and work.  Due to not having much developer numbers to pull from development is slow.

It got brought up that we should have a formal spec and design document.  I can’t disagree with that.  To follow.

Freenode Primary Group Approval

I forgot to mention, a minor announcement is due.

Freenode has officially recognized the SURRO Linux project as a primary topic group.  It took alot of work to get it to go through.  Thanks to all those who did this work as it really was quite time consuming.

Our primary channel is irc.freenode.org/#surro-linux

-CP