Revisiting Foster

Foster’s main purpose is to be a proving ground for the components I’m building that will ultimately generate the SURRO Linux installer ISO.

I’ve been busy with work so it’s not been very active. I’m slowly revisiting my work with it. As most of you who do work with software can probably relate to, when you look at older code, even your own, sometimes it’s a little different than you remember it and a little more involved.

I’ve made this way harder than it needs to be.

So, I’ll go over some of the high level about what this is, how it works, and start a little cleaner as I wrap my head around how this should work.

Environment Preparation

It starts with a makefile.

The makefile in the Foster project is basically a wrapper for a set of shell scripts: one for make, and one for clean. As you might guess, make builds the Foster project. clean removes all the artifacts.


In the build script, you’ll see it’s mostly docker commands: I wrap a docker container around the whole build process.

The build script is just a kickoff point for docker pulling an image and spinning it up as a container. That container has a shared volume, which is the directory where the examplar project files are, as well as other files necessary to bootstrap into the environment I’m creating in the container.


Examplar is, in this context, a build tool that is driving the whole build process.  It is also developed by me.

The docker image will then download the source code for Examplar, install any C++ compilation tools needed, such as the gcc compiler, and then build Examplar.

The image will then execute that freshly compiled Examplar instance against the Examplar project files from the shared volume mentioned earlier. The mechanism for this execution is the script which the docker container is told to run when it spins up. This script is located on the bootstrap shared volume as well. By the time this script is executed it means the instructions in the docker file have completed and the build is out of the environment preparation phase.

The script will execute the the examplar instance and pass on the return code from that examplar instance to its calling process to indicate success or failure of the build process.

The Examplar Project (Cycle 0)

Now that we’re in the part controlled by the Examplar project, examplar will execute units defined in its project files on the bootstrap shared volume. The order of execution of those units are defined in the plan file.

Environment Checks

It starts with environment checks and some primitive rectification attempts when it finds issues. When I say “environment” in this context I’m referring to the operating system of the running container. This includes present packages, their features and configuration, shell variables, and more.

Source Downloads

The source for the components to be compiled to create the new operating system are downloaded into a source directory.


Examplar will then create a directory at the root of the container OS filesystem. That directory will be the SYSROOT of the new operating system it is compiling, and will eventually be a CHROOT used to compile the rest of the pieces needed for a fully bootable system.


Once this directory exists, examplar will begin compiling a cross-compiler from raw source to avoid sanitation issues created by upstream patches. That cross-compiler will be configured to output to the chroot. This step is problematic and is a work in progress.

The SYSROOT / Chroot Population

Examplar then begins compiling the bare, core parts of the system, such as the kernel and its dependencies, again dumping output into the CHROOT/SYSROOT directory. Once enough packages are present to compile librpm and rpm package management pieces, examplar will chroot into the newly populated sysroot and begin compiling pieces from the inside out. From there RPM packages are used.

Spring Cleaning and More

Foster Git Repo Given a Second Chance

Some time last year I accidentally pushed some source tarballs that created huge objects in the git repo for Foster.

Our newest contributor repaired it but we lost revision history in the process.  Since we’re still in pre-release and working out design still I’ve gone ahead and approved – we won’t miss those early pushes of just trying things to see if they work for some of the early R&D.

I’ve archived the original with the bloated repo objects in its revision history as those tend to be rather important later sometimes, but, we really don’t need them.

Foster is now Makefile driven and Completely Dockerized

Wrapping up the process in a docker container was pretty important for this project as we needed the build host to be abstracted away from the user’s operating system without requiring much setup from the user.

Using the dev’s host as a build host I don’t think is very safe because of all the things that can go wrong.  This was exemplified by a destroyed host after a typo in a log cleanup script last year.

We made it safe by wrapping the examplar build up in a docker container and image build process and orchestrating all of that with a parent makefile.

The usage details are in the Foster readme.

The only prior setup or dependencies Foster has now on the build host is the presence of docker and make on the system.  It should compile on Windows, Linux, or Apple operating systems now as a pleasant side effect.

We have a simple roadmap established for Foster

Foster will use the Fedora base docker image as its build host.  I’ve got an updated snapshot of Fedora so that folks don’t have to wait on crazy download times.  I’ve also implemented source caching for the libraries it builds into the chroot.  This will still require users to download it once by design.

Once the first bootable Foster ISO is created we’ll dockerize that image and use it as the base, and then go back and edit Foster to run on itself to remove our dependency on the Fedora base image.

Curious about Surro Linux?

This post is intended to provide a rundown of exactly what we’re doing and why, to comprehensively include all of our updates and ambitions as they currently are, as there have been some direction changes that are notably significant, and, it seemed like it would be good to create a view for people to see if they feel like jumping in to help out.

This project has several goals and products associated with it, all related and towards a single end.


SURRO Linux:  A Basic Product

On the outside, we’re building a new Linux distribution — a very clean one intended to appeal to engineering and scientific environments.

The distribution is called SURRO Linux.


It’s Open-Standards Centric

During a time when critical standards that made Linux the engineering marvel that it is are no longer being adhered to by all major distributions, SURRO Linux will ultimately be LSB 5 certified and FHS 3.0 compliant unless a newer version of either of these standards is out before our first release.

There will never be a question about where a file should exist on the system.

Binary compatibility will be limited only by the LSB 5.0 standard.


It’s Not Based On Another Distro

SURRO Linux will not be based on another Linux distribution and will not be a child distribution.


It’s Raw and Fresh

There will be no copying of precompiled binaries from an ancestor distribution for its creation.  The entire distribution is compiled from raw source code for all components before being distributed as a binary package centered distribution.


It’s not just a Distro, Though

Along the way there are several tools that accomplish the creation of the distro for us that we’re building to get there.  These tools are designed to be either generic in purpose or to help others develop better products.

Aside from the technical merits:  SURRO Linux is not controlled by corporations, or investor interests — but technical needs.


Build Your Own Distribution: Foster

We’ve brought the Infrastructure as Code (IaC) concept to the OS Layer in the OSI model.

Foster is an intermediary distribution that can be easily modified to create your own Linux distribution.  We use an internally created build system that serves as configuration for the generated product, which means that the entire build process for the whole distribution, from sources, is automated and version controlled.

This will mean that the problems that we’ll be solving to get a bootable environment from nothing but source will be shared with the community, allowing others who might do better or have more resources to do so in a version-controlled, auditable manner.


We’re also targeting Container Crowds

This is also important to cloud and container solution providers in creating very minimal or controlled images with a great deal of granularity about what gets put on the image, auditability, change control and more for their base images — no more golden images that get cluttered over time — a version controlled image configuration is the apex of controlled container image management.

While we will be using Foster to create the SURRO Linux installer ISO, you could build your own — ISO or docker image.


It’s Host OS Agnostic

We use a docker container to wrap the build process for host/build OS abstraction.  Once the First bootable Foster image is reached we will be using it as our source image for that container and rewriting Foster to compile on itself.  This is for the purpose of providing the clean image to the community and to remove dependencies in our automations on third party providers.

The end result is that Foster should be able to compile on Windows, BSD, Linux, or even OSX in a very clean way.


Build and Automate Anything: Examplar

Examplar is an automation orchestration and rudimentary configuration management tool, or a build system, depending on how you look at it.  During the process of compiling Foster, Examplar is the tool that gets executed to kick off and orchestrate all of the build scripts and automations in place.  It is configured entirely in JSON allowing levels of complexity able to support automation of the scale required to build an entire operating system from source.

Examplar is its own product, generic in purpose, written entirely in C++ with no external system library dependencies, meaning it can compile and run just about anywhere on just about any operating system.

Examplar is also evolving as we build Foster with it, with new features being added frequently to accomplish our goals.



Once Foster is released we will be using Foster as the base image for the SURRO Linux installer ISO — we will be using Examplar to drive the configuration and installation of a new SURRO Linux system.

The first release of SURRO Linux will use RPM as a package manager, systemd as its init system, and will have public binary package repositories.  It will be curses-driven for installation.

Later releases will have menu options to select preferred components, such as bootloader, potentially the init system, package manager, desktop environment, and more.  Later versions are also intended to accept “profiles” that select these options in an unattended manner for CI systems.

The project is slow-moving, mostly due to resources being needed and research time.  With an increased headcount we could move faster.  With funding we could move faster.  The SURRO Linux project is currently entirely funded by Chris Punches in conjunction with donations to the project GoFundMe.  Join or contribute today.

Breakthrough: Host Build Environment Abstraction

I’m not dead, really!

Long break after some contract work.

I have found a way to successfully abstract away the host’s build environment, allowing us to move a little faster.

Rather than be informative and posting the link to the updates, I’m going to leave this announcement as is, mostly to give me time to integrate everything into the new build format and make all those minor decisions needed to get it all to work together.

This is going to be awesome.  If this works as well as it seems to so far, when it comes time to build Foster, you could even do it on a windows system with no difference in result.  Not that we’re going to try for that, but, it is expected to be possible.


New Mailing List and More

I’ve decided to go ahead and prop up the mailing list as a new communication path.

I’m considering removing the bug tracker as while it was really productive for a while we just don’t have enough people until we get to release status.  Not decided yet.

I’m going to be doing quick bursts on the Foster build.  It will try to be distro independent for the most part but I’m on a Fedora machine — so in some places stuff will be specific to Fedora for the builds — it’s encouraged for people to go behind me and make stuff distro independent so I can just focus on getting us to a bootable state.

Mailing list url:!forum/surro-linux-greybeards


We may be losing some folks on the core dev team due to work/life balance needs — understandable and regrettable — contribute where you can.  It’s going to be the wild west before Alpha release.  That’s the primary goal first.  We’ll make it pretty when we get there.  Slightly less planned than we’d hoped, but we’ve gotta move forward on this project.


Current State of Milestone 2 out of 5

I haven’t touched it since my last post.

We’re currently on Milestone 2.

I’ve been working on a closed source project called T-ORCH, an application-layer orchestration solution designed for large scale enterprise environments, as an effort to fund the SURRO Linux project.

Once sold I hope to work on SURRO full time after we get to a certain point, but it’s a long way there.

This largely stems from the fact that I totally suck at fund raising.  We actually have a fundraiser page where you can donate towards surro linux development at:

While what we really need is more people, every bit helps keep the surro linux project going.

I am concerned about how involved the design and development time is on T-ORCH and how it’s pulling time away from SURRO, so, from here forward I’ll be devoting a 5/2 split of time between T-ORCH and SURRO.

The main delay there was the design and implementation of IRCTHULU.  IRCThulu is a project of intentionally crafted traditional SOA components so that as T-ORCH is implemented it will have integration concerns in mind to make T-ORCH as well built as possible — and to have IRCThulu be a showcase for doing just that.  With IRCThulu created it’s now time to build out T-ORCH and then integrate it into IRCThulu components.

I’ll be spending Saturdays and Sundays on SURRO and Mondays through Fridays on T-ORCH until T-Orch is in release status and IRCThulu is integrated into an instance of it.

Contact Info for Chris Punches

Some very interesting people expressed interest in the SURRO Linux project.

Anyone curious on joining the team of greybeards or general inquiries feel free to reach out to me personally at

I don’t think it’s any secret any more that this is one of my projects.

-Chris Punches

Examplar Minor Release to 1.6 Beta

As mentioned previously, a new feature was needed to be added to Examplar to allow much greater simplicity in the Foster build to make its content more distributable and I could definitely see that affecting usage due to the removal of the need to hardcode paths in the various files.

So, we’ve added it.  Instructions are in the repo.

The source code for the 1.6b version is here.

The source code for Foster is here.

I believe we will probably have much slower updates from here.

Foster Initial Run

Right off the bat I noticed that there are some features that will make the foster builds quite a bit easier.

The most prominent feature I’d like to see added next will be a config file that sets any enviornment variables the user would like any executables that get called to have in their environment.

Another is per-Task STDOUT/STDERR logging as it’s currently up to the user to implement this in their scripts.

Thoughts for future versions.  In our case we’ll need some variables set across several tasks or heals that will be executed, so, I’ll be sourcing a common file that sets these.

Another is a switch for setting the current working directory for execution context to prevent having to hardcode system-dependent paths all over the place in the units.  I will make sure that gets put in place next.