[L2Ork-dev] CI and runners

[Sam Thursfield]

Hi,

> Here's my conceptual starting point for a runner:
>
> 1.  I download some docker image for, say, Ubuntu 20.04.
> 2.  I go ahead and install all the necessary dependencies for building
>     Purr Data on that image, plus doing update/upgrade on it.
>
> 3.  I set it up through gitlab-runner using the docker executor.
> 4.  Either manually or through a cron job, I set up some periodic
>     update/upgrade for that image.

This sounds promising!

Steps 1 and 2 are useful for preparing and exploring what's possible. But the best thing is to automate the image builds from the beginning.

A few tips for those who aren't already deep in the Docker world...

  * Docker has its own build system for building Docker images, the command is `docker build` and the input is called a 'Dockerfile'. The format is fairly straightforward and documented here:
  * One non-intuitive thing about `docker build` is that each instruction in the Dockerfile creates a new layer in the image -- so we try to keep them as short as possible.
  * Many people now use Podman instead of Docker. It's 99% compatible, but has an improved design (no daemon running as 'root'). The equivalent of `docker build` in this system is `buildah`.
  * Every project on a Gitlab instance can have its own Docker repository.

With this in mind the best approach is to have a GitLab project that builds and hosts the images as part of its CI. I set one up on Gitlab.com (I wasn't 100% sure if git.purrdata.org has the Docker registry enabled, although maybe it's on by default) which I was using for testing.

So far it just has the 'stretch' image, which you can build like this:

    git clone https://gitlab.com/samthursfield/purr-data-oci-images
    docker build --file ./purr-data-oci-images/debian/stretch

As usual, you can substitute 'podman' for 'docker' here depending what's available.

Adding a new target involves:

  * adding a new Dockerfile -- for example ./debian/ubuntu-20.04
    or ./fedora/latest. Update the FROM at the top.
  * creating or modifying the relevant prepare.sh script (debian and ubuntu can probably use the same script, while other distros need a separate one)
  * updating the .gitlab-ci.yml file -- the existing 'debian stretch:' block serves as a model for what's needed

> 5.  I leverage Sam's changes which cache the nw.js binary on the host.

:)

>     Sam-- is this possible? It seems that a lot of CI flows have the
>     bandwidth and CPU power to just fetch a new image and prepare it each
>     time CI gets triggered, but that seems like it would eat substantial
>     time and put us back where we started.

Yeah, I don't like that approach either ... I think having a repo that builds the images once is well worth the effort.

Looking forwards to seeing the improvements!

Sam