[L2Ork-dev] CI and runners

[Jonathan Wilkes]

On Thu, Oct 8, 2020 at 5:25 AM Sam Thursfield <samthursfield at pm.me> wrote:
>
> Hi,
>
> > Here's my conceptual starting point for a runner:
> >
> > 1.  I download some docker image for, say, Ubuntu 20.04.
> > 2.  I go ahead and install all the necessary dependencies for building
> >     Purr Data on that image, plus doing update/upgrade on it.
> >
> > 3.  I set it up through gitlab-runner using the docker executor.
> > 4.  Either manually or through a cron job, I set up some periodic
> >     update/upgrade for that image.
>
> This sounds promising!
>
> Steps 1 and 2 are useful for preparing and exploring what's possible. But the best thing is to automate the image builds from the beginning.
>
> A few tips for those who aren't already deep in the Docker world...
>
>   * Docker has its own build system for building Docker images, the command is `docker build` and the input is called a 'Dockerfile'. The format is fairly straightforward and documented here:
>   * One non-intuitive thing about `docker build` is that each instruction in the Dockerfile creates a new layer in the image -- so we try to keep them as short as possible.
>   * Many people now use Podman instead of Docker. It's 99% compatible, but has an improved design (no daemon running as 'root'). The equivalent of `docker build` in this system is `buildah`.
>   * Every project on a Gitlab instance can have its own Docker repository.
>
> With this in mind the best approach is to have a GitLab project that builds and hosts the images as part of its CI. I set one up on Gitlab.com (I wasn't 100% sure if git.purrdata.org has the Docker registry enabled, although maybe it's on by default) which I was using for testing.
>
> So far it just has the 'stretch' image, which you can build like this:
>
>     git clone https://gitlab.com/samthursfield/purr-data-oci-images
>     docker build --file ./purr-data-oci-images/debian/stretch
>
> As usual, you can substitute 'podman' for 'docker' here depending what's available.
>
> Adding a new target involves:
>
>   * adding a new Dockerfile -- for example ./debian/ubuntu-20.04
>     or ./fedora/latest. Update the FROM at the top.
>   * creating or modifying the relevant prepare.sh script (debian and ubuntu can probably use the same script, while other distros need a separate one)
>   * updating the .gitlab-ci.yml file -- the existing 'debian stretch:' block serves as a model for what's needed
>
> > 5.  I leverage Sam's changes which cache the nw.js binary on the host.
>
> :)
>
> >     Sam-- is this possible? It seems that a lot of CI flows have the
> >     bandwidth and CPU power to just fetch a new image and prepare it each
> >     time CI gets triggered, but that seems like it would eat substantial
> >     time and put us back where we started.
>
> Yeah, I don't like that approach either ... I think having a repo that builds the images once is well worth the effort.
>
> Looking forwards to seeing the improvements!

Thanks, Sam. I'll start diving into this.

-Jonathan

>
> Sam
> _______________________________________________
> L2Ork-dev mailing list
> L2Ork-dev at disis.music.vt.edu
> https://disis.music.vt.edu/listinfo/l2ork-dev