[L2Ork-dev] CI build times

[Jonathan Wilkes]

On Mon, Jun 22, 2020 at 5:18 AM Sam Thursfield <ssssam at gmail.com> wrote:
>
> On Sun, Jun 21, 2020 at 7:18 PM Jonathan Wilkes <jon.w.wilkes at gmail.com> wrote:
> > Hi all,
> >
> > CI build times are now by far the bottleneck in Purr Data development cycles.
> >
> > Anyone know if any part of our build system is parallelizable? I know
> > Albert did some experiments with this at one point.
>
> There are a couple of tools which could be interesting for PD's
> builds. Neither would be an immediate fix for the build system, more
> of an idea to work towards, but they may be interesting...
>
> One is Meson[1], which is a replacement for Make/Automake/Autoconf.
> The key difference with these tools is that it's deliberately not
> Turing complete. I ported a project to Meson and it cut the
> configure+build time by more than half. One thing Meson can't do (as
> far as I know) is wrap another project's Makefile, so a porting effort
> would need to be "inside out", starting with the core and all the
> externals. I imagine there would need to be a way to automatically
> port externals to Meson for that to be practical, because there are a
> lot.

I think at one point I offered on the Meson mailing list to pay the developer
to switch our build system over to it. :)

The tough part is that we have so many kludges-- like downloading the asio
dlls for windows-- that doing things the "right" way might be difficult.

Also, I'm surprised it only cut build times by half. I'd want something like a
10x speedup by not having to re-run configure for every single external and
using all the cores to build.

At the same time, it might be worth it to try to, say, build just the core with
meson for Linux.

>
> The other is BuildStream[2], which is a "meta build" tool, i.e. it
> wraps an existing project's Makefile. The primary goal of BuildStream
> is *reliable* builds, so it requires that everything is built in a
> containerised sandbox, and needs a base image containing the necessary
> build dependencies (this can be a rootfs tarball, a Docker image,
> etc.). It could be interesting for PD as a replacement for the
> toplevel Makefile and the 'tar-em-up.sh' script. Each PD external
> could become a separate BuildStream element, and the BuildStream tool
> would run the Makefiles for many of them in parallel. It also supports
> caching the results, so the CI infrastructure could cache successful
> builds of the externals and only rebuild them when needed.
>
> I have some experience with both these tools if you have any questions :)
> Sam

That sounds interesting. But how would that work for Windows and OSX?

-Jonathan

>
> 1. https://mesonbuild.com/
> 2. https://buildstream.build/   -- Note that there is a stable 1.4
> version and a forever-in-development 2.0 version
> _______________________________________________
> L2Ork-dev mailing list
> L2Ork-dev at disis.music.vt.edu
> https://disis.music.vt.edu/listinfo/l2ork-dev