[L2Ork-dev] CI build times

[Jonathan Wilkes]

Sam,

I deputize you to use any and all methods to get our build times in
Purr Data down to reasonable levels.

Even with two or three productive devs submitting feature branches
we're averaging well over 12 hour wait time in the CI queue. That
times out the runner and requires me to manually retry. (And
apparently 12 hours is the largest value I can give
in the relevant Gitlab config file.)

Should we be using the "-j" flag and pipe the output for each core to
a file we include in the artifacts?

Should we prioritize downloading the nw.js binary from nwjs.io (and
Gem submodule from github) first and only fall back
to git.purrdata.net if that fails?

Whatever it takes, I will merge it.

-Jonathan

On Wed, Jul 1, 2020 at 6:34 AM Sam Thursfield <ssssam at gmail.com> wrote:
>
> On Wed, Jul 1, 2020 at 1:59 AM Jonathan Wilkes <jon.w.wilkes at gmail.com> wrote:
> >
> > On Mon, Jun 22, 2020 at 5:18 AM Sam Thursfield <ssssam at gmail.com> wrote:
> > > One is Meson[1], which is a replacement for Make/Automake/Autoconf.
> ...
>
> > The tough part is that we have so many kludges-- like downloading the asio
> > dlls for windows-- that doing things the "right" way might be difficult.
>
> Ultimately you can do anything with Meson as it lets you run arbitrary
> shell scripts, but automatically downloading dependencies would
> probably be better moved to a separate "prepare dependencies" script
> that you'd run before the build system.
>
> > Also, I'm surprised it only cut build times by half. I'd want something like a
> > 10x speedup by not having to re-run configure for every single external and
> > using all the cores to build.
> >
> > At the same time, it might be worth it to try to, say, build just the core with
> > meson for Linux.
>
> This was for a "simple" project. If you managed to port all the
> externals to Meson then it would be a *huge* speedup... but would that
> create problems with merging new changes from upstream versions of the
> externals?
>
> Porting the core alone would make a lot of sense. And it wouldn't need
> to be Linux only -- Meson can also generate project files for MSVC.
>
> > > The other is BuildStream[2], which is a "meta build" tool, i.e. it
> > > wraps an existing project's Makefile.
> ...
> > That sounds interesting. But how would that work for Windows and OSX?
>
> The project is intended to be useful on Windows and OSX, and I think
> it already works if you drive the builds from a Linux machine. I
> suppose the sandboxing guarantees go away if you're building on an OS
> that doesn't provide a containerised version. It's rather in the early
> stages as yet, but I think once BuildStream 2.0 is out, we will start
> to see a story for building apps like PD across the 3 major platforms.
> Right now, I present it just as a source of inspiration for how an
> ideal build infrastructure could look :)
>
> Sam
> _______________________________________________
> L2Ork-dev mailing list
> L2Ork-dev at disis.music.vt.edu
> https://disis.music.vt.edu/listinfo/l2ork-dev