[L2Ork-dev] Interested in working on JIT-compiled Signal Graph for the Audio Engine

Fri Mar 12 12:27:51 EST 2021

On Fri, Mar 12, 2021 at 11:13 AM Prateek Pardeshi
<prateekpardeshi9 at gmail.com> wrote:
>
> Hello Jonathan,
>
> Glad to hear from you!
>
> On Thu, 11 Mar 2021 at 22:46, Jonathan Wilkes <jon.w.wilkes at gmail.com> wrote:
> >
> > On Thu, Mar 11, 2021 at 11:21 AM Prateek Pardeshi
> > <prateekpardeshi9 at gmail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > I'm Prateek Pardeshi, a junior Computer Science student at SRM IST, Chennai. I have been contributing to Open Source since October 2019 (Since Hacktober 2019).
> > >
> > > Since then I'm in love with Open Source, because of that I use Linux as my Primary OS and contribute frequently to OS organisation.
> > >
> > >
> > > My Previous Open Source Experience:
> > >
> > > 1. Open Source Fellow'21 at HackIllinois
> > > 2. GirlScript Summer of Code'20 Participant
> > > 2. Mentor for StudentCodein'20 (Similar to Google Code-in)
> > >
> > > Project link: https://git.purrdata.net/jwilkes/summer-of-code-ideas-list#jit-compiled-signal-graph-for-the-audio-engine
> > >
> > > I'm interested in this project because I worked as a fellow under Dr.Michael Kruse, maintainer of LLVM Compiler Infrastructure(Polly), also I am contributor at LLVM Compiler Infrastructure, specifically Polly, which is a loop optimiser written in C language. I'm familiar with the LLVM community and the working/debugging/building/issues of LLVM compiler Infrastructure.
> > >
> > > Recently I worked on refabricating reference-counting used for memory management to C++ binding.
> > > This project will provide me an opportunity to work on something new while contributing to the Open Source.
> > >
> > > Also, would like to know your expectations, deliverables, scope and thoughts on this project.
> > >
> > > In addition to that I've contributed to Gatsby(also an ex-maintainer), GNOME, LBRY, UNO Project, etc.
> > >
> > > Looking Forward to being a long term contributor here. :)
> >
> > Hello Prateek Pardeshi,
> >
> > Welcome!
> >
> > That is definitely one of the more ambitious projects listed there. I
> > believe there is prior art which has been open sourced here:
> >
> > https://github.com/enzienaudio/hvcc
>
> Thanks for sharing this, I went through this project and found it engrossing.
> So, I went through the docs, and got to know about the heavy framework
> used in the audio/gaming industry.
> How much is this project related to the JIT graph idea?

Just from skimming-- Heavy takes a Pd file as input and creates
functionally equivalent code in the Heavy language which can then be
output as source using a variety of code generators.

With the JIT DSP graph idea, you're basically taking an array of
function calls that represent a direct acyclic graph and either
swapping out multiple parts of that graph with a call to a single
optimized function call, or swapping out the entire array with a
single optimized function call.

So the JIT project idea is related in that both projects are
generating code from Pd code. But the JIT project idea also requires
compiling that code on the fly, and swapping it out in the signal
graph.

There's some other prior art mentioned in the project idea that is
closer to this JIT project idea. But Heavy is definitely worth looking
at since it takes the entire Pd program as input and probably has some
substantial optimizations based on that.

> Because I
> discovered about hvcc more there
>
> > A challenge for a Purr Data project related to JIT is how it
> > integrates into the current workflow. Would this be something that
> > automatically updates as the user builds their program? Or would they
> > finishing building their program and then click a button to "optimize"
> > it using this compiler?
> >
> I do agree that this part definitely catches attention. Also, In my
> opinion, there should be automatic building of the program, but it
> also depends on how much could be implemented or achieved.
>
> > As with previous years, I'd suggest an incremental approach to the
> > core part of the project:
> >
> > 1. proof of concept-- e.g., taking some of the simple DSP classes from
> > d_arithmetic.c. E.g., examples that are essentially a simple filter
> > chain of single inputs and single outputs like a -> b -> c etc.
> >
> > 2. extending that to graphs that include objects with multiple inputs
> > and outputs
> >
> > 3. consider extending this to objects that convert to/from signal data
> > to non-DSP data
> >
> > 4. extending the approach so that some subset of dynamically loaded
> > external DSP classes
> >
> > 5. and so on...
> >
> I've worked with graphs on web projects using Chart.js JavaScript
> framework before and I followed a similar approach there, where I
> generated dynamic graphs for the readings of NodeMCU(esp32) hardware.
> I would be interested in implementing it here with the DSP graphs as
> well.

The "DSP graph" here is an directed acyclic graph represented as an array
of function pointers (and pointers to function args). This is
represented visually in a
Purr Data diagram as a series of boxes connected by thick lines. (There's also a
visual "scripting language" that exists in the same diagram, but let's
ignore that for the moment.)

The project here isn't to dynamically generate new visual diagrams in
the front end.
Rather, it is to take a given visual diagram and generate (i.e.,
compile) an optimized DSP
graph swap out in the backend.

The prior art listed in the project idea is a good starting point for this.

>
> > As far as scope goes-- this project idea was written before Google
> > changed the number of hours. So perhaps focusing on the core classes
> > and leaving external libraries as a stretch goal. But I'd definitely
> > make sure to prioritize UX, even for the proof of concept.
> >
> I think that this project would take longer than Google has actually
> alloted, but I would like to work on this project post GSoC period
> too.

Yeah, it's worth studying the prior art and finding a way to narrow the project
down to a manageable set of changes.

Also, notice that the prior art for `[expr]` is essentially taking one function
pointer in the DSP graph and switching it out with an optimized one. So it's a
1:1 relationship. The JIT idea would presumably switch out multiple
function pointers
with a single one, i.e., an N:1 relationship. That's a lot more complicated.

Best,
Jonathan

>
> > Best,
> > Jonathan
> >
> > >
>
> Regards,
> Prateek Pardeshi
>
> > > _______________________________________________
> > > L2Ork-dev mailing list
> > > L2Ork-dev at disis.music.vt.edu
> > > https://disis.music.vt.edu/listinfo/l2ork-dev
> > _______________________________________________
> > L2Ork-dev mailing list
> > L2Ork-dev at disis.music.vt.edu
> > https://disis.music.vt.edu/listinfo/l2ork-dev
> _______________________________________________
> L2Ork-dev mailing list
> L2Ork-dev at disis.music.vt.edu
> https://disis.music.vt.edu/listinfo/l2ork-dev