[L2Ork-dev] Purr-data: delegating GUI stuff to the GUI

[Giulio Moro]

Trying to re-send this from a different email address, given it has not received any response since it was sent out one week ago and I previously experienced my yahoo address to end up in everyone's spam folder.

Hello everyone,
I have just subscribed to this list, though I have been on other Pd lists for a while (and in fact I wrote something related to the below on Pd-dev recently).

I work on Bela, an embedded platform for audio and sensor processing (http://bela.io) which achieves ultra-low latency (down to 1ms roundtrip) on a full Linux computer (BeagleBone).
Our users have so far been able to run Pd patches using either a Vanilla 0.46 libpd which we wrapped with our own custom (non-ALSA) audio driver code or the online Heavy compiler. We make our audio, analog and digital I/O available within Pd through [adc~][dac~].

We are now looking into how to make live-patching for Pd on Bela.
I wrote a Pd audio backend for Bela and removed the file polling from the audio thread. The latter is both good real-time programming practice AND required for Bela, as it uses Xenomai. 
Bela has no way to connect a display, so we run Pd on Bela and the GUI on the host (Bela shows up as a network device over USB to the host computer).
Great, I got 64 samples per block, below 4ms roundtrip latency (note that Pd's internal "delay" is set to 0, thanks to Bela, so these are real roundtrip figures). Did the above with both the latest Pd and with Purr-data.

With this configuration I can do live patching and everything would work just fine, except that once you have more than a handful of objects in your patch, then you start hearing dropouts as soon as you move your mouse around. This is because each mouse move triggers a `motion` message to the Pd backend, handled by `canvas_motion()` in `g_editor.h`, invokes `canvas_doclick(... doit = false)`, which in turn iterates through all the objects on the patch and asks each of them "does the cursor happen to be on top of you?" (`canvas_findhitbox()`/`canvas_hitbox()`). This takes longer than you may expect, as it calls a callback function (`w_getrectfn()`) for each of those objects.

Bottom line: this thing is iterating through the `w_getrectfn()` of each object of the patch every time you move your mouse, and this happens within the audio thread, hence when you have many objects, (and small blocksizes and an ARM CPU, as we have on Bela) you get glitches for simply moving your mouse.

I thought I'd give it a shot to leverage the javascript in the Purr-data GUI to only send `motion` messages when my mouse is on top of an object and to send along with it the Pd "tag" of the object (using `onmouseover` and `onmouseout`). This drastically improves the performance on the Pd side, because much less needs to be done in the audio thread. It still has some things not working (e.g.: copy/paste works badly now, your cursor shape may get stuck when you move out of an object), but here it is: https://github.com/giuliomoro/purrdata/tree/simpler-motion

Once these bugs are ironed out, this could be a viable solution for us (Bela). You would still get glitches when moving many objects or creating many objects at a time, but you would be able to interact with a patch (moving mouse in edit/run modes) without many problems.

At the point we are at, we are struggling to decide what is the path forward when running Pd on Bela.
Ideally we would want all the GUI stuff to be delegated to the GUI, to free up more CPU for our audio processing and also because well... that's just the way it should be, right?
As mentioned, I dirtied my hands with both Pd and PurrData code, and it seems that the latter is more promising, but it may nevertheless require a huge effort to make it entirely suitable for our application.
Also, while we can provide some development time, we cannot really afford to maintain a separate fork of Pd or Pd-l2ork on our own.

Now, after this long introduction to say what I have been up to, I'd like to ask the Purr-data developers what is in the plans for Purr-data/Pd-l2ork.
I am interested in anything in the roadmap that aims at de-entangling the GUI stuff from the core of Pd and delegating it to the GUI, along the lines of my hack above, but in a more structured and complete way (e.g.: why do we need to ask Pd if we should flash an outlet onmouseover in the GUI? Why do we need to tell Pd that we are dragging a new patch cord until we actually release it? Why do we have to ask Pd about the cursor shape?).

Also, if there is an interest in potentially accepting a merge request for an improved version of my code above, that would already be a great deal to us, so I am happy to hear your opinions on that as well as what's your general opinion on changing the GUI/Pd interface and the roles of each? Are you trying to keep the Pd code for l2ork and Purr-data exactly the same? Is Purr-data the new Pd-l2ork? When?

Thanks,
Giulio