[L2Ork-dev] Purr-data: delegating GUI stuff to the GUI

[Jonathan Wilkes]

Hi Giulio,Thanks for the message. I'll reply inline below.

> Trying to re-send this from a different email address, given it has not received any response 
since it was sent out one week ago and I previously experienced my yahoo address to end up 
in everyone's spam folder.
I've apparently had problems with my messages getting through as well.

> Hello everyone,
> I have just subscribed to this list, though I have been on other Pd lists for a while (and in fact I wrote something related to the 
> below on Pd-dev recently).

> I work on Bela, an embedded platform for audio and sensor processing (http://bela.io) which achieves ultra-low latency (down to 
1ms roundtrip) on a full Linux computer (BeagleBone).

> Our users have so far been able to run Pd patches using either a Vanilla 0.46 libpd which we wrapped with our own custom (non-ALSA) audio driver code or the online Heavy compiler. We make our audio, analog and digital I/O available within Pd through [adc~][dac~].

> We are now looking into how to make live-patching for Pd on Bela.

> I wrote a Pd audio backend for Bela and removed the file polling from the audio thread.
I'm not exactly sure what you mean by "removed" here. Doesn't Pd allow you to use
either a polling or callback API to use with the audio backend?

> The latter is both good real-time programming practice AND required for Bela, as it uses Xenomai. 
Bela has no way to connect a display, so we run Pd on Bela and the GUI on the host (Bela shows up 
as a network device over USB to the host computer).

> Great, I got 64 samples per block, below 4ms roundtrip latency (note that Pd's internal "delay" is set to 0, 
thanks to Bela, so these are real roundtrip figures). Did the above with both the latest Pd and with Purr-data.
That's quite impressive.
How did you measure the 4ms roundtrip latency?

> With this configuration I can do live patching and everything would work just fine, except that once you have 
more than a handful of objects in your patch, then you start hearing dropouts as soon as you move your mouse 
around. This is because each mouse move triggers a `motion` message to the Pd backend, handled by 
`canvas_motion()` in `g_editor.h`, invokes `canvas_doclick(... doit = false)`, which in turn iterates through all the 
objects on the patch and asks each of them "does the cursor happen to be on top of you?" (`canvas_findhitbox()`/`canvas_hitbox()`). This takes longer than you may expect, as it calls a callback function 
(`w_getrectfn()`) for each of those objects.

> Bottom line: this thing is iterating through the `w_getrectfn()` of each object of the patch every 
time you move your mouse, and this happens within the audio thread, hence when you have many objects, 
(and small blocksizes and an ARM CPU, as we have on Bela) you get glitches for simply moving your mouse.
That's correct. Note that it is actually even a bit worse when mousing over a "Put" menu array, which triggers 
a loop or two in order to find the array element underneath the mouse.

> I thought I'd give it a shot to leverage the javascript in the Purr-data GUI to only send `motion` messages when 
my mouse is on top of an object and to send along with it the Pd "tag" of the object (using `onmouseover` and 
`onmouseout`). This drastically improves the performance on the Pd side, because much less needs to be done 
in the audio thread. It still has some things not working (e.g.: copy/paste works badly now, your cursor shape may get stuck when you move out of an object), but here it is: https://github.com/giuliomoro/purrdata/tree/simpler-motion
I just quickly glanced at the code, but it looks like a workable approach.
There is one rather important edge case I can think of, which is [mousestate] from cyclone. It tracks the 
mouse position regardless of whether the mouse button is clicked. Currently Purr Data has a problem because 
there's no way I've found to track mouse motion outside of the document window (unless there's a Chrome 
App API that I haven't discovered.)

> Once these bugs are ironed out, this could be a viable solution for us (Bela). You would still get glitches 
when moving many objects or creating many objects at a time, but you would be able to interact with a patch 
(moving mouse in edit/run modes) without many problems.
Yes, I like this kind of incremental approach.
Also, Pd-l2ork already has a nice improvement for moving selections. It still must be called back on each 
x/y delta reported by the mouse, but it's only a single message instead of one per object in the selection.

> At the point we are at, we are struggling to decide what is the path forward when running Pd on Bela. 
Ideally we would want all the GUI stuff to be delegated to the GUI, to free up more CPU for our audio processing 
and also because well... that's just the way it should be, right?
One thing to keep in mind is that the lack of specification for Pd the language makes it difficult 
to predict how much work is required to do something like that. Essentially, anything that a user could 
possibly access from the language gets accessed and used for some creative purpose. For example, 
look on the Pd forum at what weird things somebody is doing with the data structures API, or even 
using iemguis to create a display surface.
I'm all for a clean separation between GUI and audio process, but I do need to keep in mind 
that such a separation _will_ initially end up degrading the UX for some set of cases. And because we 
don't have a spec to go by, such users will interpret that degradation as damage to the integrity of 
the language, even if the benefit of the change clearly outweighs everything else. So we have to 
label such degradations as "bugs" when they occur and find a way to fix them.
Most of the work right now on Purr Data is labeling such bugs, and finding workarounds 
for them.

Here's an example-- a clean separation between GUI and audio process may decide to only 
send object position updates on mouseup. Such an approach would forget that [cnv] 
has a "get_pos" method to find the current coordinates of the object. So if the GUI doesn't 
send updates for each mouse motion tick, control surfaces based on that method will fail.That's a bug that would need to be fixed.
Just thinking about that example-- another degradation would be that moving an [inlet] 
relative to another [inlet] inside a GOP wouldn't update corresponding connections on 
the parent in realtime. That's definitely not as important as above. But it's an example 
of what to keep in mind when refactoring the current API.

> As mentioned, I dirtied my hands with both Pd and PurrData code, and it seems that the latter 
is more promising, but it may nevertheless require a huge effort to make it entirely suitable for our application. 
Also, while we can provide some development time, we cannot really afford to maintain a separate fork of Pd or 
Pd-l2ork on our own.
Well, once you work the kinks out of your currrent improvements, please make a merge request. 
I think it would be a huge usability improvement-- I know several Pd users have complained about mouse 
motion eating their CPU in the past.

> Now, after this long introduction to say what I have been up to, I'd like to ask the Purr-data developers what is 
in the plans for Purr-data/Pd-l2ork.

> I am interested in anything in the roadmap that aims at de-entangling the GUI stuff from the core 
of Pd and delegating it to the GUI, along the lines of my hack above, but in a more structured and 
complete way (e.g.: why do we need to ask Pd if we should flash an outlet onmouseover in the GUI? 
Why do we need to tell Pd that we are dragging a new patch cord until we actually release it? Why do 
we have to ask Pd about the cursor shape?).
Those are both low-hanging fruit that could pretty easily be moved to the GUI.
I don't yet have a roadmap for doing more separation work.
One starting point would be to go through and figure out which parts of the GUI 
are eating the most CPU in the audio process.
Another would be to go through and figure out which parts are just 
trivial low hanging fruit-- like xlet highlighting, and which are most 
complex-- like rtext_senditup.

For example: rtext_senditup actually hold parts of the missing 
specification within it. For one thing, that function has the recipe for how to 
split lines of text in objects, messages, and comments. But even if 
line splitting upon object creation eats a lot of CPU, it's unwise to 
go about refactoring that function right now because you risk losing key 
parts of the line-splittng algorithm in the process.

But I'm certainly open to ideas here.
> Also, if there is an interest in potentially accepting a merge request for 
an improved version of my code above, that would already be a great deal to us, 
so I am happy to hear your opinions on that as well as what's your general 
opinion on changing the GUI/Pd interface and the roles of each? Are you trying 
to keep the Pd code for l2ork and Purr-data exactly the same? Is Purr-data the 
new Pd-l2ork? When?
Purr Data is the 2.0 release of Pd-l2ork.
I basically removed the tcl/tk code from the core in order to do improvements 
like the ones you are suggesting.
-Jonathan

> Thanks,
> Giulio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://disis.music.vt.edu/pipermail/l2ork-dev/attachments/20170616/d294ab7b/attachment-0001.html>