[Csnd] Passing Audio - Sharing Samples - from one Csound to Another

The Csound Mailinglist
1

Hello Everyone.

As some of you know from the work I shared at the ICSC Vienna, we have been passing control from one running Csound to another (with John ffitch) using OSC (and jamming in concert and in each other’s home studios), and we have been playing CsoundMeta Concerts using CsoundUnity and being joined in the performance space (concert hall and classrooms) by players from all over the world casting from within our Quest3 XR headsets and communicating via a tethered Samsung Cellphone HotSpot and running a ZeroTier VPN.

I am excited to announce that we now have live audio processing and sample capture in CsoundMeta running on the Quest2,3,pros! (We were not doing live audio sampling and processing yet at ICSC).

What we are working on now is how to send/pass the recorded samples from one ‘player’ to another. We can currently do this by recording the samples in Unity and passing them from Unity to Csound, BUT… we are hoping that we can do everything in Csound.

And so, I am wondering if any of you might have some suggestions or solutions for us.

Without folding everything into a massive FX orchestra driven by a sampling synth and using lots of Global variables or ZAK (and I am doing this now… it works and it is cool, but getting pretty huge.)

Could any of you recommend (or share) your way of passing audio from One running synth or sampling CSD to multiple independent running FX CSDs. Is it only possible with OSC or are there some opcodes that I have not used or am not aware of that might make this possible?

Essentially, our open source release of CsoundMeta would include a huge Synth and FX collection of available (independently runnable) csd files (or launchable from a huge library of them) and the CsoundMeta Modular System could pass audio between any of them serially or in parallel.

Is it currently possible to do this in Csound?

Thanks for any advice.

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

2

Hi Richard.
Oh, an interesting problem, and it can be useful to modularize a setup like this.
You could use some variant of a virtual audio cable (Jack, and similar), and then use extra in/outs in a standard manner in Csound.
Then there is also socksend/sockrecv, which I have tried earlier, but I had problems with synchronization/dropouts when sending audio over network in a raw manner like this. I think that Dante and similar technologies solves the network buffering in a smart manner to get around this and still keep latency low. Talking of which, I’ve found that I sometimes can get
lower* latencies for live audio i/o with Dante (even Dante virtual soundcard) then I can get with the ASIO soundcard drivers on windows (similar on Mac). So perhaps letting Csound talk to a Dante virtual soundcard can be a solution? Then it just looks like a regular audio device from Csound’s point of view. I kind of like that there does not have to be a special opcode to solve a generic problem like this, just use regular audio i/o.
A simpler and more everyday example of sending audio between Csound instances is when using Cabbage plugins in a DAW, routing between tracks freely. That is also an example of just using regular audio i/o, and then using the DAW to provide the communication. I can easily run >100 Csound instances like this, sending audio back and forth. Doing the routing in Reaper is very flexible, there is even a routing matrix overview to help you keep track of what goes where when it starts getting hairy :slight_smile:

all best
Øyvind

fre. 15. nov. 2024 kl. 01:07 skrev Dr. Richard Boulanger <rboulanger@berklee.edu>:

Hello Everyone.

As some of you know from the work I shared at the ICSC Vienna, we have been passing control from one running Csound to another (with John ffitch) using OSC (and jamming in concert and in each other’s home studios), and we have been playing CsoundMeta Concerts using CsoundUnity and being joined in the performance space (concert hall and classrooms) by players from all over the world casting from within our Quest3 XR headsets and communicating via a tethered Samsung Cellphone HotSpot and running a ZeroTier VPN.

I am excited to announce that we now have live audio processing and sample capture in CsoundMeta running on the Quest2,3,pros! (We were not doing live audio sampling and processing yet at ICSC).

What we are working on now is how to send/pass the recorded samples from one ‘player’ to another. We can currently do this by recording the samples in Unity and passing them from Unity to Csound, BUT… we are hoping that we can do everything in Csound.

And so, I am wondering if any of you might have some suggestions or solutions for us.

Without folding everything into a massive FX orchestra driven by a sampling synth and using lots of Global variables or ZAK (and I am doing this now… it works and it is cool, but getting pretty huge.)

Could any of you recommend (or share) your way of passing audio from One running synth or sampling CSD to multiple independent running FX CSDs. Is it only possible with OSC or are there some opcodes that I have not used or am not aware of that might make this possible?

Essentially, our open source release of CsoundMeta would include a huge Synth and FX collection of available (independently runnable) csd files (or launchable from a huge library of them) and the CsoundMeta Modular System could pass audio between any of them serially or in parallel.

Is it currently possible to do this in Csound?

Thanks for any advice.

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

3

Hi!

I think, Richard, you meant sending recorded samples or similar audio buffers to other player over network? It means, not to stream in realtime but just send the audio chunks so the other player can use it?

Using OSC and tables is probably the most straightforward solution, I think storing the incoming data to a table and playing that is probably quite reliable, real-time playback most definitely not (due wifi dropouts and similar).

socksend/sockrecv is an interesting idea. I have not got it working reliably either but if you store the signal to a table with some good conditions, it might work. This is less likely to work though… I am not sure if that works also on an Android system as the Quests are.

Very interesting idea and a truly imperssive project!

Best!
tarmo

Kontakt Oeyvind Brandtsegg (<obrandts@gmail.com>) kirjutas kuupäeval R, 15. november 2024 kell 10:40:

Hi Richard.
Oh, an interesting problem, and it can be useful to modularize a setup like this.
You could use some variant of a virtual audio cable (Jack, and similar), and then use extra in/outs in a standard manner in Csound.
Then there is also socksend/sockrecv, which I have tried earlier, but I had problems with synchronization/dropouts when sending audio over network in a raw manner like this. I think that Dante and similar technologies solves the network buffering in a smart manner to get around this and still keep latency low. Talking of which, I’ve found that I sometimes can get
lower* latencies for live audio i/o with Dante (even Dante virtual soundcard) then I can get with the ASIO soundcard drivers on windows (similar on Mac). So perhaps letting Csound talk to a Dante virtual soundcard can be a solution? Then it just looks like a regular audio device from Csound’s point of view. I kind of like that there does not have to be a special opcode to solve a generic problem like this, just use regular audio i/o.
A simpler and more everyday example of sending audio between Csound instances is when using Cabbage plugins in a DAW, routing between tracks freely. That is also an example of just using regular audio i/o, and then using the DAW to provide the communication. I can easily run >100 Csound instances like this, sending audio back and forth. Doing the routing in Reaper is very flexible, there is even a routing matrix overview to help you keep track of what goes where when it starts getting hairy :slight_smile:

all best
Øyvind

fre. 15. nov. 2024 kl. 01:07 skrev Dr. Richard Boulanger <rboulanger@berklee.edu>:

Hello Everyone.

As some of you know from the work I shared at the ICSC Vienna, we have been passing control from one running Csound to another (with John ffitch) using OSC (and jamming in concert and in each other’s home studios), and we have been playing CsoundMeta Concerts using CsoundUnity and being joined in the performance space (concert hall and classrooms) by players from all over the world casting from within our Quest3 XR headsets and communicating via a tethered Samsung Cellphone HotSpot and running a ZeroTier VPN.

I am excited to announce that we now have live audio processing and sample capture in CsoundMeta running on the Quest2,3,pros! (We were not doing live audio sampling and processing yet at ICSC).

What we are working on now is how to send/pass the recorded samples from one ‘player’ to another. We can currently do this by recording the samples in Unity and passing them from Unity to Csound, BUT… we are hoping that we can do everything in Csound.

And so, I am wondering if any of you might have some suggestions or solutions for us.

Without folding everything into a massive FX orchestra driven by a sampling synth and using lots of Global variables or ZAK (and I am doing this now… it works and it is cool, but getting pretty huge.)

Could any of you recommend (or share) your way of passing audio from One running synth or sampling CSD to multiple independent running FX CSDs. Is it only possible with OSC or are there some opcodes that I have not used or am not aware of that might make this possible?

Essentially, our open source release of CsoundMeta would include a huge Synth and FX collection of available (independently runnable) csd files (or launchable from a huge library of them) and the CsoundMeta Modular System could pass audio between any of them serially or in parallel.

Is it currently possible to do this in Csound?

Thanks for any advice.

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

4

Hello Øyvind,

Thank you so much for all your excellent suggestions - and especially your focus on latency and stability issues too!
In some situations, for sure, latency will matter, but in others, we just want to ‘share’ the recorded samples so that others in the “performance” can use them.

I and my students use Loopback and Blackhole too - and these are wonderful for interApplication Communication! (I am using Loopback now as I test and design for CsoundMeta from my Mic, Mixers, Cabbage, CsoundQt, etc.). ((I have done some work with Jack too… wondering if any of this (or things like this) are in Unity

At Berklee, Dante is used in our MultiChannel Classrooms and ConcertHalls, but I was an ‘early’ adopter and did not get far with it back in the day (so I have not used in years) - but in the past months one of my Thesis students has been using it in class and it has been working really well. (Again, I wonder if there is such a utility in Unity.)

Yes you are correct to deduce that we are essentially trying to model (in the Quest 2,3,3s,Pro) how we would use Cabbage instruments and effects in a DAW - like Live or Logic or Reaper. And, as you know, these capabilities work perfectly (and are now the assumed standard)

I have not yet tried or explored socksend/sockrecv and if that were to work, it might make the Live model work in the quest and from within Csound itself.

I will keep you all posted as we continue this work - and we do plan to share it all by the summer of 2025 (I am a keynote speaker at the 2025 ICMC - being hosted in Boston - and will be showing/workshopping CsoundMeta - hopefully will all sorts of live audio capture/sharing/processing too.

Thank you so much for your suggestions (and for all your AMAZING and INSPIRING performances, compositions and design work that you have shared!

Best

5

Thank you Tarmo for your suggestions and advice.

You are correct that one of the main things we are wanting to do is share samples that I record here in my studio with players who are collocated with me and playing with me from Ohio, LA, HongKong.

We are also wanting to work with every synth and effect CSD in our system as if they were modules in a Modular so that their audio could be routed from, for example, a generative FM csd, to a Granular FX csd and from the Granular FX csd to a ShimerVerb csd and from that to a LoFi Bit Reduction csd and maybe even back to a Sampling csd to live capture that and patch it to a WahWah fx csd.

(( Currently, if we use UNITY to capture and store the captured Audio )) we can pass it to any of our effects (fx csds) when we ‘turn them on’ - but we are thinking that there might be lower latency and other benefits if we can keep everything in Csound. ))

Essentially, choose (and control with the hands or Quest controllers) csdA (from our generative collection of instruments) and then patch the output to csdB(fxA) + csdB(fx2) + csdC(fx3) in parallel or in series (and control these with our hands or the Quest controllers)

And of course, we want a Frippertronics Live Looper (inspired by Victor Lazzarini’s ICSC Keynote).

  • did Victor share that fantastic .csd yet? :wink:

For sure, we need to try socksend/sockrecv in the next few days and see where that might lead.

We will keep you posted.
And… thank YOU Tarmo for all your AMAZING and INSPIRING music, performance, systems, teaching, CsoundQt work.

  • so grateful.

Best,

6

Ah, yes, memory sharing of audio buffers would help you a lot.
I did some investigations on how that could be done some years ago, and IIRC discussed it with Victor.
It could be possible via the Csound API to create memory buffers (tables, arrays) that could be accessed from several Csound instances running on the same computer (partly guessing here, but think it might be doable).
Øyvind

fre. 15. nov. 2024 kl. 14:47 skrev Dr. Richard Boulanger <rboulanger@berklee.edu>:

Hello Øyvind,

Thank you so much for all your excellent suggestions - and especially your focus on latency and stability issues too!
In some situations, for sure, latency will matter, but in others, we just want to ‘share’ the recorded samples so that others in the “performance” can use them.

I and my students use Loopback and Blackhole too - and these are wonderful for interApplication Communication! (I am using Loopback now as I test and design for CsoundMeta from my Mic, Mixers, Cabbage, CsoundQt, etc.). ((I have done some work with Jack too… wondering if any of this (or things like this) are in Unity

At Berklee, Dante is used in our MultiChannel Classrooms and ConcertHalls, but I was an ‘early’ adopter and did not get far with it back in the day (so I have not used in years) - but in the past months one of my Thesis students has been using it in class and it has been working really well. (Again, I wonder if there is such a utility in Unity.)

Yes you are correct to deduce that we are essentially trying to model (in the Quest 2,3,3s,Pro) how we would use Cabbage instruments and effects in a DAW - like Live or Logic or Reaper. And, as you know, these capabilities work perfectly (and are now the assumed standard)

I have not yet tried or explored socksend/sockrecv and if that were to work, it might make the Live model work in the quest and from within Csound itself.

I will keep you all posted as we continue this work - and we do plan to share it all by the summer of 2025 (I am a keynote speaker at the 2025 ICMC - being hosted in Boston - and will be showing/workshopping CsoundMeta - hopefully will all sorts of live audio capture/sharing/processing too.

Thank you so much for your suggestions (and for all your AMAZING and INSPIRING performances, compositions and design work that you have shared!

Best

Hi Richard.
Oh, an interesting problem, and it can be useful to modularize a setup like this.
You could use some variant of a virtual audio cable (Jack, and similar), and then use extra in/outs in a standard manner in Csound.
Then there is also socksend/sockrecv, which I have tried earlier, but I had problems with synchronization/dropouts when sending audio over network in a raw manner like this. I think that Dante and similar technologies solves the network buffering in a smart manner to get around this and still keep latency low. Talking of which, I’ve found that I sometimes can get
lower* latencies for live audio i/o with Dante (even Dante virtual soundcard) then I can get with the ASIO soundcard drivers on windows (similar on Mac). So perhaps letting Csound talk to a Dante virtual soundcard can be a solution? Then it just looks like a regular audio device from Csound’s point of view. I kind of like that there does not have to be a special opcode to solve a generic problem like this, just use regular audio i/o.
A simpler and more everyday example of sending audio between Csound instances is when using Cabbage plugins in a DAW, routing between tracks freely. That is also an example of just using regular audio i/o, and then using the DAW to provide the communication. I can easily run >100 Csound instances like this, sending audio back and forth. Doing the routing in Reaper is very flexible, there is even a routing matrix overview to help you keep track of what goes where when it starts getting hairy :slight_smile:

all best
Øyvind

fre. 15. nov. 2024 kl. 01:07 skrev Dr. Richard Boulanger <rboulanger@berklee.edu>:

Hello Everyone.

As some of you know from the work I shared at the ICSC Vienna, we have been passing control from one running Csound to another (with John ffitch) using OSC (and jamming in concert and in each other’s home studios), and we have been playing CsoundMeta Concerts using CsoundUnity and being joined in the performance space (concert hall and classrooms) by players from all over the world casting from within our Quest3 XR headsets and communicating via a tethered Samsung Cellphone HotSpot and running a ZeroTier VPN.

I am excited to announce that we now have live audio processing and sample capture in CsoundMeta running on the Quest2,3,pros! (We were not doing live audio sampling and processing yet at ICSC).

What we are working on now is how to send/pass the recorded samples from one ‘player’ to another. We can currently do this by recording the samples in Unity and passing them from Unity to Csound, BUT… we are hoping that we can do everything in Csound.

And so, I am wondering if any of you might have some suggestions or solutions for us.

Without folding everything into a massive FX orchestra driven by a sampling synth and using lots of Global variables or ZAK (and I am doing this now… it works and it is cool, but getting pretty huge.)

Could any of you recommend (or share) your way of passing audio from One running synth or sampling CSD to multiple independent running FX CSDs. Is it only possible with OSC or are there some opcodes that I have not used or am not aware of that might make this possible?

Essentially, our open source release of CsoundMeta would include a huge Synth and FX collection of available (independently runnable) csd files (or launchable from a huge library of them) and the CsoundMeta Modular System could pass audio between any of them serially or in parallel.

Is it currently possible to do this in Csound?

Thanks for any advice.

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

7

We will look deeper into memory sharing via the API. That might be the way. THANKS.

8

Hi Dr.B,

I think it can be worth investigating Unity Native Audio Plugins: https://docs.unity3d.com/Manual/AudioMixerNativeAudioPlugin.html
You will be able to add a custom AudioEffect to every AudioSource that grabs its output and maybe routes it to another AudioSource.
I have no idea how to do it but it may be possible :smiley:

9

Thanks Gio! We will look into that and we look forward to jamming with you soon too.

Dr. Richard Boulanger
Professor
Electronic Production and Design
Berklee College of Music

10

Be aware that shared memory in WASM code is somewhat different than in native code. There are different APIs and there are some security considerations re CORS.

11

Thank you Michael.

Dr. Richard Boulanger
Professor
Electronic Production and Design
Berklee College of Music