Hello everyone!
I speak about the topic gesture / sound (360 degrees).
I was wondering what could be the possible ways to make the (physical) gesture of a performer interact (directly or indirectly) with Csound.
Let me explain: with what tools could I capture the gestures of my body (for example of the arms, or of the headband for me who am a double bassist), make them data, then use that data for (for example) a control of the sound parameters within the Csound code, or to do something else in the csd?
I have seen some experiences such as the fantastic performance of Richard Boulanger at the ICSC of Cagli in 2019, or the sensors applied to the arm for the feedback control of Oevind Brandstegg, also an idea made with Pd and Processing and honestly not much else...
That perhaps a first approach could be to put Csound in communication with Arduino through Processing... later I read about the existence of the Csoundo library, recently developed by Rory Walsh, to communicate Processing with Csound.
I have read several posts here on the list regarding the sensors and I wonder what could be a road already traveled that is in fact re-stradable?
Center for Digital Music (C4DM) at Queen Mary University London (QMUL)
has done interesting work in this area, might be worth looking into and apply
it to a Csound project.
Here is a link where I posted an example on how to track hands/fingers using mediapipe and control a csound instrument with it: https://forum.csound.com/t/short-latency-sound-control/1802/3
You could theoretically build up on that by tracking pose instead of hands and, of course, make more fitting csound instrument.
You don’t need Processing or Pd for communication between Arduino and Csound.
With the opcodes arduinoRead/arduinoReadF, serial data from Arduino can be easily handled as k-rate value.
You can use any sensors with Arduino to capture the gestures.
Another easy example is to use accelerometer and gyroscope of iPhone with touchOSC and Csound.
Check a template file “Sensors” in touchOSC.
If one is running on Intel/nvidia on Windows, you might be interested in trying nvidia’s Maxine, which uses a regular video feed (webcam or even stored video) for body and face tracking. It is not a complete Kinect replacement but it does quite a bit.
I’ve been experimenting with it on out-of-the-box Touch Designer, (which is IMO a great platform), but it could adapted to other platforms as well.
Hi Enrico
I've been working with getting data from human gestures/movements (mostly dancers) and translating it into sound in different ways over the years including computer vision techniques like Kinect, always using Csound to translate it into sound.
Lately I've been mostly into IMU sensors, basically the same as in your phone, which typically give you data on acceleration, angular velocity and orientation. The NGIMU sensors I use send OSC messages over wi-fi, making them very easy to use. They are also low latency (down to ca. 10 ms).
Here's a video from some recent experiments I did with a dancer and choreographer using 5 NGIMU sensors and a 8 channel setup: https://youtu.be/EK7bkLWfuwY
The sounds are made from five second running buffer containing the raw sensor data played back in different transpositions + some simple processing. The different dimensions (x,y,z) were a lot of the time routed to separate channels (but you can't hear that in the stero version online).
Best, Andreas