Short-latency sound control

General
#21

Meanwhile, here is the report of the installation of the mediapipe library with the command:
python3 -m pip install mediapipe

python3 -m pip install mediapipe
Collecting mediapipe
Downloading mediapipe-0.10.18-cp311-cp311-macosx_11_0_x86_64.whl.metadata (9.7 kB)
Collecting absl-py (from mediapipe)
Downloading absl_py-2.1.0-py3-none-any.whl.metadata (2.3 kB)
Collecting attrs>=19.1.0 (from mediapipe)
Downloading attrs-24.2.0-py3-none-any.whl.metadata (11 kB)
Collecting flatbuffers>=2.0 (from mediapipe)
Downloading flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes)
Collecting jax (from mediapipe)
Downloading jax-0.4.35-py3-none-any.whl.metadata (22 kB)
Collecting jaxlib (from mediapipe)
Downloading jaxlib-0.4.35-cp311-cp311-macosx_10_14_x86_64.whl.metadata (983 bytes)
Collecting matplotlib (from mediapipe)
Downloading matplotlib-3.9.2-cp311-cp311-macosx_10_12_x86_64.whl.metadata (11 kB)
Collecting numpy<2 (from mediapipe)
Downloading numpy-1.26.4-cp311-cp311-macosx_10_9_x86_64.whl.metadata (61 kB)
Collecting opencv-contrib-python (from mediapipe)
Downloading opencv_contrib_python-4.10.0.84-cp37-abi3-macosx_12_0_x86_64.whl.metadata (20 kB)
Collecting protobuf<5,>=4.25.3 (from mediapipe)
Downloading protobuf-4.25.5-cp37-abi3-macosx_10_9_universal2.whl.metadata (541 bytes)
Collecting sounddevice>=0.4.4 (from mediapipe)
Downloading sounddevice-0.5.1-py3-none-macosx_10_6_x86_64.macosx_10_6_universal2.whl.metadata (1.4 kB)
Collecting sentencepiece (from mediapipe)
Downloading sentencepiece-0.2.0-cp311-cp311-macosx_10_9_x86_64.whl.metadata (7.7 kB)
Collecting CFFI>=1.0 (from sounddevice>=0.4.4->mediapipe)
Downloading cffi-1.17.1-cp311-cp311-macosx_10_9_x86_64.whl.metadata (1.5 kB)
Collecting ml-dtypes>=0.4.0 (from jax->mediapipe)
Downloading ml_dtypes-0.5.0-cp311-cp311-macosx_10_9_universal2.whl.metadata (21 kB)
Collecting opt-einsum (from jax->mediapipe)
Downloading opt_einsum-3.4.0-py3-none-any.whl.metadata (6.3 kB)
Collecting scipy>=1.10 (from jax->mediapipe)
Downloading scipy-1.14.1-cp311-cp311-macosx_10_13_x86_64.whl.metadata (60 kB)
Collecting contourpy>=1.0.1 (from matplotlib->mediapipe)
Downloading contourpy-1.3.0-cp311-cp311-macosx_10_9_x86_64.whl.metadata (5.4 kB)
Collecting cycler>=0.10 (from matplotlib->mediapipe)
Downloading cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB)
Collecting fonttools>=4.22.0 (from matplotlib->mediapipe)
Downloading fonttools-4.54.1-cp311-cp311-macosx_10_9_universal2.whl.metadata (163 kB)
Collecting kiwisolver>=1.3.1 (from matplotlib->mediapipe)
Downloading kiwisolver-1.4.7-cp311-cp311-macosx_10_9_x86_64.whl.metadata (6.3 kB)
Collecting packaging>=20.0 (from matplotlib->mediapipe)
Downloading packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pillow>=8 (from matplotlib->mediapipe)
Downloading pillow-11.0.0-cp311-cp311-macosx_10_10_x86_64.whl.metadata (9.1 kB)
Collecting pyparsing>=2.3.1 (from matplotlib->mediapipe)
Downloading pyparsing-3.2.0-py3-none-any.whl.metadata (5.0 kB)
Collecting python-dateutil>=2.7 (from matplotlib->mediapipe)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pycparser (from CFFI>=1.0->sounddevice>=0.4.4->mediapipe)
Downloading pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Collecting six>=1.5 (from python-dateutil>=2.7->matplotlib->mediapipe)
Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Downloading mediapipe-0.10.18-cp311-cp311-macosx_11_0_x86_64.whl (49.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.0/49.0 MB 2.0 MB/s eta 0:00:00
Downloading attrs-24.2.0-py3-none-any.whl (63 kB)
Downloading flatbuffers-24.3.25-py2.py3-none-any.whl (26 kB)
Downloading numpy-1.26.4-cp311-cp311-macosx_10_9_x86_64.whl (20.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.6/20.6 MB 1.6 MB/s eta 0:00:00
Downloading protobuf-4.25.5-cp37-abi3-macosx_10_9_universal2.whl (394 kB)
Downloading sounddevice-0.5.1-py3-none-macosx_10_6_x86_64.macosx_10_6_universal2.whl (107 kB)
Downloading absl_py-2.1.0-py3-none-any.whl (133 kB)
Downloading jax-0.4.35-py3-none-any.whl (2.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 2.1 MB/s eta 0:00:00
Downloading jaxlib-0.4.35-cp311-cp311-macosx_10_14_x86_64.whl (87.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.9/87.9 MB 2.2 MB/s eta 0:00:00
Downloading matplotlib-3.9.2-cp311-cp311-macosx_10_12_x86_64.whl (7.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.9/7.9 MB 2.2 MB/s eta 0:00:00
Downloading opencv_contrib_python-4.10.0.84-cp37-abi3-macosx_12_0_x86_64.whl (66.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.3/66.3 MB 2.1 MB/s eta 0:00:00
Downloading sentencepiece-0.2.0-cp311-cp311-macosx_10_9_x86_64.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 2.1 MB/s eta 0:00:00
Downloading cffi-1.17.1-cp311-cp311-macosx_10_9_x86_64.whl (182 kB)
Downloading contourpy-1.3.0-cp311-cp311-macosx_10_9_x86_64.whl (266 kB)
Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)
Downloading fonttools-4.54.1-cp311-cp311-macosx_10_9_universal2.whl (2.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 2.2 MB/s eta 0:00:00
Downloading kiwisolver-1.4.7-cp311-cp311-macosx_10_9_x86_64.whl (65 kB)
Downloading ml_dtypes-0.5.0-cp311-cp311-macosx_10_9_universal2.whl (736 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 736.8/736.8 kB 2.0 MB/s eta 0:00:00
Downloading packaging-24.2-py3-none-any.whl (65 kB)
Downloading pillow-11.0.0-cp311-cp311-macosx_10_10_x86_64.whl (3.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 2.1 MB/s eta 0:00:00
Downloading pyparsing-3.2.0-py3-none-any.whl (106 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading scipy-1.14.1-cp311-cp311-macosx_10_13_x86_64.whl (39.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.1/39.1 MB 2.0 MB/s eta 0:00:00
Downloading opt_einsum-3.4.0-py3-none-any.whl (71 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading pycparser-2.22-py3-none-any.whl (117 kB)
Installing collected packages: sentencepiece, flatbuffers, six, pyparsing, pycparser, protobuf, pillow, packaging, opt-einsum, numpy, kiwisolver, fonttools, cycler, attrs, absl-py, scipy, python-dateutil, opencv-contrib-python, ml-dtypes, contourpy, CFFI, sounddevice, matplotlib, jaxlib, jax, mediapipe
Attempting uninstall: numpy
Found existing installation: numpy 2.1.3
Uninstalling numpy-2.1.3:
Successfully uninstalled numpy-2.1.3
WARNING: The script f2py is installed in ‘/Library/Frameworks/Python.framework/Versions/3.11/bin’ which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts fonttools, pyftmerge, pyftsubset and ttx are installed in ‘/Library/Frameworks/Python.framework/Versions/3.11/bin’ which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed CFFI-1.17.1 absl-py-2.1.0 attrs-24.2.0 contourpy-1.3.0 cycler-0.12.1 flatbuffers-24.3.25 fonttools-4.54.1 jax-0.4.35 jaxlib-0.4.35 kiwisolver-1.4.7 matplotlib-3.9.2 mediapipe-0.10.18 ml-dtypes-0.5.0 numpy-1.26.4 opencv-contrib-python-4.10.0.84 opt-einsum-3.4.0 packaging-24.2 pillow-11.0.0 protobuf-4.25.5 pycparser-2.22 pyparsing-3.2.0 python-dateutil-2.9.0.post0 scipy-1.14.1 sentencepiece-0.2.0 six-1.16.0 sounddevice-0.5.1

#22

mediapipe seems ok.

ctcsound you can check by simply running the example above and if you don’t get import error for ctcsound then you are fine. Otherwise you need to install/link it

#23

I read on the net that ctcsound from version 6.07 of Csound is configured.

However, I will run the test you suggest (I hope I don’t do damage to Csound… :slight_smile:

Can this command in Terminal work?

  • python3 -m pip install ctcsound==[Csound 6.18]
#24

Hi Lovre,

I launched on IDLE Shell 3.11 of Python your example (script) that you had posted;
At the bottom of the script you can read the comment.
(But maybe ctcsound is installed…?)

Python 3.11.9 (v3.11.9:de54cf5be3, Apr 2 2024, 07:12:50) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type “help”, “copyright”, “credits” or “license()” for more information.
CAMERA_ID = 0 # if there are multiple cameras connected, with this you can select which one to use

MediaPipe parameters

MAX_NUM_HANDS = 2
MODEL_COMPLEXITY = 0
MIN_DETECTION_CONFIDENCE = 0.5
MIN_TRACKING_CONFIDENCE = 0.5

------------------------------------------------------------

Imports

import cv2
import mediapipe as mp
import ctcsound

------------------------------------------------------------

MediaPipe setup

mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles
mp_hands = mp.solutions.hands

------------------------------------------------------------

------------------------------------------------------------

Csound setup

------------------------------------------------------------

orc = ‘’’
ksmps = 128
nchnls = 2
0dbfs = 1

instr 1
; IO channels
kamp chnget “amp” ; reading value from amp input channel
kfreq chnget “freq” ; reading value from freq input channel

aout poscil kamp, kfreq                     ; audio synthesis
outs aout, aout                           ; sending audio to speakers

endin
‘’’

cs = ctcsound.Csound()
pt = None # csound performance thread

#cs.compileCsdText(csd_text)
cs.setOption(‘-odac’)
cs.setOption(‘-b1024’) # SW buffer size
cs.setOption(‘-B1024’) # HW buffer size
cs.compileOrc(orc)
cs.readScore(‘i 1 0 10000’) # run for a day
cs.start()

pt = ctcsound.CsoundPerformanceThread(cs.csound())
pt.play()

Create control IO channels

def createChannel(channelName):
chn, _ = cs.channelPtr(channelName,
ctcsound.CSOUND_CONTROL_CHANNEL | ctcsound.CSOUND_INPUT_CHANNEL)
return chn

ampChannel = createChannel(“amp”) # uses utility method to create a channel and get numpy array to write to
freqChannel = createChannel(“freq”)

------------------------------------------------------------

------------------------------------------------------------

Processing part

pressed_button = False
cap = cv2.VideoCapture(CAMERA_ID)
with mp_hands.Hands(
static_image_mode = False,
max_num_hands = MAX_NUM_HANDS,
model_complexity = MODEL_COMPLEXITY,
min_detection_confidence = MIN_DETECTION_CONFIDENCE,
min_tracking_confidence = MIN_TRACKING_CONFIDENCE) as hands:

# Main loop
while cap.isOpened():
    success, image = cap.read()
    if not success:
        print("Ignoring empty camera frame.")
        continue 
       
    # To improve performance, optionally mark the image as not writeable to pass by reference.
    image.flags.writeable = False
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    results = hands.process(image)

    image_height, image_width, _ = image.shape

    # Iterate through hands, draw hand landmarks on image and write to csound control channels
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    ampChannel[0] = 0.0 # if hand is not detected, set amp to 0 to turn off sound
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            mp_drawing.draw_landmarks(
                image,
                hand_landmarks,
                mp_hands.HAND_CONNECTIONS,
                mp_drawing_styles.get_default_hand_landmarks_style(),
                mp_drawing_styles.get_default_hand_connections_style())
            
            handIndex = results.multi_hand_landmarks.index(hand_landmarks)
            handLabel = results.multi_handedness[handIndex].classification[0].label

            # image is mirrored so we need to invert left and right hand detection 
            if handLabel == 'Left':  handLabel = 'Right'
            elif handLabel == 'Right':  handLabel = 'Left'

            # Write to control channels
            if handLabel == 'Left':
                freqChannel[0] = 400 - abs(hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP].x)*300 
                ampChannel[0] = 1-abs(hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP].y) 

    # Flip the image horizontally for a selfie-view display.
    cv2.imshow('MediaPipe Hands', cv2.flip(image, 1))
    if cv2.waitKey(5) & 0xFF == 27:
        break

cap.release()
SyntaxError: multiple statements found while compiling a single statement

#25

Hi @Enrico .

It’s difficult to say what is exactly wrong there.

You can copy my example into a file and name that file e.g. hands_csound.py and after that run that using python3 hands_csound.py. Your active terminal directory needs to be that directory where your hands_csound.py file is.

Alternately, download Visual Studio Code and run it from there. If you plan further to work on such creative ideas I strongly recommend you to learn python a bit. That will make realization of your ideas much easier.

#26

Hi Lovre,

With the help of a friend I managed to create (with Xcode) a python file (.py) containing your script.

Then I launched the file from Terminal. Everything works!
The camera captures the movements of my hands and makes them sound.

I see that the vast mediapipe library uses Artificial Intelligence.

Now I would like to do some tests by modifying, for the moment, something only in the section that contains the Csound code in the script that you shared here.

Clearly I still have everything to learn about the Python language.

Thanks Lovre!
E

#27

Cool! I’m glad that you managed to run it :beers: