I thought you might be interested in checking this out. This is Csound running on
baremetal no OS in the ARM processor of a Xylinx Zynq 7000 board,
output sent to the FPGA processor where a stereo reverb reverbsc port is running,
then to the DAC.
This demonstrates an integration of Csound with accelerated code running in the
FPGA. This is Aman Jagwani’s work, which is showing good potential for
further applications.
Wow… the future!!!
Congratulations.
Wonderful.
Dr. Richard Boulanger
Professor
Electronic Production and Design
Berklee College of Music
That is really inspiring!
So, perhaps in the future it would be possible to whip up a nice Csound orchestra file, compile it for the Xylinx or similar, flash the device with the compiled code, and presto! A custom eurorack module!
Ok, it would need a few ins and outs and perhaps a few nobs.
It would be nice if someone designed a standardized hardware (kit) in eurorack format with a given number of patch points and a given number of buttons and knobs, and released the documentation so that anything could be ported to it - Csound, pd, chuck …
Regards,
Anders
Yes! I hope that Aman starts that company!!!
Dr. Richard Boulanger
Professor
Electronic Production and Design
Berklee College of Music
We shan't be bothered with pd or chuck, to be honest. 
Prof. Victor Lazzarini
Maynooth University
Ireland
And here I was trying NOT to play King of the Hill. Csound, of course, is the one and only choice. But mentioning these other suspicious pieces of software is how we lure the innocent bystanders in.
Super-cool. Gear-lust!
I had to look this up, b/c FPGA is a new term for me, so I’d thought I’d share this link:
thanks for the link aaron.
That’s great,
It may be of interest that the Faust team have been working on compiling to FPGA using the Syfala toolchain.
It’s a different approach to using baremetal apps and custom interfaces to FPGA IP and I’d love to see Aman’s code if it is to be made available, although I do understand that it is much more complicated to document an FPGA based system compared to a simple baremetal app, but if it is part of an ongoing thesis etc. then I’d love to see the final version too.
Currently the Syfala toolchain has been ported to a couple of Zybo boards (such as the one Aman is using in the youtube video) and I think one other but there are hints the support for other boards (such as my Zedboard) will be developed.
I was fortunate enough in my latter years as an electronics engineer before retirement to have worked with the Zynq 7000 platform and our prototypes were based on the Zedboard, which happened to include an I2S interface DAC/ADC so I’m quite familiar with the process of creating Fpga cores and then compiling firmware to interface with them. It’s quite a steep learning curve but I would recommend anyone seriously interested to start out by searching for “Adam Taylors microzed chronicles” which are available in multiple formats and free to read on the Web.
The PCB layout was perhaps not to high audio quality but I have been looking into porting Syfala to Zedboard since at present the Syfala team have only had the bandwidth to target a couple of Zybo boards and I think one other.
To me this is a perfectly logical extension of Barry Vercoes (and no doubt others) work on using DSP boards (Analog Devices SHARC if I remember correctly) given the massive parallelism offered by FPGA’s, and in the case of Zynq devices, large numbers of DSP resources (invokable using the useDSP pragma in the constraints file for the design).
Obviously the Faust/Syfala team have a different approach to Csound but they have a large library of example code, albeit from a largely academic user base, or at least that’s my early impression.
I’ve seen some mentions on this list of interfaces between Faust and Csound, but nothing yet on Syfala so perhaps there are synergies here for cross pollination in future.
In the mean time I’m trying to work out how to add Zedboard to the Syfala supported boards as it’s really very similar to the Zybo, more expensive but it was all that was available when I started down the Zynq route.
Anyway, great work Aman, I fully appreciate the effort you have put into this!
Regards
Ges
I know this is a future (and challenging) topic but just to follow up on Ges’s idea.
Ability to design a FPGA (without running soft or hard core processors in it) using CSound would of course bring some big benefits in terms of I/O latency reduction, increase in processing parallelisation and also it would allow usage of much smaller (cheaper) FPGA-s (there are of course significant downsides also 
).
But here I think it would make more sense to directly take advantage of High Level Synthesis (HLS) instead of going for specific toolchains because we would have many more options when it comes to choosing a FPGA or a FPGA board.
I’m not 100% sure right now what would be the best or the easiest way to that but if this becomes a topic I would gladly try to contribute 
Best regards,
Lovre
Lovre is of course right that the HLS tools are most likely the way to generate the fpga code, and indeed the Syfala tools do that under the hood. Hand coding fpga logic in, say, Vhdl or Verilog, is not for the faint hearted.
For those unfamiliar, HLS stands for High Level Synthesis and is a feature of the Xilinx tools that can convert suitable C++ code into an fpga block ready to program into the Fpga to be driven from a suitable firmware wrapper.
This is already probably way OT so perhaps we should keep this thread to congratulations to Aman for his excellent demonstration and I for one am waiting in hushed awe of further developments.
I am hugely encouraged by the enthusiastic response this thread has received. There is a long way to go on this road and it may take many paths. If there is anything I can help contribute from my limited fpga experience or need for testing or porting I’ll gladly do what I can but my bandwidth is limited and my availability is better in the winter months when gardening and family needs are lessened.
Ges
Thanks to Ges and Lovre for their comments.
I suppose I should give some context to this work, and then maybe Aman can speak in more detail about the
specifics.
I got interested in FPGAs after learning about their use to implement oscillators at extremely high sampling
rates (25 - 40 MHz) in a couple of hybrid digital-analogue synthesizers. At the time, I was working on a
problem whose solution could be provided by that technology. At that time, the Faust team was also starting
to work with FPGAs (the FAST project) and I contacted Romain Michon to discuss this possible application.
They were only starting, but maybe a year or so later, we got in touch again to see if we could reproduce
their setup here to do some tests. Also the idea of doing a custom codec running on the FPGA for ultra high
sampling rate started to develop.
So we had their system running here, and I believe at the time it was the only other place outside, independent from
their project that had Faust and Syfala running. We also collaborated on a paper on the codec idea. When
Aman came to study here, I suggested to him to look at this and he worked with Faust and Syfala to start with.
We then began to wonder about the possibilities of using HFS directly, independent of that system. I thought
it would be an interesting comparison. So he worked at that and got a proof of concept, it was fairly hard because
there was virtually no documentation in terms of audio processing with HFS, and all the information was fairly
dispersed. Outside Syfala, it seems no one else is using that route to do audio programming.
While Syfala provided some pointers, because it is so idiosyncratic, depending on Faust and its constraints,
a lot of it was not relevant to the work. We decided that the best design would follow a modular system, where
IPs could be developed to emulate synthesizer modules (vcos, vcfs, etc) and patched together. This is quite
different from Syfala, where a Faust program is compiled into a single massive IP. It did not make sense to
go that route, since we want to provide means for people to do the translation of audio C++ code into
FPGAs in a more granular way (we are not interested in doing a compiler). The final goal is to have simple
patching means in a drag-and-drop way. The system as designed affords three levels of interaction:
- High: a collection of programs ready to be flashed, users only need to select them.
- Mid: a collection of IPs ready to be patched to make a program.
- Low: a framework for creating new IPs.
Late last year, we were doing an unrelated project, which was to port Csound to run on the Daisy board. That
involved making it so that we could compile libcsound.a and make a program to use it. There were some
difficulties with diagnosing issues because of the limitations of the development environment. While we were
doing that, Aman noted that the same type of ARM CPU was used in the Zybo board, and so we then
switched to using it for that work. Since we had much better tools there, we were able to fix all the problems
and got the port working under both platforms.
That made us wonder what we could do with that. We knew we could run Csound under Linux in that CPU,
the Syfala team was running Linux on the board to support their control program. However, baremetal seemed
to be better suited to our system. So Aman then worked on getting the audio connections between CPU and
FPGA, and after that was working, ported reverbsc as a proof-of-concept.
So now we have some really interesting prospects of marrying the original modular design with Csound. Aman
has reported that running Csound on the Zybo is more performative than comparable-size systems such as
Bela, and we have the FPGA on top of that. We are also only using a single ARM core, there are two on the
board. So potentially we could have two instances of Csound running in parallel.
The idea now is to use the FPGA as a co-processor. This is not really a new thing, we have done things
like that with GPUs way back 10 years. However that was on a regular desktop/laptop platform. The fact
that this is embedded (or embeddable) gives it a different dimension.
So I hope this gives you some more info. Code is available, Aman can give the github link.
best
Wow, great reply and lots of info mirroring many of my own thoughts on how this might all work. I am so pleased someone is working on how to pull this together and that you are already collaborating with the FAST team.
I really like the idea of patching stuff together (vcf/vco etc) as it is more granular than the Syfala approach so has the potential to increase functionality and should enable the optimisation stage of synthesis/instantiation to be able to shuffle and optimise the final logic.
If it’s ok with you I’ll keep in the background as you are much more familiar with the Csound internals than I.
If it helps I will look into porting to Zedboard as that’s what I have to work with.
Congrats on progress so far and keep up the good work.
BTW, are you using the same versions of Xilinx tools as the Syfala team?
Regards
Ges
Yes, we are working with the Zybo (ZyQ 7000), as we first mirrored their setup and then moved on
from there. It’s interesting to hear that you share the opinion that separating the processing into IPs
may facilitate optimisation.
We found that, for one, converting the code to fixed-point was quite
straightforward this way, and it made a big difference regarding resource utilisation.
With a big HLS source, I’d venture that things may become more complex.
Anyway, we have interest in the Zedboard too. Aman will have more to say, I’m sure.
Thanks for the context Victor and sorry now for cross emailing.
“…we are not interested in doing a compiler…” → I agree. Ideally there would be a framework/library to prepare a CSound code for HLS and then everything else should be done automatically. There will be of course very strict limitations like no strings, I/O files and, at least for the beginning, maybe it would make sense to stay with fixed point arithmetic (integers). I could imagine that various DSP algorithms could be used out of the box but organizing available memory usage could be a bit tricky.
“The system as designed affords three levels of interaction…” → something like that would be a dream!
And just briefly to extend the context you provided for those who are not that familiar with FPGA to hopefully gain some interest in it .
FPGAs are in general more expensive as components then DSPs and their development is also few times more expensive, and arguably more difficult, then for DSP so to justify those expenses an application needs to have some specific requirements, like very low latency, high throughput and high processing parallelization on the data coming through, which only FPGA can fulfil.
FPGAs can be used, so to say, in two ways as “standalone” and together with a processor. FPGA with processor design can be realized in three ways:
Extern processor
Intern soft-core processor
Intern hard-core processor
Standalone use-case:
If you want to develop an instrument that is based on one or more high speed sensors and you want to sonify your interaction with them then a FPGA could give you a latency in a micro second order and in this period, you could process sensors inputs quite a bit before putting them on the output pins. This is something that no CPU/GPU/DSP/uC can provide. Amount of parallel processing that would be possible is restricted (mostly) by FPGA resources so lets say we are driven by “better safe than sorry” slogan so we go for a 50€ beast.
FPGA together with a processor:
If your instrument needs to communicate with an outside world by USB/WiFi/Display or you would like to have some more advanced control options, like some presets or logging, then this is where a processor can come as a huge help.
1) Extern processor
If your processor only needs to interact with your ‘FPGA instrument’ in a k-rate so to say then an external processor could be the best option. Going with the same slogan we choose a 10€ beast as an external processor (total cost 60€).
2) Intern soft-core processor
If you want to have a lower latency between your control signals and audio stream or you have some other issues like PCB space shortage, then using an intern soft-core processor is maybe a better solution but this would mean that we need to take a bigger FPGA since a soft-core processor eats quite a bit of resources so let’s say we go for a 100€ FPGA.
3) Intern hard-core processor
Intern soft-core processors are quite slow and less powerful compared to hard-core ones so if you need a low latency with complex DSP algorithms then this is most likely the way to go. This is typically used in machine vision applications or for prototyping before cost reduction comes into play. These FPGAs normally have quite a lot of resources but cost let’s say around 200-400€.
Last lower cost Xilinx (now AMD) FPGA generation, Spartan 7, is easily capable of processing quite a large amount of data in parallel with a 100MHz. But by going down with a speed it is possible to utilize resources to gain more functionality (like trading 50 100MHz oscillators for approx. 200 25MHz oscillators).
This is all very simplified but I hope it’s a bit clearer where FPGAs are nice to have.
This is so interesting. I always wanted to work on audio applications with FPGAs but I always ended up working with optical sensors. I was so jealous of Aman when I read about this project 
.
I’m planning to come to a CSound conference in September so maybe if this will still be a topic of interest we could talk about/brainstorm it over a beer or two.
Best,
Lovre
Interesting, where’s the conference?
Ges
We’re hoping to bring some of this to the conference, even if it’s just a demo.
Excellent, as usual I’ll eavesdrop the conference but wish you the best of luck getting ready for it.
Let me know if I can help in any way.
Ges
Hello Everyone,
Thank you for your messages, kind words and it is great to see the interest in this work!
https://github.com/amanjagwani/FPGAModSynth
Here is a repository with the HLS code for the modules we have been working on. Each one synthesizes into a separate IP core. There are around 10 modules at this stage and we are working on adding more, covering different areas of signal processing and synthesis. However, just to note, at this stage the optimizations we have applied are limited, we are working on this and will update the modules regularly.
These modules, with various options for interconnections through AXI Stream interfaces, run on the programmable logic (FPGA) part of the chip and their initialization, parameter settings and, in-general, control rate operations happen on the processing system (CPU). For the filters we also do the coefficient calculations on the CPU to conserve FPGA resources.
Now with Csound running on the CPU as well, we have a good opportunity to interface csound with accelerated processing through these modules. We are also working on doing that from within Csound itself through plug-in opcodes. For example, if we have an FPGA design such as the one in the video with a ReverbSC module, we can have a reverbscFPGA plug-in opcode that can communicate with the module on the FPGA. We use the AXI DMA driver for sending audio data from the CPU to the FPGA fabric and AXILITE interfaces for sending control signals.
Also, to answer the questions from Ges, we are using the same version of Xilinx tools as the syfala project, currently it is the 2022.2 version. Additionally, porting this system to other boards like the Zedboard should be a little more straightforward since we are working at a more granular level with the separate modules and separate control system. So these HLS modules, for example, can be used with Vitis HLS to synthesize for the Zedboard instead of the Zybo quite easily.
Thank you,
Aman Jagwani