Tutorial 2 - Audio programming with gtkIOStream

[flatmax]

gtkIOStream is a versatile software for signal processing and audio processing, amongst other features (such as GUI programming). It interfaces directly with jackd for low latency audio processing and routing. gtkIOStream also provides a nice port monitor interface which shows which jackd ports are in use and how (similar to the qjackctl connect GUI but different).

In this tutorial we will focus on full duplex streaming using jackd. Full duplex is a term given to both audio output and audio input (half duplex means either output or input). This tutorial will also introduce how to start jackd, the JackPortMonitorGui application and show you the playback and capture mixers settings required for the Audio Injector stereo cards to get input and output audio working so that you can hear something on the input at the output of your hardware.

Before we start, you have to have compiled and installed gtkIOStream. For certain components of gtkIOStream its compilation and installation isn't necessary, but for certain parts of audio processing it is necessary. Tutorial 0 and contents guides you through how to install and test the installation of gtkIOStream. Before you start this tutorial, you must have it installed first.

If you haven't already done so, install jackd2 like so :

Code: Select all

sudo apt-get install jackd2

Our first step is to open the standard lightweight Raspberry Pi IDE with a blank file for us to code up in ... like so :

Code: Select all

geany JackClient.C

Now if you have used ssh to get onto your Pi, then you may have to logout and log in with the "-X" flag like so :

Code: Select all

ssh -X pi@raspberrypiIP

In geany, cut and past the following code :

Code: Select all

#include <JackClient.H>
#include <iostream>
using namespace std;

#include <unistd.h> // for sleep

/** This jack client will play back what it captures
*/
class JackFullDuplex : public JackClient {

	/// The Jack client callback - copy input to output
    int processAudio(jack_nframes_t nframes) {
		if (outputPorts.size()!=inputPorts.size()){
			cout<<"Different input and output port count, don't know how to copy. In port cnt="<<inputPorts.size()<<" output port count="<<outputPorts.size()<<endl;
			return -1;
		}

        for (uint i=0; i<outputPorts.size(); i++) { // for each channel of audio, copy input to output
            jack_default_audio_sample_t *out = ( jack_default_audio_sample_t* ) jack_port_get_buffer ( outputPorts[i], nframes ); // the output sample buffer
            jack_default_audio_sample_t *in = ( jack_default_audio_sample_t* ) jack_port_get_buffer ( inputPorts[i], nframes ); // the inputs sample buffer
            for (uint j=0; j<nframes; j++) // rather then copying, you could do some signal processing here!
                out[j]=in[j];
        }
        return 0;
    }
};

int main(int argc, char *argv[]) {
    JackFullDuplex jackClient; // init the full duplex jack client

    // connect to the jack server
    int res=jackClient.connect("jack full duplex client");
    if (res!=0)
        return JackDebug().evaluateError(res);

    cout<<"Jack : sample rate set to : "<<jackClient.getSampleRate()<<" Hz"<<endl;
    cout<<"Jack : block size set to : "<<jackClient.getBlockSize()<<" samples"<<endl;

	// create the ports
    res=jackClient.createPorts("in ", 2, "out ", 2);
    if (res!=0)
        return JackDebug().evaluateError(res);

    // start the client connecting ports to system ports
    res=jackClient.startClient(2, 2, true);
    if (res!=0)
        return JackDebug().evaluateError(res);

	while (1) sleep(10); // sleep forever
    return 0;
}

Lets step through the code in order of execution.
We start with good old main, which has the standard (unused) input variables argc and argv. The first thing main does is instantiate our jackd client JackFullDuplex :

Code: Select all

int main(int argc, char *argv[]) {
    JackFullDuplex jackClient;

The JackFullDuplex is our C++ class which inherits from the gtkIOStream JackClient, so we include this in order to define it :

Code: Select all

#include <JackClient.H>
#include <iostream>
using namespace std;

#include <unistd.h>

Here you will see the JackClient.H file which defines our JackClient. You will also notice that we include iostream and unistd for console output (cout, endl, etc.) and the call to the function sleep respectively.

Next we define our JackFullDuplex class which inherits from the JackClient class :

Code: Select all

class JackFullDuplex : public JackClient {

Now inside this class, we have to overload the "processAudio" method, because that method will be called by jackd every time it has nframes of audio for us to process. It looks like so :

Code: Select all

    int processAudio(jack_nframes_t nframes) {
		if (outputPorts.size()!=inputPorts.size()){
			cout<<"Different input and output port count, don't know how to copy. In port cnt="<<inputPorts.size()<<" output port count="<<outputPorts.size()<<endl;
			return -1;
		}

This class method inputs the number of frames (audio samples for each channel) require processing and returns a number which indicates 0 for success and keep going, and any other number for error and stop calling this jackd client. The first thing we do is confirm that we have the same number of input and output channels (channels are ports in jackd) and if they differ, we print an error to console and return non zero to tell jackd to stop calling us.

Next, as we are a full duplex "copy" client, we take the input audio and copy it directly to the output. We loop over all of our ports (channels) :

Code: Select all

        for (uint i=0; i<outputPorts.size(); i++) { // for each channel of audio, copy input to output
            jack_default_audio_sample_t *out = ( jack_default_audio_sample_t* ) jack_port_get_buffer ( outputPorts[i], nframes ); // the output sample buffer
            jack_default_audio_sample_t *in = ( jack_default_audio_sample_t* ) jack_port_get_buffer ( inputPorts[i], nframes ); // the inputs sample buffer
            for (uint j=0; j<nframes; j++) // rather then copying, you could do some signal processing here!
                out[j]=in[j];
        }

and we acquire the output and input audio buffers for channel i using the jack_port_get_buffer function. We then loop over all the frames using the variable j and copy the input sample to the output sample.
Finally, we return 0 indicating that everything is good and we want to keep processing next time there are nframes of audio samples available :

Code: Select all

        return 0;
    }

Great ! So now you understand how to program a jackd audio client - it is that simple ! Oh ... if you don't want to use the input channels, then simply don't deal with them in the processAudio JackClient method. If you don't want to use the output channels, then simply don't deal with them in the processAudio JackClient method.

OK - what about the rest of main ? In the rest of the main function we tell our JackFullDuplex client (jackClient in the code) to connect to the jackd server :

Code: Select all

    int res=jackClient.connect("jack full duplex client");
    if (res!=0)
        return JackDebug().evaluateError(res);

we return here on error, where the JackDebug::evaluateError method prints the error to console and returns the error number for us to return to the main caller (the shell).

We print information about the sample rate in use and the number of samples per channel per frame (nframes) :

Code: Select all

    cout<<"Jack : sample rate set to : "<<jackClient.getSampleRate()<<" Hz"<<endl;
    cout<<"Jack : block size set to : "<<jackClient.getBlockSize()<<" samples"<<endl;

We want to create ports (channels) for our client, and in our case we will simply work with a stereo full duplex client, so we define 2 input and output ports :

Code: Select all

    res=jackClient.createPorts("in ", 2, "out ", 2);
    if (res!=0)
        return JackDebug().evaluateError(res);

again, returning on error.

We want to start our client and auto-connect our ports to the system input and output ports :

Code: Select all

    res=jackClient.startClient(2, 2, true);
    if (res!=0)
        return JackDebug().evaluateError(res);

and we are at the races ... at this point, jackd starts calling our JackFullDuplex::processAudio client.

Finally we sleep forever and if for whatever reason we stop sleeping, we return 0 :

Code: Select all

	while (1) sleep(10); // sleep forever
    return 0;

That is it ! You have now completed the flow of this jackd client program. We now have to compile and run this program.

We compile it with the following command :

Code: Select all

g++ `pkg-config --cflags --libs gtkIOStream` -o JackClient JackClient.C 

this command should be reasonably familiar to you now, we are compiling the code (JackClient.C) and calling it JackClient.

We have to start jackd before we start our client, if you are using a Pi, you will have to have a sound card installed with audio input. You start jackd with two input and output ports (channels) like so :

Code: Select all

jackd -d alsa -r 48000 -i 2 -o 2

Here we tell jackd to use the sample rate of 48 kHz.
If you don't have a DISPLAY env. variable, jackd will complain and not stare, so either log back in with "ssh -X" (assuming you used ssh to access your term) or declare one first :

Code: Select all

export DISPLAY=:0.0
jackd -d alsa -r 48000 -i 2 -o 2

One of the nice things about gtkIOStream is that it includes a port monitor which is available in both gui and non gui form. Make sure you "ssh -X" into your remote computer (if you are working over a network) so that you can see and execute the following GUI command :

Code: Select all

JackPortMonitorGui

which brings up a gui which looks like so :

JackPortMonitorGui without any jackd clients

JackPortMonitorGui.noClients.png (15.81 KiB) Viewed 7622 times

Note that you can drag and drop ALL the buttons to connect/disconnect ports.

Finally we start our jackd client :

Code: Select all

$ ./JackClient 
Jack : sample rate set to : 48000 Hz
Jack : block size set to : 1024 samples
input port 0 latency = 0
input port 1 latency = 0
output port 0 latency = 0
output port 1 latency = 0
JackClient::bufferSizeChangeStatic : New buffer size = 1024

It tells us the sample rate and block size and some information about latency - which isn't relevant here.

You will now observe that the JackPortMonitorGui app shows that the system and our client are fully connected and looks like so :

JackPortMonitorGui with our JackFullDuplex client connected.

JackPortMonitorGui.withClient.png (25.8 KiB) Viewed 7622 times

You will be able to drag and drop the buttons from the left side to the right side dropping on any buttons on the right to connect ports together. You can disconnect ports by dragging buttons from the right side to the left side dropping them on the ports you want to disconnect.


At this point you can verify that your client is working by plugging in an audio source to the input lines and playing the output lines through an amplifier or pair of headphones.

If you don't hear any audio then either your player isn't playing or your mixer isn't set up correctly. In the case of the audio injector stereo cards, your playback alsamixer should look like so :

Audio Injector playback settings for line input.

AudioInjector.playback.mixerSettings.lineInput.png (29.82 KiB) Viewed 7622 times

Your capture alsamixer setting should look like so :

Audio Injector capture settings for line output.

AudioInjector.capture.mixerSettings.lineInput.png (24.13 KiB) Viewed 7622 times

[pegodk]

Thank you so much for these tutorials! I have been trying for a long time to get duplex to work, but it just wont. I've tried simple ALSA scripts and recently also Jack. I hope your tutorials can finally help me reach my goal.

I have followed your tutorials on a fresh install of Raspbian with your stereo sound card.

- First I installed the sound card following your guide and made sure that ALSA works.
- Then I followed your guide in tutorial 0 and installed all the packages. I can confirm that the script in tutorial 1 works.

- Now, in tutorial 2, I get some problems. If I run the command 'JackPortMonitorGui' I get "bash: JackPortMonitorGui: command not found".
Also if I just run "jackd -d alsa -r 48000 -i 2 -o 2" and "./JackClient ", I get no errors, but there is no sound coming through.

Do I need to setup something for Jack myself? Like edit the .asoundrc file?

Thank you so much for these guides. It is a tremendous help.
Peter

[flatmax]

I certainly do plan to take this to full duplex processing. When I get some more time I will post the next tute.

Regarding your JackPortMonitorGui question, can you confirm you see the following :

Code: Select all

pi@raspberrypi:~ $ which JackPortMonitorGui 
/usr/bin/JackPortMonitorGui

[pegodk]

Thank you for the quick response.

On my newly reinstalled SD OS I get no output from running that command. However, on my old SD card I get the same output as you. I wonder how that can be?

If I just run "JackPortMonitorGui", I get the same error on both SD cards though.

[flatmax]

OK,

First thing, check for JackPortMonitor as well :

Code: Select all

pi@raspberrypi:~ $ which JackPortMonitor
/usr/bin/JackPortMonitor

If nothing is installed, then /usr/include/gtkiOStream will be empty :

Code: Select all

pi@raspberrypi:~ $ ls /usr/include/gtkIOStream/

If nothing is installed, you should probably re-run tutorial 0 to install everything.
If JackPortMonitor is present but JackPortMonitorGui isn't then it is likely that you don't have the necessary gtk libraries installed on your system to compile the GUI components of gtkIOStream. You can see if they are installed by looking at the end of the ./configure command from tutorial 0 :

Code: Select all

./configure --disable-octave
...
...SNIP
...
configure: 
configure: 
configure: 
configure: SOX ............................................ Present
configure: GTK2 ............................................ Present
configure: Jack ........................................... Present
configure: Octave ......................................... Not present
Octave tests and applications will not be built.
configure: Alsa   ....................................... Present

[pegodk]

Dear Flatmax,

If I run $which JackPortMonitor I also get the same output as you (on new SD card).

And I also have all the packages installed in /usr/include/gtkIOStream/

If I run

Code: Select all

$jackd -d alsa -r 48000 -i 2 -o 2

And then

Code: Select all

$JackPortMonitor

I get the following output (second half of the code appears after running ./JackClient)

Code: Select all

JackBase port monitor
JackBase connected
JackPortMonitor::autoConnectNetClientsPorts
Jack : sample rate set to : 48000 Hz
=== 2 input ports, 2 output ports ===
	playback_1		capture_1
	playback_2		capture_2

=== Connections ===
system:
playback_1 ---> {}
playback_2 ---> {}
JackPortMonitor::autoConnectNetClientsPorts
=== 4 input ports, 4 output ports ===
	playback_1		capture_1
	playback_2		capture_2
	in 0		out 0
	in 1		out 1

=== Connections ===
system:
playback_1 ---> {jack full duplex client(out 0)}
playback_2 ---> {jack full duplex client(out 1)}
jack full duplex client:
in 0 ---> {system(capture_1)}
in 1 ---> {system(capture_2)}
JackPortMonitor::autoConnectNetClientsPorts
=== 4 input ports, 4 output ports ===
	playback_1		capture_1
	playback_2		capture_2
	in 0		out 0
	in 1		out 1

If I then run ./JackClient I get:

Code: Select all

Jack : sample rate set to : 48000 Hz
Jack : block size set to : 1024 samples
input port 0 latency = 0
input port 1 latency = 0
output port 0 latency = 0
output port 1 latency = 0
JackClient::bufferSizeChangeStatic : New buffer size = 1024

But there is still no sound. (And JackPortMonitorGui also doesnt work)

Best wishes,
Peter

[flatmax]

OK ,

Regarding audio silence, have you plugged in an audio source to the input RCAs ? Made sure it is playing ?
Also confirmed that your alsa mixer is setup correctly ?

Regarding the lack of GUI, can you confirm what your ./configure --disable-octave command outputs at the very end ?

thanks
Matt

[pegodk]

It works!!

I found out that I actually had a problem to record sound. I guess because I use 2 SD cards it messed with my mind. The solution turned out to be to remove comment in /boot/config.txt

Code: Select all

dtparam=i2c_arm=on

I am sorry that I didn't catch this mistake before.

The output from ./configure --disable-octave is as you say

Code: Select all

config.status: creating gtkIOStream.pc
config.status: creating gtkIOStreamORB.pc
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
configure: 
configure: 
configure: 
configure: SOX ............................................ Present
configure: Jack ........................................... Present
configure: Octave ......................................... Not present
Octave tests and applications will not be built.
configure: Alsa   ....................................... Present

But this GUI is not strictly needed right? I can get similar overview from just JackPortMonitor?

Thank you so much for the help. I cant even describe how awesome that is!

Best wishes,
Peter

[flatmax]

OK - good that you caught the problem.
JackPortMonitor will do the same job (I think) however no drag and drop connections/disconnections will be possible.

Matt

[niyer]

Hi Matt,

I have been using this board for the past few weeks and love it. Thanks for the in-depth tutorials.
The sample rate (jackClient.getSampleRate()) is defaulting to 48000 samples per second. I would like to set it to 44100 samples per second as my input audio and processing is sampled at this rate. I would very much appreciate if you could show me how to change the input sample rate.

I would like to add that I have been running ./JackClient without running the jackd -d alsa -r 48000 -i 2 -o 2 command first. It is working fine with out starting the jackd server. Do you think that is odd ?
The only thing I am missing is the ability to change the sample rate to 44100, which I would like to ideally do inside the JackClient.c code itself.

Regards,
Nathan