Extension of the library for audio

[ankit6979]

Hi @jeffbass ,
Can the current library be extended to transfer audio feeds as well.

[jeffbass]

Hi @ankit6979,

The imageZMQ library can send audio clips (as-is, without being extended) by using the send_jpg and receive_jpg methods. The 2 jpg methods send / receive (message, binary_buffer) tuples. The binary_buffer can be any arbitrary binary blob, including an audio clip. The test programs and example programs in this imageZMQ repo are sending image frames, but a jpg image is just a binary blob. And audio clips are just a different kind of binary blob.

But note that a library like imageZMQ cannot continuously stream audio. What imageZMQ does is send "frames" which are distinct (message, blob) pairs. The "blob" can be an openCV image in its native format or the blob can be an arbitrary binary blob when using the send_jpg methods. It is important to note that imageZMQ transmits a sequence of "frames" or "binary blobs", but it is not a "continuous streaming" protocol. When imageZMQ sends a "video stream", it is actually sending a sequence of frames, with each frame being an atomic ZMQ message.

Blob sizes can be quite large, but should be kept smaller than about 500MB due to ZMQ internal limits. Here are some rough sizes of some common examples for size comparisons:

• A 1600x1200 RGB frame is about 6MB (1600 x 1200 x 3) in native OpenCV image format
• A 1600x1200 RGB frame compressed to jpg can be between 0.2MB to 2MB depending on what is in the frame
• An audio clip of a 4 minute song is about 4MB (can vary greatly depending on compression; this is for the common MP3 compressed audio format)
• An audio clip of a 30 minute voice podcast is about 10MB to 30MB depending on how much talking is going on; again, this is for the common MP3 compressed audio format

So imageZMQ can definitely send audio clips, but cannot do continuous audio streaming.

Here is example of a modified version of /tests/test_3_rpi_send_jpg to show the sending of some mp3 songs as audio clips as binary blobs (rather than sending jpg blobs):

# use either of the formats below to specify the address of receiving computer
sender = imagezmq.ImageSender(connect_to='tcp://jeff-macbook:5555')
# sender = imagezmq.ImageSender(connect_to='tcp://192.168.1.190:5555')

# this simple example assumes the song files are in the current directory
# 4 songs by The Eagles...you might like something else ;-)
song_files_list = ['take_it_easy.mp3', 'hotel_california.mp3', 'desperado.mp3', 'tequila_sunrise.mp3']
for song_name in song_files_list:
    # song_name is a string; open the mp3 file with that name and read it as binary blob
    song_file = open(song_name, 'rb')  # open song file as read, binary
    audio_clip_blob = song_file.read()  # reads the song file as a binary blob
    # send the song file name as the text part of tuple; send the audio clip as the binary blob part of the tuple
    OK_msg = sender.send_jpg(song_name, audio_clip_blob)

You would modify the tests/test_3_mac_receive_jpg program similarly, either saving the received clips to disk, or adding them to an audio player queue. Or feeding them to a "audio to text" bot. Or to some other audio processing program.

I have used imageZMQ to send audio clips this way and it works well. I used it for "query" / "response" audio clips for an audio bot agent. But those are distinct (relatively small) audio clips, not continuous audio streaming. I should probably update the documentation with some of this audio info and add some audio examples to the /examples folder. I'll put it on my to-do list.

Jeff