Author Topic: Streaming incoming audio data in realTime  (Read 487 times)

soundgals

  • Posts: 65
Streaming incoming audio data in realTime
« on: 27 Mar '24 - 09:12 »
Hi Ian,

I tried to PM you on this as it's very specific to my app and the features of BASS I'm trying to use are working fine. Although I got the response that the message was sent successfully it doesn't show up in my message space. So I'm posting it here on the public forum.

I’d like some advice on the following…

What I’m trying to achieve is to stream out in realtime, lossless audio, as I’m receiving it via an audio input on the users system. I think this should be possible, because internet radio streams exist in flac format, which can be played by players that support http lossless streams. Here are the steps I’ve taken so far…

1/ Use BASS_Record_Start with a callback function to receive the data:
Code: [Select]
rChannel = BASS_RecordStart(DWORD(freq), DWORD(channels), 0, recordCallbackBridge, nil)
2/ Add an encoder to encode the incoming audio to flac:
Code: [Select]
encoder = BASS_Encode_FLAC_Start(rChannel, nil,  DWORD(BASS_ENCODE_AUTOFREE), nil, nil)
3/ The callback function is bridged from a c bridging function to my Swift function, as I found this necessary to get it working.

4/ Within the Swift callback, which I’ve named “myRecordProc”, I’m receving the data ok and can write it to a file if I wish. I could either write the file within the callback, or I could call BASS_Encode_FLAC_StartFile within the calling function to write the file.

5/ I don’t wish to write to a file though, instead as already mentioned I want to stream it out to an external player on the user’s lan that supports lossless http streams. No streaming is done outside of the user’s lan.

So far all appears to be working well as far as BASS is concerned.

Where I’m getting stuck is on how to stream it out to the external app, using the http server that’s built into my app. I already serve complete files via this server to the external player and that works well. What I’m attempting to do is modify the code I use to serve up the data for a complete file, to serve streams instead.

I’ve got as far as loading the URI to the external app. It loads; but I get the error “CB Flac lost sync” and the stream doesn’t commence playing.

I’m loading the URI within the callback function where the data is collected. What I don’t understand is how to get my http server to respond to requests for data as it arrives.

At the moment I’m incrementally increasing a Swift data variable as the data comes in, and passing the data to my function, which should respond to the http requests. I’m doing this in the hope that it would be served in chunks to the external player, and the player will determine how much of the data it wants to consume on each request. Once I’ve got a working solution in place I will want to remove any data that’s already been served. of course, otherwise my app’s memory usage would continue to mount.

I think I need to make use of the range header to control how much data is consumed on each request; but I’m failing to understand how to correctly make use of this.

Here are the headers I’ve attempted to use in the function that should handle the http requests…

   
Code: [Select]
response.headers["Accept-Ranges"] = "Bytes"
            response.headers["Cache-Control"] = "no-store"
            response.headers["Transfer-Encoding"] = "chunked"
//            response.headers["Content-Length"] = "*"
            response.headers["Content-Range"] = "0-12348/*"
            response.headers["Content-Type"] = "audio/flac"
            response.headers["X-Sample-Rate"] = "44100"
            response.headers["X-Bits-Per-Sample"] = "16"
            response.headers["X-Number-Of-Channels"] = "2"
//            response.send(data:data)
            value = true
            print("Served amount", " " + String(data.count))
            served = value
            response.headers.append("Custom-Header", value: "HeaderValue")
           do{
               try response.send(data:data).end()
           } catch {
               print("No data sent")
           }
            next()


The “Content-Length” header is commented out, because if I include it the URI doesn’t even get loaded to the external player. The values in the “Content-Range” are just a guess. I have also tried setting the upper limit to the length of the incoming data buffer; but that didn’t work either. I’m also unclear as to whether to use “response.send(data:data).end on each request or just “response.send(data:data)”. I’ve tried both; but neither worked.

Perhaps I should be taking a different approach, and maybe there are other features of BASS I could be making use of, that would be more likely to work.

Any advice you can provide me with on this feature will be much appreciated. It’s something I would really like to get working before the first release of my app.

Thanks in advance.

Geoff

Ian @ un4seen

  • Administrator
  • Posts: 25780
Re: Streaming incoming audio data in realTime
« Reply #1 on: 27 Mar '24 - 12:38 »
When streaming FLAC data, it should be in an Ogg container. You can get that by including "--ogg" in the BASS_Encode_FLAC_Start call options.

If you have no luck with your own server implementation then BASSenc includes one that you could use. Please see BASS_Encode_ServerInit and its documentation.

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #2 on: 27 Mar '24 - 13:01 »
Unfortunately the ogg container doesn't help and I prefer to use my own server for some of its particular security features. I will check the documentation of BASS_Encode_ServerInit though. Thanks.

Since my last post on this, I realised I made a mistake in thinking I could write to a file from the data in the callback function. The bytes appear to be written; but the file is not playable.

I can only create playable files from BASS_Record_Start when I follow it with an BASS_Encode_FLAC_StartFile, passing a URI path in the file system.

So I started wondering if I was really getting the data in the callback function. I added a
Code: [Select]
BASS_ChannelGetData(DWORD(rChannel), UnsafeMutableRawPointer(mutating: buffer), 0xFFFFFFF)
… to see if it made any difference.

Now the error from the external app has changed from "no sync" to "failed to open URI". So it indicates that I do need to use BASS_ChannelGetData somewhere.

If so, should this be in the callback function or the calling function containing "BASS_Record_Start"?, and how do I allocate a buffer when the length of the content is unknown?

« Last Edit: 27 Mar '24 - 13:13 by soundgals »

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #3 on: 27 Mar '24 - 14:07 »
… update. I now have this working very well using BASS_Encode_ServerInit. Thanks very much, once again, for your help Ian.

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #4 on: 27 Mar '24 - 21:01 »
Just one point I'd like clarified. Am I right in saying that recording channels are limited to 16bit?

If so, that would be a pity, since all incoming sample rates appear to be supported.

Of course I can set the BASS_SAMPLE_FLOAT flag on the recording channel and then set the BASS_ENCODE_FP_24BIT on the flac decoder to change the floating point samples back to integer; but what would be the point if the incoming recorded audio bit size is limited to 16bit?

And when I set those two flags as above the sound is broken up. I'd rather not have to truncate the incoming audio to 16 bits when it's 88200/96000 sample rate, or even 44100/48000 at 24 bit.

…; but maybe there is a way of specifying more than 16 bits for the incoming audio data on the recording channel and I'm just missing it?

Chris

  • Posts: 2203
Re: Streaming incoming audio data in realTime
« Reply #5 on: 27 Mar '24 - 22:07 »
you can try this
rChannel = BASS_RecordStart(DWORD(freq), DWORD(channels), BASS_SAMPLE_FLOAT, recordCallbackBridge, nil)// will record in 32Bit Float
// Flac will not supported Floating Point so convert it to 32Bit Integer
encoder = BASS_Encode_FLAC_Start(rChannel, nil,  DWORD(BASS_ENCODE_FP_32BIT or BASS_ENCODE_AUTOFREE), nil, nil)

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #6 on: 28 Mar '24 - 09:27 »
Thanks very much Chris.

I guess this means recording channels are not limited to 16bits? I was going by this page: https://www.un4seen.com/doc/#bass/BASS_RECORDINFO.html from the documentation though, where the maximum mentioned is "16bit". Does it mean we are affectively up-sampling when we add the "BASS_SAMPLE_FLOAT" flag?

Anyway your suggestion seemed to work for 96khz/24bit originating material, as long as I used the "BASS_ENCODE_FP_24BIT" in the encoder, (the "BASS_ENCODE_FP_32BIT" flag results in broken up sound). Unfortunately, after a while I got broken up sound whenever I used the "BASS_SAMPLE_FLOAT" flag and subsequent "BASS_ENCODE_FP_24BIT" flag in the encoder. So I still think that recording channels must be inherently limited to 16bit and any attempt to push the resulting encoded audio over this leads to problems.

Geoff

« Last Edit: 28 Mar '24 - 09:53 by soundgals »

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #7 on: 28 Mar '24 - 10:47 »
After further testing I believe the broken up sound is due to limited resources on my Mac Air and perhaps the LAN. This is evident when I check the resources being used via xCode. I will do further testing at the weekend on my Mac Studio

Ian @ un4seen

  • Administrator
  • Posts: 25780
Re: Streaming incoming audio data in realTime
« Reply #8 on: 28 Mar '24 - 12:54 »
I guess this means recording channels are not limited to 16bits? I was going by this page: https://www.un4seen.com/doc/#bass/BASS_RECORDINFO.html from the documentation though, where the maximum mentioned is "16bit". Does it mean we are affectively up-sampling when we add the "BASS_SAMPLE_FLOAT" flag?

Those BASS_RECORDINFO "formats" flags date back to the old Windows days. They aren't set on modern Windows (Vista and above) or other platforms. When the BASS_SAMPLE_FLOAT flag is used with BASS_RecordStart, it will request floating-point data and the OS will convert from the native format if necessary. On macOS, you can see (and set) what the native format is in its "Audio MIDI Setup" app.

After further testing I believe the broken up sound is due to limited resources on my Mac Air and perhaps the LAN. This is evident when I check the resources being used via xCode. I will do further testing at the weekend on my Mac Studio

Is resource/memory usage rising? If so, are you still retaining the FLAC data in memory via a callback? That won't be necessary for streaming purposes. When using BASSenc_FLAC with the BASSenc server, you basically just need these calls:

Code: [Select]
recorder = BASS_RecordStart(freq, chans, BASS_SAMPLE_FLOAT | BASS_RECORD_PAUSE, RecordProc, 0); // setup recording (paused)
encoder = BASS_Encode_FLAC_Start(recorder, "--ogg", BASS_ENCODE_FP_24BIT | BASS_ENCODE_QUEUE, 0, 0); // setup encoding
BASS_Encode_ServerInit(encoder, ...); // setup serving
BASS_ChannelStart(recorder); // start it

...

BOOL CALLBACK RecordProc(HRECORD handle, const void *buffer, DWORD length, void *user)
{
return true; // continue recording
}
« Last Edit: 28 Mar '24 - 17:42 by Ian @ un4seen »

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #9 on: 28 Mar '24 - 14:21 »
Quote
On macOS, you can see (and set) what the native format is in its "Audio MIDI Setup" app.

Thanks for the clarification Ian. Yes I use Audio MIDI Setup all the time to check and set devices.

For this particular input device the bit depth is set at "32-Bit" float and can't be changed.

Quote
Is resource/memory usage rising? If so, are you still retaining the FLAC data in memory via a callback? That won't be necessary for streaming purposes. When using BASSenc_FLAC with the BASSenc server, you basically just need these calls:

Actually memory usage is holding steady at around 40mb, CPU is on the mid to high thirties. Energy impact shows "High".

I no longer retain flac data in the callback. It just has this line now…

Code: [Select]
return (BASS_Encode_IsActive(handle) != 0)
I've since added a switch on the interface so 16bit or 24bit can be selected.

When 24bit is selected I use the "BASS_SAMPLE_FLOAT" in "BASS_Record_Start" and "BASS_ENCODE_FP_24BIT" in the encoder. When 16bit, I don't

Memory and CPU don't really change significantly when I switch to 24bit; but playback becomes unreliable when using my Mac Air. So I've concluded the extra BASS_SAMPLE_FLOAT processing must be having some effect.


soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #10 on: 28 Mar '24 - 15:31 »
I've just realised that if I remove the "BASS_ENCODE_FP_24BIT from the encoder; but leave the "BASS_SAMPLE_FLOAT" from the record channel, I get 16bit data. This plays with far fewer interruptions; but not as reliably as without "BASS_SAMPLE_FLOAT".

Why though, am I getting 16bit data? When I read this in the documentation; "If none of the floating-point conversion flags are specified, then one will be applied automatically based on the channel's origres value…" it leads me to believe I should be receiving 32bit floating point data, because that's what the device is set too?

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #11 on: 28 Mar '24 - 15:37 »
Here is what the recording channel info looks like. I don't know if that tells us anything about the origres?

Ian @ un4seen

  • Administrator
  • Posts: 25780
Re: Streaming incoming audio data in realTime
« Reply #12 on: 28 Mar '24 - 17:44 »
"origres" is currently always 0 (undefined) on recording channels. It could be nice to have the device's native format in there - I'll look into that. But it wouldn't really help in your case if your device's native format is floating-point, because the FLAC format doesn't support floating-point. So you would still need to specify a conversion flag in the BASS_Encode_FLAC_Start call (or accept the 16-bit default).

One thing I forgot to include in the example code snippet above is the BASS_ENCODE_QUEUE flag. I've added it now. It's generally a good idea to use that flag when encoding a playback or recording.

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #13 on: 28 Mar '24 - 19:48 »
Quote
I'll look into that. But it wouldn't really help in your case if your device's native format is floating-point, because the FLAC format doesn't support floating-point.

Thank Ian, I'll appreciate you looking into that. Correct me if I'm wrong; but reading the documentation for BASS_CHANNELINFO structure here, https://www.un4seen.com/doc/#bass/BASS_CHANNELINFO.html it states for origres…
"The original resolution (bits per sample)... 0 = undefined. If the original sample format is floating-point then the BASS_ORIGRES_FLOAT flag will be set and the number of bits will be in the LOWORD.

This leads me to believe that because my device is showing in Audio Midi Setup as 32bit-float, it will be sending 32bit - float data, on its input. So according to the above the original sample format will be floating-point and should have the BASS_ORIGRES_FLOAT flag set and the number of bits should be in the LOWORD.

As you've explained though, currently for recording channels the origres is undefined, so will be treated as 16bit integer and the above won't be happening.

So although I agree with you that the flac format doesn't support floating point, surely before we set the "BASS_SAMPLE_FLOAT" on the recording channel and subsequently decode back to integer format in the encoder with the "BASS_ENCODE_FP_24BIT" flag, we ought to be getting that BASS_ORIGRES_FLOAT flag set; etc.?

Could this be causing my problems with broken up playback?

Ideally, since we know what a device will be sending, we should be able to set this somewhere, so origres on a recording channel will agree with what is being sent?

I will set the BASS_ENCODE_QUEUE flag as per your advice.

Thanks again.

Ian @ un4seen

  • Administrator
  • Posts: 25780
Re: Streaming incoming audio data in realTime
« Reply #14 on: 29 Mar '24 - 13:33 »
The "origres" value tells what the resolution/bitdepth of the sample data was before any conversion (eg. due to the BASS_SAMPLE_FLOAT flag). In other words, it tells what the data was rather than what it is. It's purely informational and has no bearing on any BASS processing, except that BASSenc_FLAC will use it to determine a default conversion for floating-point data.

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #15 on: 29 Mar '24 - 13:53 »
Ok thanks. My problem though is that whereas I have perfectly reliable playback at 16bit, I don't at 24bit, regardless of the sample rate.

I actually have no particular desire to use "BASS_SAMPLE_FLOAT"; but it seems to be the only way to get audio data that was originally 24bit to playback at 24bit (although unreliably).

The incoming data is treated as 16bit even though it's actually 24/32bit.

So how do I get the rChannel to recognise the incoming audio data at its actual bit depth, rather than always assuming 16bit, before it gets to the encoder?

If I take away the "BASS_SAMPLE_FLOAT" I only get (the assumed) 16 bit data, despite using "BASS_ENCODE_FP_32BIT" on the encoder. (32 bit float is what the device should be feeding in, according to Audio Midi Setup.
« Last Edit: 29 Mar '24 - 14:13 by soundgals »

Ian @ un4seen

  • Administrator
  • Posts: 25780
Re: Streaming incoming audio data in realTime
« Reply #16 on: 29 Mar '24 - 15:07 »
My problem though is that whereas I have perfectly reliable playback at 16bit, I don't at 24bit, regardless of the sample rate.

It could be that the server's buffer is too small for the 24-bit FLAC data (which will naturally be larger than 16-bit). What "buffer" parameter are you currently using in your BASS_Encode_ServerInit call? Try increasing it. If the problem persists, also confirm how you're playing the FLAC stream. If you aren't using BASS with the BASSFLAC add-on then please try that for comparison (eg. using the NETRADIO example included in the BASS package).

The incoming data is treated as 16bit even though it's actually 24/32bit.

So how do I get the rChannel to recognise the incoming audio data at its actual bit depth, rather than always assuming 16bit, before it gets to the encoder?

The data will only be treated as 16-bit by BASS_Encode_FLAC_Start if you don't use a BASS_ENCODE_FP flag to tell it how to treat floating-point data. The recording channel is still floating-point regardless, assuming you set BASS_SAMPLE_FLOAT on it.

soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #17 on: 29 Mar '24 - 16:40 »
Quote
It could be that the server's buffer is too small for the 24-bit FLAC data (which will naturally be larger than 16-bit). What "buffer" parameter are you currently using in your BASS_Encode_ServerInit call?

Oh, I didn't think about that. I was just concentrating on getting it working. So I just used the values from the examples in the documentation. Both "buffer" and "burst" are at 64000.

Would you suggest I increase both, or just the buffer? I'll try doubling the size, unless you have a particular value in mind for 24bit data.

Thanks again.

I'll try increasing the buffer first.




soundgals

  • Posts: 65
Re: Streaming incoming audio data in realTime
« Reply #18 on: 29 Mar '24 - 17:04 »
Terrific! Doubling the buffer did the trick. At least for a 96khz/24bit incoming stream. I'll test 192khz next and if necessary apply a larger buffer for that.

I'll probably use different buffers for different incoming sample frequencies as the longer buffers obviously take a little longer before playback commences.

I already have the BASSFlac add on loaded. Though I haven't made use of it yet. Will look into that further.

Thanks again Ian.