Topic: Modify FFT while playback

[Lowdown]

Hi!

I had been working on small service that will modify mp3 samples. Our test software should be able to modify mp3 samples while playback.
I've used and tried different approaches which didn't yield result. My question is, is it possible to do that at all?

My last approach is to create playback stream BASS_StreamCreateFile and BASS_StreamCreatePush (same freq,chans). Mute mp3 stream and copy modified data to push stream with timer set to trigger every 1ms.
This approach works, but there is some strange quality drop, playback lag even without any modification to sample data. I have tried using FFT 32768, all bellow are unusable.

I did try using BASS_DATA_AVAILABLE and then copy data to stream, however, same issue as above.

So for example, I would like to completely mute 1k-3k Hz samples, I know how to get the slot, change FFT to 0 but not sure why the sound from push stream sounds so poor.

Additional questions:
1. Bass.BASS_StreamPutData(handle, buffer, buffer.Length / 4) => Why we divide length with 4 ?
2. Bass.BASS_ChannelGetData(handle, buffer, (int)BASSData.BASS_DATA_FFT32768); => What amplitudes represent when going into minus (e.g. -0.023)
3. Bass.BASS_ChannelGetData(handle, buffer, (int)BASSData.BASS_DATA_FFT32768); => Can we get MONO from this call?

[Ian @ un4seen]

I had been working on small service that will modify mp3 samples. Our test software should be able to modify mp3 samples while playback.
I've used and tried different approaches which didn't yield result. My question is, is it possible to do that at all?

My last approach is to create playback stream BASS_StreamCreateFile and BASS_StreamCreatePush (same freq,chans). Mute mp3 stream and copy modified data to push stream with timer set to trigger every 1ms.
This approach works, but there is some strange quality drop, playback lag even without any modification to sample data. I have tried using FFT 32768, all bellow are unusable.

If your processing isn't changing the amount of data then perhaps you can do it in a DSP callback via BASS_ChannelSetDSP? If the processing requires a certain amount of data, you can also use the BASS_ATTRIB_GRANULE option via BASS_ChannelSetAttribute to specify that.

Otherwise, your current approach could be improved by making the original stream a "decoding channel" via the BASS_STREAM_DECODE flag, which allows you to use BASS_ChannelGetData to fetch the required amount of data whenever needed. You could still feed that data to a "puch" stream, or perhaps create a "pull" stream with BASS_StreamCreate and have its STREAMPROC callback fetch the data from the source.

Please see the documentation for details on the mentioned functions/options.

1. Bass.BASS_StreamPutData(handle, buffer, buffer.Length / 4) => Why we divide length with 4 ?

The BASS_StreamPutData "length" parameter is in bytes, so it shouldn't usually be necessary to divide by 4. Are you sure it is necessary in your case? If so, perhaps the provided buffer is only 25% full?

2. Bass.BASS_ChannelGetData(handle, buffer, (int)BASSData.BASS_DATA_FFT32768); => What amplitudes represent when going into minus (e.g. -0.023)

The FFT values shouldn't ever be negative (unless the BASS_DATA_FFT_COMPLEX flag is set). What index/bin are you seeing negative values at? Note the returned amount of data is half the FFT size, eg. 16384 values for a 32768 sample FFT.

3. Bass.BASS_ChannelGetData(handle, buffer, (int)BASSData.BASS_DATA_FFT32768); => Can we get MONO from this call?

The returned FFT data is mono by default. The BASS_DATA_FFT_INDIVIDUAL flag can be used to request separate FFT data for each channel, eg. left and right in stereo.

[Lowdown]

Hi Ian,
Thanks for quick response!

1. Bass.BASS_StreamPutData(handle, buffer, buffer.Length / 4) => Why we divide length with 4 ?

You are correct. I had to divide by 4 to get proper output in one case, new case I do not have to do that .. I'm a bit confusing here as method summary says I need to?

2. Bass.BASS_ChannelGetData(handle, buffer, (int)BASSData.BASS_DATA_FFT32768); => What amplitudes represent when going into minus (e.g. -0.023)

Again, I'm not sure why is this happening, I do get negative values ... any idea ?


At the moment I have "decoding" channel and "push" channel. Reading from one inserting in "push" works fine I can continue from there. What I'm now interested is how can I do frequency clipping?
e.g. I want to remove everything bellow 50Hz and above 3 kHz. I tried this method, but I'm getting 'clicking' sound and in mp3 spectral analyze I still have entire frequency?

Code: [Select]

var startSlot = fftValue * 50 / info.freq;
var endSlot = fftValue * 3000 / info.freq;
for (int i = 0; i < startSlot; i++)
 encBuffer[i] = 0;
for (int i = endSlot; i < encBuffer.Length; i++)
 encBuffer[i] = 0;
I have also tried to get 440Hz tone by modifying this data to set all slots to 0 and only 440Hz slot to 1, but this again produces clicking sound and not single fixed tone. (works with Sample, but I would like to achive this with stream FFT modifications)

[Ian @ un4seen]

1. Bass.BASS_StreamPutData(handle, buffer, buffer.Length / 4) => Why we divide length with 4 ?

You are correct. I had to divide by 4 to get proper output in one case, new case I do not have to do that .. I'm a bit confusing here as method summary says I need to?

That looks like it may be a typo in the BASS.Net docs. It probably meant to say buffer.Length*4 for converting samples to bytes.

2. Bass.BASS_ChannelGetData(handle, buffer, (int)BASSData.BASS_DATA_FFT32768); => What amplitudes represent when going into minus (e.g. -0.023)

Again, I'm not sure why is this happening, I do get negative values ... any idea ?

That BASS_ChannelGetData call doesn't include an FFT flag, so it will be giving PCM sample data rather than FFT data. PCM sample values can indeed be negative.

At the moment I have "decoding" channel and "push" channel. Reading from one inserting in "push" works fine I can continue from there. What I'm now interested is how can I do frequency clipping?
e.g. I want to remove everything bellow 50Hz and above 3 kHz. I tried this method, but I'm getting 'clicking' sound and in mp3 spectral analyze I still have entire frequency?

Code: [Select]

var startSlot = fftValue * 50 / info.freq;
var endSlot = fftValue * 3000 / info.freq;
for (int i = 0; i < startSlot; i++)
 encBuffer[i] = 0;
for (int i = endSlot; i < encBuffer.Length; i++)
 encBuffer[i] = 0;
I have also tried to get 440Hz tone by modifying this data to set all slots to 0 and only 440Hz slot to 1, but this again produces clicking sound and not single fixed tone. (works with Sample, but I would like to achive this with stream FFT modifications)

Please note that streams created with BASS_StreamCreate (or BASS_StreamCreatePush) expect PCM sample data, so FFT data would need to converted back to PCM before passing it to them, ie. an inverse FFT. BASS doesn't include inverse FFT processing, so that would need to be done separately (there are dedicated FFT libraries you can use for that).

If you just want to remove the sound outside a certain band then using a bandpass filter (or lowpass + highpass) via BASS_ChannelSetDSP/FX would be a simpler way to do it. You could first try the BASS_FX add-on's BASS_FX_BFX_BQF effect to set lowpass and highpass filters at the wanted frequencies, something like this:

Code: [Select]

HFX bpfx[2]; // bandpass (lowpass + highpass) FX handles

// set highpass biquad filter on stream
bpfx[0] = BASS_ChannelSetFX(stream, BASS_FX_BFX_BQF, 0);
// set its parameters
BASS_BFX_BQF param;
param.lFilter = BASS_BFX_BQF_HIGHPASS;
param.fCenter = 50;
param.fGain = 0;
param.fBandwidth = 0;
param.fQ = 0.707;
param.fS = 0;
param.lChannel = BASS_BFX_CHANALL;
BASS_FXSetParameters(bpfx[0], &param);
// set lowpass biquad filter on stream
bpfx[1] = BASS_ChannelSetFX(stream, BASS_FX_BFX_BQF, 0);
// set its parameters
param.lFilter = BASS_BFX_BQF_LOWPASS;
param.fCenter = 3000;
param.fGain = 0;
param.fBandwidth = 0;
param.fQ = 0.707;
param.fS = 0;
param.lChannel = BASS_BFX_CHANALL;
BASS_FXSetParameters(bpfx[1], &param);

These filters are 2nd order, which means the level drops 12dB every octave beyond the cutoff frequency (fCenter). If that isn't steep enough for your purposes then it is possible to implement higher order filters via BASS_ChannelSetDSP instead.

Note the BASS_FX add-on needs to be loaded before you can use effects from it. You can force it to be loaded by calling a function from it, eg. BASS_FX_GetVersion in your initialization code.

[Lowdown]

Thank you for clarification, now I completely understand why there are issues in my tests.

Can you suggest nuget library to convert FFT (from bass) to PCM?

[Ian @ un4seen]

I'm not a .Net user myself, so I haven't tried any FFT libraries for it but they should all have an inverse FFT option. You could also use the FFT library to do the forward FFT part, ie. get PCM data from BASS_ChannelGetData, forward FFT, your processing, inverse FFT. But note you will also need to overlap the blocks to avoid clicks, so it will actually be a bit more complicated than that. I would recommend trying a BASS_ChannelSetDSP/FX solution first to see if that'll work for you.

[radio42]

Regarding the 'typo' - I assume you are correct, it is a copy of the ...GetData function - here we would have to convert samples to byte; e.g. buffer.Length*4 !
I will correct the docs this in the next version.

[Lowdown]

I managed to process PCM from BASS_ChannelGetData using forward and then inverse FFT.
Did some filtering bellow 50Hz and above 3kHz and this is what I've got.



This are clicks that you mentioned I would need to overlap? Any suggestion on how to handle it?

[Ian @ un4seen]

Yes, you would need to overlap the blocks to avoid those clicks. Overlapping will require retaining some input data from each block, so that it can be used again in the next block. For example, if your FFT size is 2048 samples then you could reuse 1024 old samples and fetch 1024 new samples from BASS_ChannelGetData. Those fetched samples will then become the old samples for the next block. After processing the input data, you would then overlap/mix the first 1024 samples of the output with the last 1024 samples of the previous block's output, and retain the last 1024 samples for next time. It might look something like this:

Code: [Select]

// shift previous data to 1st half of input buffer
memcpy(inbuf, inbuf + fftsize / 2, fftsize / 2 * sizeof(float));
// get new data in 2nd half of buffer
BASS_ChannelGetData(decoder, inbuf + fftsize / 2, fftsize / 2 * sizeof(float));
// FFT processing here (on inbuf with output in outbuf)
float outbuf[fftsize];
...
// overlap/mix the output
for (int a = 0; a < fftsize / 2; a++)
output[a] = ((outbuf[a] * a + lastoutbuf[a] * (fftsize / 2 - 1 - a)) / (fftsize / 2 - 1);
// retain 2nd half
memcpy(lastoutbuf, outbuf + fftsize / 2, fftsize / 2 * sizeof(float));

This code is assuming mono (you can modify it for stereo or more if needed). You may also want to more smoothly transition the filter rather than brickwall it with 0s in the FFT data, as the latter may result in audible artifacts.

[Lowdown]

Again, I must be doing something wrong, can you please help.

Withought FFT modifications sound produced is fine. But when I do FFT modifications (clearing bellow 50Hz and above 3 kHz) I'm still getting same issue as in above spectogram.

Here is my code (don't mind dirty code, it is just for testing):

Code: [Select]

var fftSize = 2048;
var encBuffer = new float[fftSize];
var lastoutbuf = new float[fftSize / 2];
while (Bass.BASS_ChannelIsActive(decoder) == BASSActive.BASS_ACTIVE_PLAYING)
{
  var tempBuffer = new float[fftSize/2];
  Bass.BASS_ChannelGetData(_currentStream, tempBuffer, tempBuffer.Length);

  // copy lastoutbuffer to 1st part of full buffer
  Array.Copy(lastoutbuf, 0, encBuffer, 0, fftSize / 2);
  // copy new data to 2nd part of full buffer
  Array.Copy(tempBuffer, 0, encBuffer, fftSize / 2, fftSize / 2 - 1);

  var output = new float[fftSize];

  /*
     here i'm converting encBuffer to FFT, doing modifications, then inverse FFT result to output array
  */

  // overlap/mix the output
  for (int a = 0; a < fftSize / 2; a++)
    output[a] = ((output[a] * a + lastoutbuf[a] * (fftSize / 2 - 1 - a)) / (fftSize / 2 - 1));

  // retain 2nd half of output
  Array.Copy(output, fftSize / 2  - 1, lastoutbuf, 0, fftSize / 2);
  // copy first part of output for stream put data
  Array.Copy(output,0, tempBuffer, 0, fftSize / 2);

  Bass.BASS_StreamPutData(recStream, tempBuffer, tempBuffer.Length);

  // call get data just to move to the end of recorded stream
  Bass.BASS_ChannelGetData(recStream, tempBuffer, tempBuffer.Length);

} // end while


[Ian @ un4seen]

The .Net Array.Length property is in elements rather than bytes, so you will need to multiply by 4 (the size of a "float") for the BASS_ChannelGetData and BASS_StreamPutData calls. For example:

Code: [Select]

  Bass.BASS_ChannelGetData(_currentStream, tempBuffer, tempBuffer.Length * sizeof(float));

Also, the input and output data should be retained separately, ie. don't feed the last block's output to the next block's input. I think this modification should fix that:

Code: [Select]

  // shift last data to 1st part of full buffer
  Array.Copy(encBuffer, fftSize / 2, encBuffer, 0, fftSize / 2);
  // copy new data to 2nd part of full buffer
  Array.Copy(tempBuffer, 0, encBuffer, fftSize / 2, fftSize / 2);

This is assuming that encBuffer still contains the last input data. If not, you will need to retain a copy of it (the 2nd half).

Also be sure to retain the right data here (fftSize / 2 rather than fftSize / 2 - 1):

Code: [Select]

  // retain 2nd half of output
  Array.Copy(output, fftSize / 2, lastoutbuf, 0, fftSize / 2);


[Lowdown]

Alright, got it! Thank you so much!

This is what I got now after FFT modifications (remove below 50Hz and above 3kHz) here I've also added cleaning 600-800Hz so there is a hole there. Do you think it is possible to clean it even more?

[Ian @ un4seen]

Good to hear that you've got it working now.

Are you currently simply zeroing all FFT bins outside of the passbands? If so, you could try going from passband (1) to stopband (0) a bit more gradually over some number of bins.

Also, using a Hann window instead of a triangular window in the output overlapping may help a little:

Code: [Select]

// overlap/mix the output
for (int a = 0; a < fftsize / 2; a++) {
float w = (1 - cos(2 * M_PI * a / fftsize)) * 0.5;
output[a] = outbuf[a] * w + lastoutbuf[a] * (1 - w);
}

The window can/should be precalculated (eg. in a "window" array) to improve performance.