Topic: Samples and effects?

[RobJellinghaus]

I am working on a live looping application which will capture recorded mike audio via ASIO, copy it to memory buffers, and then create and play samples from the memory buffers, all concurrent with continuing to record new loops.  I'm building a Kinect video-game-like interface, so low latency is critical.  I'm writing it with XNA, C#, and .NET.  (Some details here: http://robjsoftware.org.)

This application could wind up with significant amounts of sampled audio (multiple minutes), and I'd ideally like to get up to as many as ten tracks.  Using stereo float samples at 48KHz, 300 seconds of uncompressed audio would be 115MB of memory, which seems pretty manageable even for a 32-bit .NET application.  This is running on a quad-core Q9300, so there should be sufficient CPU.

I'm making some design assumptions that I would love some feedback on:

- My strong hunch is that I should just go with samples for this application, rather than using decoding streams.  A push stream is probably more convenient for what I want to do, but it would introduce latency when looping (e.g. when rewinding), which I absolutely don't want.  Is this sound thinking?

- The .NET-to-native BASS documentation talks about needing to pin managed buffers when they are being played by BASS.  This raises some tricky memory management questions, since I don't know in advance how long the user might be singing into a particular loop.  I am assuming I can do something like preallocate a significant amount of memory (e.g. have a pool of preallocated 64KB byte[]s on the managed heap), then manage my own copying of data from the ASIO input buffers to the managed byte[]s, and then juggle multiple BASS samples (one per 64KB chunk) to essentially let me play each buffer and then hop to the next with no audible interruptions.  Is this likely to be a feasible approach? 

(The only alternative seems to be to preallocate a longer byte[] than I'm likely to need, and then copy the used portion into a new byte[] once the user closes the loop, but this would lead to much more heap pressure and very large allocations, which would probably kill performance after a while.  Not to mention the large latency of copying megabytes of buffered sample, or the nasty consequences of pinning basically all my multi-megabyte samples for their whole lifetime....)

- The BASS documentation mentions that it's not possible to use DX8 effects with samples.  What's the nature of this restriction?  I would ideally like access to as many low-latency effects as I can get.  If I have to do a background thread and/or dummy channel (or whatever) to apply effects in the background and then save the sampled result, I could live with that.  But it would be better to apply effects directly.  What's the best way to get the maximum effects flexibility with a sample-heavy application?

Thanks for any and all comments -- the fewer false paths I can go down here, the better!  This forum is awesome and I'm glad I picked BASS for this fairly aggressive problem.

[RobJellinghaus]

...and on further reading it looks like there are really a lot of things that samples can't do, e.g. be mixed, have DX8 effects applied, etc.  It also looks like a fully memory-buffered decode stream (e.g. a push decode stream which has had a lot of byte[]s pushed into it) should be pretty fast.  What should I know about the tradeoffs here?

[Ian @ un4seen]

Yes, as you note, samples channels can't have DSP/FX applied to them. The reason for that limitation is that an individual sample (HSAMPLE) may have multiple playback channels (HCHANNEL) sharing the same data, and so applying DSP/FX to the data would affect all of them. But it is possible to replace samples with streams (HSTREAM) without too much compromise. Multiple simultaneous playbacks can be achieved with streams (one for each simultaneous playback) by loading the data to memory and having the streams' STREAMPROC function feed on that. That is slightly less efficient than using samples as there is an extra memory copy operation involved, but nothing very significant.

[RobJellinghaus]

Thanks very much.  Your support on this forum is extraordinary.

One more question:  if I put data into a push stream with BASS_StreamPutData, I assume it has to be pinned?  Since it takes an IntPtr argument, I assume I've got to pin it while I call StreamPutData... but does it have to remain pinned after StreamPutData returns?

The sample code in the static help is:

private DSPPROC _dupCallback;
...
// create stream on device 1
Bass.BASS_SetDevice(1);
int orig = Bass.BASS_StreamCreateFile("test.mp3", 0, 0, BASSFlag.BASS_SAMPLE_LOOP);
Bass.BASS_ChannelPlay(orig, false);
...
// create a clone on device 2
BASS_CHANNELINFO info = Bass.BASS_ChannelGetInfo(stream);
Bass.BASS_SetDevice(2);
int clone = Bass.BASS_StreamCreatePush(info.freq, info.chans, info.flags, IntPtr.Zero);
// pause source stream to synchonise buffer contents
Bass.BASS_ChannelPause(stream);
int c = Bass.BASS_ChannelGetData(stream, IntPtr.Zero, (int)BASSData.BASS_DATA_AVAILABLE);
byte[] buf = new byte[c];
Bass.BASS_ChannelGetData(stream, buf, c);
Bass.BASS_StreamPutData(clone, buf, c);
// set DSP to copy new data from source stream
_dupCallback = new DSPPROC(DupDSP);
Bass.BASS_ChannelSetDSP(orig, _dupCallback, new IntPtr(clone), 0);
Bass.BASS_ChannelPlay(orig, false); // resume source
Bass.BASS_ChannelPlay(clone, false); // play clone
...
private void DupDSP(int handle, int channel, IntPtr buffer, int length, IntPtr user)
{
  Bass.BASS_StreamPutData(user.ToInt32(), buffer, length);
}

That does no pinning at all.  So I am confused about where these buffers are and how they're being managed.  The only way to go from (say) a managed byte[] to an IntPtr seems to be via

fixed (int* bytep = bytes) { Bass.BASS_StreamPutData(user.ToInt32(), new IntPtr(bytep), bytes.Length); }

But is that enough, or does the data have to be pinned subsequently?  And if so, for how long?

Also, is it possible to set the IntPtr passed to BASS_StreamPutData to be somewhere in the middle of a larger buffer, thereby being able to put just a slice of a very long buffer?  If so, that would be awesome :-)

[radio42]

There exist several overloads for the "BASS_StreamPutData" method.

The one which takes an 'IntPtr' as a parameter: Yes, you must 'pin' this address when calling BASS_StreamPutData, but can savely be freed once BASS_StreamPutData returns!

The others take an array as an argument (e.g. a byte[], short[], float[] etc.): When using these overloads there is no need to pin anything, as .Net does this for you already internally.

[RobJellinghaus]

Thanks for replying, Bernd.  Awesome support from you as well.

My main question is why there are no StreamPutData, etc. overloads that take a byte[], a length, and a start index?  In my looping app I plan to allocate large chunks of managed heap (1MB or so), and divide them into smaller buffers that I will allocate myself.  Right now I will probably do

fixed (byte* bufPtr = myBuffer) {
    IntPtr bufIntPtr = new IntPtr(bufPtr);
    IntPtr offsetBufIntPtr = new IntPtr(bufIntPtr.ToInt32() + (index * 4)); // starting index, in floats
    Bass.BASS_StreamPutData(handle, bufIntPtr, length * 4); // length, in floats
}

But if there were a BASS_StreamPutData(int handle, byte[] buffer, int startIndex, int length) API, this would be unnecessary.  In other words, why assume the data I want to put is at the beginning of my byte[]?arrgh, last line should of course have been

    Bass.BASS_StreamPutData(handle, offsetBufIntPtr, length * 4); // length, in floats

[radio42]

I might add such overloads to the next Bass.Net version!

But shouldn't the following function also work - which seems to be a bit easier?

Code: [Select]

public static unsafe int MyBASS_StreamPutData(int handle, byte[] buffer, int startIdx, int length)
{
    fixed (byte* p = &buffer[startIdx])
    {
        return BASS_StreamPutData(handle, new IntPtr(p), length);
    }
}
At least this is how I would implement the overload you mentioned.

[RobJellinghaus]

I may add that as an extension method :-)

I have my buffering code written and I am able to record from the ASIO input using a DSPPROC.  Turns out 256MB of preallocated memory is plenty.  So that part is great. 

My main question now is, what is the most straightforward way to play both the "full-duplex" stream from the ASIO input channel, and the (multiple) recorded tracks?  I am creating those separate tracks with something like:

        public void StartPlaying(HolofunkBass bass)
        {
            bool ok;

            m_bassHStream = Bass.BASS_StreamCreate(48000, 2, BASSFlag.BASS_SAMPLE_FLOAT | BASSFlag.BASS_STREAM_DECODE, m_streamProc, new IntPtr(m_id));
            BASSError error = Bass.BASS_ErrorGetCode();

            BASS_CHANNELINFO info = new BASS_CHANNELINFO();
            Bass.BASS_ChannelGetInfo(m_bassHStream, info);

            BASS_ASIO_CHANNELINFO inputInfo = BassAsio.BASS_ASIO_ChannelGetInfo(true, 0);
            BASS_ASIO_CHANNELINFO outputInfo = BassAsio.BASS_ASIO_ChannelGetInfo(false, 0);

            ok = bass.BassAsioHandler.Stop();
            error = BassAsio.BASS_ASIO_ErrorGetCode();

            for (int a = 1; a < info.chans; a++) {
                ok = BassAsio.BASS_ASIO_ChannelJoin(false, a, 0);
                error = BassAsio.BASS_ASIO_ErrorGetCode();
            }

            ok = bass.BassAsioHandler.Start(1024);
            error = BassAsio.BASS_ASIO_ErrorGetCode();
        }

        // the main ASIO callback - filling the ASIO buffer with sample data to play...
        int StreamProc(int handle, IntPtr buffer, int length, IntPtr user)
        {
            // CopyTo takes a length argument in *floats*, not in bytes.
            return CopyTo(buffer, length / 4);
        }

All the API calls above work -- "ok" is always true -- but my StreamProc is never being called.  I think I am probably getting my ASIO channels and my non-ASIO channels confused.  Any pointers to my obvious mistake will be appreciated :-)

[RobJellinghaus]

...and I should mention that of course BASS_ChannelPlay(m_bassHStream, true); fails with BASS_ERROR_DECODE, because you can't play a decode stream directly.  So what then should I be doing here?  Do I really need a mixer to just play two separate channels?  Also of course, I need to use BASS_STREAM_DECODE because I am using BASS_SAMPLE_FLOAT.

[radio42]

Looks like you are mixing the BassNet "BassAsioHandler" with your own ASIO code?!
Where are you passing your 'StreamProc' to BASSASIO - this is normally done in a BASS_ASIO_ChannelEnable call - which seems to be missing?!

[RobJellinghaus]

You are right that I am mixing BassAsioHandler code and direct BassAsio code :-)  Like I said, I was pretty sure this is all newbie BASS confusion.  Thanks for your patience!

I assume you meant "where are you passing your ASIOPROC to BASSASIO" since BASS_ASIO_ChannelEnable takes an ASIOPROC, not a StreamProc.  Probably the ASIOPROC is being set by the full-duplex handler in BassAsioHandler.

Is it possible that you could share the source to BassAsioHandler?  It would be useful to know exactly what it is doing :-)  Since it seems to be purely a convenience library (e.g. it's using the underlying BassAsio APIs), I can't see where this would be disclosing anything too proprietary?  Still, the documentation is pretty clear about what it's doing (e.g. how SetFullDuplex works).  

So if I need to write my own ASIOPROC to populate the ASIO output stream, and if I want that ASIOPROC to be consuming both the full-duplex input stream and the "m_bassHStream" I'm playing in the code above, what do I do?  Do I need to use a software mixing stream?  Reading between the lines, it looks like ASIO doesn't support hardware mixing; is that right?  So if I want to have multiple tracks playing back together with the full-duplex mike stream, I need BASS software mixing?  

There don't seem to be any BASS.NET samples that demonstrate playing back multiple tracks simultaneously through BassAsio.  I am going to be open sourcing my project on codeproject.net within the next week or so, so if I can get this working, it will be a rather nice BASS example app for this problem :-)

Thanks again for your frankly unbelievably supportive support!

Edit: it looks like you explain exactly what to do in this thread: http://www.un4seen.com/forum/?topic=12661.msg87891#msg87891 -- I'll give that a shot.

[RobJellinghaus]

OK, I really gave this a good try, and it is so close to working it is making me a bit crazy :-)

I have full duplex ASIO microphone input going through a mixer stream to the ASIO output, with a level meter DSP on it, even.  That part is all working as well as it was before.  Here's the setup code:

Code: [Select]

           // not playing anything via BASS, so don't need an update thread
            Bass.BASS_SetConfig(BASSConfig.BASS_CONFIG_UPDATEPERIOD, 0);
            // setup BASS - "no sound" device but 48000 (default for ASIO)
            Bass.BASS_Init(0, 48000, BASSInit.BASS_DEVICE_DEFAULT, IntPtr.Zero);
            BassAsio.BASS_ASIO_Init(0);

            m_mixerHStream = BassMix.BASS_Mixer_StreamCreate(
                48000,
                1,
                BASSFlag.BASS_MIXER_RESUME | BASSFlag.BASS_MIXER_NONSTOP | BASSFlag.BASS_STREAM_DECODE | BASSFlag.BASS_SAMPLE_FLOAT);

            m_asioOutputHandler = new BassAsioHandler(1, 0, m_mixerHStream);

            // // now you can add sources to that mixer...to be played
            // int stream = Bass.BASS_StreamCreateFile(fileName, 0, 0, BASSFlag.BASS_STREAM_DECODE | BASSFlag.BASS_SAMPLE_FLOAT);
            // BassMix.BASS_Mixer_StreamAddChannel(_mixerStream, stream, BASSFlag.BASS_MIXER_DOWNMIX | BASSFlag.BASS_MIXER_FILTER);

            m_asioInputHandler = new BassAsioHandler(true, 0, 0, 2, BASSASIOFormat.BASS_ASIO_FORMAT_FLOAT, 48000);
            m_asioInputHandler.BypassFullDuplex = true; // bypass any output processing
            m_asioInputHandler.SetFullDuplex(0, BASSFlag.BASS_STREAM_DECODE, false);

            BassMix.BASS_Mixer_StreamAddChannel(
                m_mixerHStream,
                m_asioInputHandler.OutputChannel,
                BASSFlag.BASS_MIXER_DOWNMIX | BASSFlag.BASS_MIXER_NORAMPIN);
            m_asioInputHandler.BypassFullDuplex = false; // enable output processing

            m_plmRec = new DSP_PeakLevelMeter(m_asioInputHandler.OutputChannel, 0);
            m_plmRec.Notification += new EventHandler(Plm_Rec_Notification);

            // Register DSPPROC handler for input channel.  Make sure to hold the DSPPROC itself.
            // See documentation for BassAsioHandler.InputChannel
            m_inputDspProc = new DSPPROC(InputDspProc);

            // get the stream channel info
            BASS_CHANNELINFO info = new BASS_CHANNELINFO();
            Bass.BASS_ChannelGetInfo(m_asioInputHandler.InputChannel, info);

            // set up our recording DSP -- priority 10 hopefully means "run first first first!"
            Bass.BASS_ChannelSetDSP(m_asioInputHandler.OutputChannel, m_inputDspProc, new IntPtr(0), 10);

            m_asioInputHandler.Start(512);
            m_asioOutputHandler.Start(512);
I am definitely able to record float values through the DSPPROC, like so:

Code: [Select]

// Consume arriving audio in the ASIO input buffer, and copy it into m_currentRecordingTrack
        void InputDspProc(int handle, int channel, IntPtr buffer, int lengthBytes, IntPtr user)
        {
            if (lengthBytes == 0 || buffer == IntPtr.Zero) {
                return;
            }

            if (!IsRecording) {
                return;
            }

            // float samples are 4 bytes per
            Sample<float> sample = m_samplePool.GetSample(lengthBytes / 4);

            sample.CopyFrom(buffer);

            m_currentRecordingTrack.Append(sample);
        }  
Sample<float> is a struct that wraps a managed float[], including a start index and a length.  It basically refers to a slice of a large float[].  I can stop in the debugger and see that all kinds of float data is being recorded into the backing float[], and that the index updating seems to be working.  To do the copy from the ASIO buffer, I am doing

Code: [Select]

       public static void CopyFrom(this Sample<float> floatSample, IntPtr buffer)
        {
            Marshal.Copy(buffer, floatSample.Chunk.Storage, floatSample.Index, floatSample.Length);
        }
This seems right -- Marshal.Copy deals in units of array elements (not byte counts).  The backing float[] winds up with no obvious gaps or other anomalies, and I am able to record for considerable time.

Once I have recorded a track and want to start playing it, I call the following code:

Code: [Select]

           trackHStream = Bass.BASS_StreamCreate(48000, 2, BASSFlag.BASS_SAMPLE_FLOAT | BASSFlag.BASS_STREAM_DECODE, m_streamProc, new IntPtr(m_id));
            BASSError error = Bass.BASS_ErrorGetCode();

            BASS_CHANNELINFO info = new BASS_CHANNELINFO();
            Bass.BASS_ChannelGetInfo(m_bassHStream, info);

            ok = m_asioInputHandler.Stop();
            ok = m_asioOutputHandler.Stop();

            BASS_CHANNELINFO info = new BASS_CHANNELINFO();
            Bass.BASS_ChannelGetInfo(trackHStream, info);

            ok = BassMix.BASS_Mixer_StreamAddChannel(
                m_mixerHStream,
                trackHStream,
                BASSFlag.BASS_MIXER_DOWNMIX | BASSFlag.BASS_MIXER_FILTER | BASSFlag.BASS_MIXER_BUFFER | BASSFlag.BASS_MIXER_NORAMPIN);

            ok = BassMix.BASS_Mixer_ChannelPlay(trackHStream);

            ok = m_asioInputHandler.Start(512);
            ok = m_asioOutputHandler.Start(512);
m_streamProc is this:

Code: [Select]

       int StreamProc(int handle, IntPtr buffer, int length, IntPtr user)
        {
            // CopyTo takes a length argument in *floats*, not in bytes,
            // and returns the number of floats copied, not the number of bytes!
            return CopyTo(buffer, length / 4) * 4;
        }
This keeps an offset into the float[] that was recorded above, and copies it bit by bit into the buffer.  With the code above, this StreamProc indeed does get called, and I can see the indices advancing and evidently copying correctly (at least I'm not getting illegal access exceptions).

All this is great... except I don't hear the prerecorded track.  I know something is happening internally, because whenever I breakpoint in my StreamProc, then I hear a rapidly looping (stuck) sound, like a single buffer's amount of data looping.  In a fit of desperation I added some hacky code to multiply all the newly recorded float sample values by 100 (clamping to 1), and then I started hearing a periodic clicking, like the sample was clicking every time it looped (and over-mixing beyond 1.0, or something).

Anyway... I am not sure what else to look for, and it's late enough that I'm going to stop here.  Here's hoping that something I'm missing is obvious to you!  I am now going to work on getting this project on CodeProject so you can see the whole source if you like :-)

Edit: OK, I have published the code:  http://holofunk.codeplex.com -- you can go to the Source Code tab and choose Browse to look around, if there are some details you're curious about.

[RobJellinghaus]

Alright, it looks like Bernd has gone away for the weekend, HOW DARE HE!  (that was a joke ;-) )

So I decided to try a couple of experiments.  First, I wanted to get rid of BassAsioHandler, since it was making it harder to tell exactly what was happening.  Second, I wanted to make the looping streams into push streams instead of plain old StreamCreate streams.

Rather than blow out this post by quoting the entire code, which is quite lengthy now, I will just point at the open source links:
HolofunkBass.cs: http://holofunk.codeplex.com/SourceControl/changeset/view/a93240d0db46#Holofunky%2fHolofunkBass%2fHolofunkBass.cs
Track.cs: http://holofunk.codeplex.com/SourceControl/changeset/view/a93240d0db46#Holofunky%2fHolofunkBass%2fTrack.cs

This is working much better -- I am actually hearing the sound I recorded now!

The main problem now is that my stupendous hack, of pushing data into my track streams from -- of all places -- the ASIO input proc, is really not a good hack; the looping audio sounds buzzy, and when there are multiple loops, it sounds even buzzier.  Next thing I will try is using a SYNCPROC on the track streams (at least I think that's where I want it...).

I also hear some soft, frequent clicking just in the input track alone, so things are not perfect there either.

Anyway, it's definitely much closer to working and I am considerably encouraged.  Would still welcome suggestions from any and all.  I will continue updating this thread as matters progress.

I still hope Bernd will consider posting the full BassAsioHandler source code -- it could answer some key questions for me (number one, why the BassAsioHandler full duplex code sounds completely perfect, while my code has this soft clicking going on...).

[RobJellinghaus]

The SYNCPROC is working great.  I have much better-sounding playback now!

The remaining flies in the ointment:

- My input stream is still slightly crackly; got to sort that out.

- The mixing is sounding muddy; levels probably too high, or something.  Not sure exactly what to do there, but experimentation is needed.

- When adding a new track, I stop and then restart ASIO; this causes an audible glitch.  I also have a CPU meter (printing the sum of BASS_GetCpu and BASS_ASIO_GetCpu), and I can see the meter plunge to 0 and then rise back up to 1% - 3%.  I hope there is some way to add tracks to a mixer without having to cause an audible hiccup, or else I'll have to do something like add a bunch of tracks ahead of time and just adjust their volume.  Any tips on this?  I plan to search the forum for clues here.

- And, of course, garbage collections also cause audible glitches; when one happens, the CPU meter spikes to 11% or so, and then goes back down to 1% - 3%.  Obviously this is because the callbacks are not getting serviced quickly enough.  I hope I don't have to go to unmanaged code... I first plan to try to absolutely minimize the amout of garbage my code produces -- CLRProfiler is called for, perhaps.  But if need be, I'll look into the "mixed-mode" solution Bernd outlined in other posts here, though it's seriously complicated by the fact that all my buffers are managed byte[] arrays, and I'm not sure how the mixed-mode code is supposed to get a pinned handle to those without potentially having to synchronize with the GC.  Anyone know the story on that one?

Anyway, progress is good, and I hope Bernd returns to this forum sooner or later, though I'm sure getting the latest BASS release out was (is) his priority.

[Ian @ un4seen]

- My input stream is still slightly crackly; got to sort that out.

That might be the same issue as in this thread...

   www.un4seen.com/forum/?topic=12880

Please see if the BASSmix update posted there helps.

- The mixing is sounding muddy; levels probably too high, or something.  Not sure exactly what to do there, but experimentation is needed.

For a master volume control, you can use BASS_ASIO_ChannelSetVolume to adjust the ASIO output level. The level of the individual mixer sources can also be adjusted via BASS_ChannelSetAttribute (with BASS_ATTRIB_VOL).

- When adding a new track, I stop and then restart ASIO; this causes an audible glitch.

Are you sure you need to restart the ASIO output? It generally shouldn't be necessary to do that in order to add sources to a mixer that's feeding the ASIO output. One time it would be necessary is when the channel configuration needs to change, eg. switching from stereo to 5.1 output (the mixer would need to be recreated then too).

[RobJellinghaus]

The bassmix.dll update there TOTALLY HELPS!  The clicking is gone, gone, GONE.  Beautiful.   It looks like that is bassmix.dll version 2.4.5.4 -- you should push it out officially :-D

Also, you're absolutely right, I didn't need to start or stop anything -- that greatly helps the situation when adding a new track!

I tried setting "Bass.BASS_SetConfig(BASSConfig.BASS_CONFIG_BUFFER, 10);" to minimize mixer latency.  I am definitely hearing some oddity -- it sounds like portions of my tracks are replaying when I would not expect (but only when I have multiple tracks added -- just one sounds fine, but two sounds very odd).  I will be debugging this next.

Also, the documentation says that BASS_CONFIG_BUFFER should be larger than BASS_CONFIG_UPDATEPERIOD, but (following the SimpleAsio example) I have BASS_CONFIG_UPDATEPERIOD set to 0, which I gather is common practice when using ASIO.  I don't exactly understand ASIO in this regard, but it seems that everything is working reasonably well (except for the somewhat garbled mixing) -- even without a single ChannelUpdate call in my app.

Is 10 (msec) a reasonable setting for BASS_CONFIG_BUFFER?  This is a reasonable quad-core machine (Intel Core 2 Quad Q9300), so I don't think CPU usage is much of an issue, especially given that I'm not doing any decoding (everything is getting Marshal.Copy'ed around as straight byte arrays).

[Ian @ un4seen]

The BASS_CONFIG_BUFFER setting only apples when using the standard BASS/DirectSound output, so you don't need to bother with that when using ASIO output. In that case, the output buffer size is determined by the BASS_ASIO_Start call's "buflen" parameter. You can still set BASS_CONFIG_UPDATEPERIOD (or BASS_CONFIG_UPDATETHREADS) to 0 though, to disable the BASS update thread, as it isn't needed.

Good to hear the BASSmix update sorted your crackliness. It should be officially released shortly