I don't think it'll be possible to merge OGG channels in a lossless way, ie. without decoding and re-encoding. How are you currently merging the channels? If you aren't already using the BASSmix add-on's matrix mixing feature then I would suggest trying that, ie. create a mixer with the required number of output channels and then add the source OGG file to it with a matrix to merge/mix the channels as wanted. You can use the BASSenc_OGG add-on to encode the mix to a new OGG file in memory. It could look something like this:
decoder = BASS_StreamCreateFile(false, inputfile, 0, 0, BASS_SAMPLE_FLOAT | BASS_STREAM_DECODE); // create decoder for source file
mixer = BASS_Mixer_StreamCreate(freq, chans, BASS_SAMPLE_FLOAT | BASS_MIXER_END | BASS_STREAM_DECODE); // create mixer that ends when its source(s) does
BASS_Mixer_StreamAddChannel(mixer, decoder, BASS_MIXER_CHAN_MATRIX | BASS_MIXER_CHAN_NORAMPIN); // plug the decoder into it with matrix mixing enabled
// initialize matrix here
BASS_Mixer_ChannelSetMatrix(decoder, matrix); // set the matrix
BASS_Encode_OGG_Start(mixer, NULL, BASS_ENCODE_AUTOFREE, OggEncodeProc, NULL); // set an OGG encoder on the mixer
for (;;) { // processing loop
BYTE buf[20000]; // processing buffer
int got = BASS_ChannelGetData(mixer, buf, sizeof(buf)); // process the mixer
if (got < 0) break; // done
}
BASS_StreamFree(mixer); // free the mixer (and encoder due to AUTOFREE)
...
void CALLBACK OggEncodeProc(HENCODE handle, DWORD channel, const void *buffer, DWORD length, void *user)
{
// add "buffer" contents to file in memory
}
Please see the documentation for details on the mentioned functions.