I'm not sure if there is an OGG ACM codec available. Windows does come with an MP3 ACM codec, but that is limited to a sample rate of 22050 Hz. If that's fine, you can use it something like this...
DWORD formlen=BASS_Encode_GetACMFormat(0, NULL, 0, NULL, 0); // get suggested ACM format buffer size
void *form=malloc(formlen); // allocate the format buffer
formlen=BASS_Encode_GetACMFormat(channel, form, formlen, NULL,
MAKELONG(BASS_ACM_SUGGEST|BASS_ACM_RATE|BASS_ACM_CHANS, WAVE_FORMAT_MPEGLAYER3)) // get MP3 format info (remove BASS_ACM_SUGGEST to let user choose)
BASS_Encode_StartACM(channel, form, 0, EncodeProc, NULL); // set an MP3 encoder on the recording
Regarding BASS_StreamCreateFileUser, that is indeed for decoding/playback (BASSWASAPI isn't required though). You mentioned that you will be handling the network data transfer yourself. You would then need some way to decode the data on the receiving end, and using BASS_StreamCreateFileUser is the way. VB6 could be an issue though. I'm not very familiar with VB6, but I believe it isn't very multi-threading/callback friendly, and BASS_StreamCreateFileUser requires callback functions (to supply the file data). If that does turn out to be a problem, perhaps you could use VB.Net instead? Or perhaps you could use BASSenc's built-in server support via BASS_Encode_ServerInit and use BASS_StreamCreateURL on the client side?