You could use BASS_SYNC_POS syncs to achieve what you want, eg. set a sync at a point that you want to split the encoding, and then in your callback function, stop the old encoder and start a new one. That can be applied to both recording and transcoding. It could look something like this...
QWORD syncbyte=BASS_ChannelSeconds2Bytes(handle, synctime); // byte position to sync/split at
BASS_ChannelSetSync(handle, BASS_SYNC_POS|BASS_SYNC_MIXTIME, syncbyte, SplitSyncProc, NULL); // set a mix-time POS sync there
...
void CALLBACK SplitSyncProc(HSYNC handle, DWORD channel, DWORD data, void *user)
{
BASS_Encode_Stop(channel); // stop the old encoder
BASS_Encode_Start(channel, ...); // start a new one
}