My best guess is that BASS is assuming a 5.1 speaker setup when loading a 6 channel ogg, and a 7.1 speaker setup when loading an 8 channel ogg.
BASS does indeed reorder the channels to match its (and Windows') channel ordering for up to 8 channels, which is as follows:
3 channels = front left, front right, center.
4 channels = front left, front right, rear left, rear right.
5 channels = front left, front right, center, rear left, rear right.
6 channels = front left, front right, center, LFE, rear left, rear right.
8 channels = front left, front right, center, LFE, rear left, rear right, side left, side right.
While Vorbis's channel order is as follows:
3 channels = front left, center, front right
4 channels = front left, front right, rear left, rear right
5 channels = front left, center, front right, rear left, rear right
6 channels = front left, center, front right, rear left, rear right, LFE
8 channels = front left, center, front right, rear left, rear right, side left, side right, LFE
The center and LFE channels are in different positions. There isn't currently any way to disable the reordering, but perhaps you can undo it in your matrix?
If the channel splitters are being plugged into the same mixer then it seems to me like they aren't really needed, and you could just plug the original multi-channel stream into the mixer and use one big matrix to enable (and reorder) the channels? For example, your 8 matrices above could be replaced with one like this:
{
{1, 0, 1, 0, 1, 0, 1, 0}, // L1, L2, L3, L4, L5, L6, L7, L8
{0, 1, 0, 1, 0, 1, 0, 1} // R1, R2, R3, R4, R5, R6, R7, R8
}
After reordering, it could look like this:
{
{1, 1, 0, 0, 1, 0, 1, 0}, // L1, L3, L2, L8, L4, L5, L6, L7
{0, 0, 1, 1, 0, 1, 0, 1} // R1, R3, R2, R8, R4, R5, R6, R7
}
