That looks like the UDJ6 device initialization failed. To get an idea of why, please check the error code...
BASS_DEVICEINFO di;
for (int a=1; BASS_GetDeviceInfo(a, &di); a++) {
printf("%d: name=[%s] dev=[%s] flags=%x\n", a, di.name, di.driver, di.flags);
if (BASS_Init(a, 44100, 0, 0, 0)) {
BASS_INFO i;
BASS_GetInfo(&i);
printf("\tspeakers=%d\n", i.speakers);
BASS_Free();
} else
printf("\tinit fail=%d\n", BASS_ErrorGetCode());
}
Perhaps PulseAudio is using the device? If so, the default ALSA device should probably be set to that ("pulse") in the asound.conf file.
Please also confirm whether you have been able to use 5.1 output with any other software on the device, to confirm whether the problem you're having is something that is BASS-specific.