When a BASS function call fails, you can use BASS_ErrorGetCode to find out why (look up the error code in the failed function's documentation). In this case, it may be that an old BASS version is being loaded because the BASS_ATTRIB_VOLDSP option was added in BASS 2.4.17. You can check the loaded BASS version with BASS_GetVersion.
The BASS_ATTRIB_VOLDSP value should be set before calling BASS_ChannelPlay so that it has effect right from the start. Also note that the -5/20 parameter in your pow10 call will be rounded to 0 because it isn't floating-point. You could change it like this:
handle = BASS_StreamCreateFile (False, file, 0, 0, 0);
float vol = pow10((float)-5/20); // convert the replaygain dB gain to a linear value
bRet = BASS_ChannelSetAttribute(handle, BASS_ATTRIB_VOLDSP, vol); // apply it
BASS_ChannelPlay(handle, false);