First I thought I could just use the IP, but the URL is https. Is there any way to ignore certificate errors?
BASS does already ignore certificate errors
From my experience this is not the case - at least on Windows via bass.net (I would assume, that Bass.Net just passes the calls to Bass), as long as the certificate cannot be validated by the OS, Bass.BASS_StreamCreateURL returns 0 while Bass.BASS_ErrorGetCode() gives BASSError.BASS_ERROR_UNKNOWN. It works as soon as the self signed root certificate is imported.
At the moment it looks like we may have fixed the problem: Wireshark capture logs revealed, that the system in this specific environment receives replies from DNS in milliseconds. In the majority of (if not all) attempts to play the file immediately after our software has been started, and at certain time intervals (we haven't tried to find out how often actually), the OS tried to access ctldl.windowsupdate.com which appears to block BASS_StreamCreateURL. The unpleasant thing is, that the DNS replies with an external IP, but the firewall prevents access. So BASS_StreamCreateURL timesout eventually.
ctldl.windowsupdate.com is used to update some Windows internal certificate lists. Unfortunately, this cannot be circumvented by overriding the IP with the hosts file. However, it seems the certificate updates can be disabled altogether by a small change in registry, eg with this PowerShell snippet:
Set-ItemProperty -Path 'HKLM:\Software\Policies\Microsoft\SystemCertificates\AuthRoot' -Name DisableRootAutoUpdate -Value 1
So far, the timeout has not occurred again - fingers crossed...