Author Topic: BASS_Encode_ServerInit - server stops accepting connections  (Read 482 times)

fmcoder

  • Posts: 485
Hello Ian,

After a while, sometimes couple of hours, sometimes more, bassenc's server stops accepting connections. It looks like it only happens with FLAC. Tried both command line version of FLAC, and bassenc_flac. In both cases, FLAC encoder is started with "--ogg" option.

Why this could happen? What should I check? Latest versions of bass, bassenc, etc are used, it runs under Windows.
« Last Edit: 18 May '23 - 08:34 by fmcoder »

Ian @ un4seen

  • Administrator
  • Posts: 25287
Did you mean to write BASS_Encode_ServerInit (rather than BASS_Encode_CastInit) in the thread title? If so, are you using an ENCODECLIENTPROC callback function with it, and does that stop getting called when the problem happens? Does BASS_Encode_IsActive say the encoder/server is still running at that point?

fmcoder

  • Posts: 485
Yes, sorry, it was _ServerInit function of course. I edited the title.

I'll check if ENCODECLIENTPROC is called and the encoder status, and write here with the results, thank you.

fmcoder

  • Posts: 485
Looks like the problem is a bit different. BASSenc doesn't seem to detect that the port is busy. It's possible to start multiple servers on one the same port, which results only in first one working, and the rest are not. I created a small example, please see the source code below. It starts two servers on the same port, and there's no BASS_ERROR_BUSY error. If I start two instances of the program, second instance also "initializes" two servers with no error (and they are not working).

Code: [Select]
procedure CheckError;
begin
  if BASS_ErrorGetCode <> 0 then
    raise Exception.CreateFmt('error: %d', [BASS_ErrorGetCode]);
end;

function MyServerClientConnectCb(handle: HENCODE;
  connect: BOOL; client: PAnsiChar; headers: PAnsiChar; user1:Pointer): BOOL; {$IFDEF MSWINDOWS}stdcall{$ELSE}cdecl{$ENDIF};
begin
  Result := True;
end;

procedure TForm1.Button1Click(Sender: TObject);
const
  BufSz = 256;
var
  c: HCHANNEL;
  enc1, enc2: HENCODE;
  srv: PAnsiChar;
begin
  BASS_Init(0, 44100, 0, Handle, nil);
  CheckError;
  //
  c := BASS_StreamCreateFile(False, PChar('1.mp3'), 0, 0, BASS_UNICODE or BASS_STREAM_DECODE or BASS_SAMPLE_FLOAT);
  CheckError;
  //
  enc1 := BASS_Encode_FLAC_Start(c, '--ogg', BASS_ENCODE_PAUSE or BASS_ENCODE_AUTOFREE or BASS_ENCODE_FP_24BIT or
            BASS_UNICODE or BASS_ENCODE_CAST_NOLIMIT, nil, nil);
  CheckError;
  enc2 := BASS_Encode_FLAC_Start(c, '--ogg', BASS_ENCODE_PAUSE or BASS_ENCODE_AUTOFREE or BASS_ENCODE_FP_24BIT or
            BASS_UNICODE or BASS_ENCODE_CAST_NOLIMIT, nil, nil);
  CheckError;
  //
  srv := '8000';
  BASS_Encode_ServerInit(enc1, srv, BufSz * 2, BufSz * 2, BASS_ENCODE_SERVER_META, MyServerClientConnectCb, nil);
  CheckError;
  BASS_Encode_ServerInit(enc2, srv, BufSz * 2, BufSz * 2, BASS_ENCODE_SERVER_META, MyServerClientConnectCb, nil);
  CheckError;
end;

Ian @ un4seen

  • Administrator
  • Posts: 25287
Oh right, I see what you mean. It's caused by BASS_Encode_ServerInit enabling the SO_REUSEADDR option on the server socket, which I see behaves differently on Windows compared to other platforms (this issue doesn't happen on the other platforms). Here's an update that doesn't enable that:

   www.un4seen.com/stuff/bassenc.zip

Please let me know if you still see the problem, or any unexpected instances of BASS_Encode_ServerInit failing with BASS_ERROR_BUSY. The SO_REUSEADDR option is/was enabled to prevent an old socket in the TIME_WAIT state causing the requested port to be unavailable (BASS_ERROR_BUSY).

fmcoder

  • Posts: 485
Thanks! That fixed the problem. Can I use this bassenc version in production?

Ian @ un4seen

  • Administrator
  • Posts: 25287
Yes, it should be fine to use in production (it's a "release" build). The only other change (besides the SO_REUSEADDR removal) compared to the 2.4.16 release is that a 0 handle can be used with BASS_Encode_Start for pre-encoded data (provided via BASS_Encode_Write).

fmcoder

  • Posts: 485
Back to the original question, I left the test instance running (using the "stuff" bassenc), and after about 13 hours, the players were unable to connect to the FLAC stream. The BASS_Encode_GetCount(...BASS_ENCODE_COUNT_OUT) function shows that the data is being encoded (the returned number increases) - so I assume the encoder is alive and well. I didn't find a way to check server status.

* I've set up logging to see if ENCODECLIENTPROC is being called - now I need to wait for this bug to appear again.
« Last Edit: 21 May '23 - 19:08 by fmcoder »

fmcoder

  • Posts: 485
The bug appeared, after 30+ hours running, the server failed to accept any new connections (it did work initially - I was able to connect and listen to the stream). ENCODECLIENTPROC is being called - for some reason, two times for each connection attempt, first two times with "connect" parameter = True, then immediately both times with False (or at least it's logged that way, I guess it could be two threads that call logging function at about the same time). My ENCODECLIENTPROC returns True for all connection attempts.

The log #1 shows connection attempts, entries from 21:50:44 are caused by Foobar2000; 21:51:13 are connection attempts from Opera browser. If connecting using "netradio" example, it shows "connecting... HTTP/1.0 200 OK " and stays like this (nothing is playing), causing one connection attempt in my log (see log #2 below).

My ENCODECLIENTPROC also modifies "headers" parameter, in case of FLAC it uses the following string (#13#10 is Delphi's equivalent for \r\n):
Code: [Select]
'HTTP/1.0 200 OK'#13#10'Content-Type: ' + BASS_ENCODE_TYPE_OGG + #13#10;

Here's the result of my logging (from ENCODECLIENTPROC):
Code: [Select]
[2023-05-21 21:50:44] 127.0.0.1:63792 ClientConnect [True]
[2023-05-21 21:50:44] 127.0.0.1:63792 (0) CONN
[2023-05-21 21:50:44] 127.0.0.1:63792 (0) ClientConnect-Out
[2023-05-21 21:50:44] 127.0.0.1:63793 ClientConnect [True]
[2023-05-21 21:50:44] 127.0.0.1:63793 (0) CONN
[2023-05-21 21:50:44] 127.0.0.1:63793 (0) ClientConnect-Out
[2023-05-21 21:50:44] 127.0.0.1:63793 ClientConnect [False]
[2023-05-21 21:50:44] 127.0.0.1:63793 (0) DCON
[2023-05-21 21:50:44] 127.0.0.1:63793 (0) ClientConnect-Out
[2023-05-21 21:50:44] 127.0.0.1:63792 ClientConnect [False]
[2023-05-21 21:50:44] 127.0.0.1:63792 (0) DCON
[2023-05-21 21:50:44] 127.0.0.1:63792 (0) ClientConnect-Out
[2023-05-21 21:51:13] [::1]:63806 ClientConnect [True]
[2023-05-21 21:51:13] [::1]:63806 (0) CONN
[2023-05-21 21:51:13] [::1]:63806 (0) ClientConnect-Out
[2023-05-21 21:51:13] [::1]:63807 ClientConnect [True]
[2023-05-21 21:51:13] [::1]:63807 (0) CONN
[2023-05-21 21:51:13] [::1]:63807 (0) ClientConnect-Out
[2023-05-21 21:51:13] [::1]:63806 ClientConnect [False]
[2023-05-21 21:51:13] [::1]:63806 (0) DCON
[2023-05-21 21:51:13] [::1]:63806 (0) ClientConnect-Out

Code: [Select]
[2023-05-21 22:00:55] [::1]:63901 ClientConnect [True]
[2023-05-21 22:00:55] [::1]:63901 (0) CONN
[2023-05-21 22:00:55] [::1]:63901 (0) ClientConnect-Out
« Last Edit: 21 May '23 - 19:33 by fmcoder »

Ian @ un4seen

  • Administrator
  • Posts: 25287
Do you also use OGG Vorbis or OPUS encoding, and if so, are they never affected, ie. the problem only ever happens with FLAC? If so, what options are you using with the FLAC encoder? If it's just "--ogg" then please try "--ogg --no-padding --no-seektable --limit-min-bitrate" (which is what the SERVER.C example uses) with the latest BASSenc_FLAC add-on. Also try having your ENCODECLIENTPROC just return TRUE without setting custom headers, as BASSenc should automatically detect OGG-based formats and send a "Content-Type" header to clients.

If the problem still happens, please see if you can reproduce it with the pre-compiled SERVER-ADDONS.EXE example included in the BASSenc package (C\BIN folder) and see if an "encoder died" message pops up. If that works fine then you can compare its and your code for differences.

fmcoder

  • Posts: 485
This problem was originally reported by a user of my software and he told me that OGG/OPUS never had this issue, only FLAC. I'll try running it with the new options to see if it fixes the problem, thank you.

fmcoder

  • Posts: 485
That (adding encoder options, disabling header modification) didn't help. After 7.5 hours running, server stopped accepting connections, log from ENCODECLIENTPROC included below.
This not only happens with bassenc_flac addon, this problem also exists when using flac.exe command line encoder with bassenc.

Code: [Select]
[2023-05-23 20:44:53] 127.0.0.1:63292 ClientConnect [True]
[2023-05-23 20:44:53] 127.0.0.1:63292 (1) CONN
[2023-05-23 20:44:53] 127.0.0.1:63292 (1) ClientConnect-Out
[2023-05-23 20:44:53] 127.0.0.1:63293 ClientConnect [True]
[2023-05-23 20:44:53] 127.0.0.1:63293 (1) CONN
[2023-05-23 20:44:53] 127.0.0.1:63293 (1) ClientConnect-Out
[2023-05-23 20:44:53] 127.0.0.1:63292 ClientConnect [False]
[2023-05-23 20:44:53] 127.0.0.1:63292 (1) DCON
[2023-05-23 20:44:53] 127.0.0.1:63292 (1) ClientConnect-Out
[2023-05-23 20:44:53] 127.0.0.1:63293 ClientConnect [False]
[2023-05-23 20:44:53] 127.0.0.1:63293 (1) DCON
[2023-05-23 20:44:53] 127.0.0.1:63293 (1) ClientConnect-Out

fmcoder

  • Posts: 485
This was also reproduced with server_addons precompiled example, it stopped accepting connections after about 15 hours. Attached is the screenshot of the configuration I used.

Ian @ un4seen

  • Administrator
  • Posts: 25287
That (adding encoder options, disabling header modification) didn't help. After 7.5 hours running, server stopped accepting connections, log from ENCODECLIENTPROC included below.
This not only happens with bassenc_flac addon, this problem also exists when using flac.exe command line encoder with bassenc.

Were you definitely using the latest BASSenc_FLAC or FLAC.EXE version? The reason I ask is because the "--limit-min-bitrate" option is new, and helps with streaming periods of silence. Btw, does your stream contain silence or is there always some sound?

Does the ENCODECLIENTPROC log indicate that the ENCODECLIENTPROC is still being called when the problem happens? Please also confirm what the error code is if you try to play the broken stream with BASS, eg. in the NETRADIO.EXE example.

In the meantime, I have started an instance of the SERVER-ADDONS.EXE example with BASSenc_FLAC to see if I can reproduce the problem and hopefully find out what it is. It hasn't happened yet (after around 10 hours).

fmcoder

  • Posts: 485
All bass*.dll are latest versions. Please see the screenshot attached.

The stream always, or at least majority of the time, had sound.

When the problem happens, ENCODECLIENTPROC is still being called (see logs from there in my posts above). My ENCODECLIENTPROC always returns True. I'll check return code later when I get this problem again.

Ian @ un4seen

  • Administrator
  • Posts: 25287
The SERVER-ADDON.EXE file isn't visible in that screenshot but the CAST EXEs look old, so perhaps SERVER-ADDON.EXE is too, in which case it may be missing the new "--limit-min-bitrate" option I mentioned. Please try re-downloading the BASSenc package to get the latest example EXEs, and see if you can reproduce the problem with the SERVER-ADDON.EXE example. The problem still hasn't happened yet with that here.

Once the problem begins, does it happen on every connection after that or does it eventually start working again? Are there any existing connections from before the problem began and do they continue to work in the meantime?

fmcoder

  • Posts: 485
I got this error again with my test app (it uses the "--limit-min-bitrate" flag), error code when connecting to such a stream with netradio.exe is 41, loos like that's because it doesn't have flac addon to play this stream.

The SERVER-ADDON.EXE file isn't visible in that screenshot but the CAST EXEs look old, so perhaps SERVER-ADDON.EXE is too, in which case it may be missing the new "--limit-min-bitrate" option I mentioned. Please try re-downloading the BASSenc package to get the latest example EXEs, and see if you can reproduce the problem with the SERVER-ADDON.EXE example. The problem still hasn't happened yet with that here.
I started a new test using the latest server-addon.exe, will see. But I guess it's something wrong with bassenc's server - because my test app failed after some time, and it was using the new limit-min-bitrate flag.

Once the problem begins, does it happen on every connection after that or does it eventually start working again? Are there any existing connections from before the problem began and do they continue to work in the meantime?
It stops working completely, no further connections are possible.
I didn't have existing streams playing so I can't tell about such case.

Ian @ un4seen

  • Administrator
  • Posts: 25287
error code when connecting to such a stream with netradio.exe is 41, loos like that's because it doesn't have flac addon to play this stream.

Is BASSFLAC.DLL present? The NETRADIO example will use it if it is. To confirm, is the stream working with the NETRADIO example before the problem begins? If so, it would appear that the stream is becoming corrupt at some point (error code 41 is BASS_ERROR_FILEFORM).

The problem still hasn't happened yet with SERVER-ADDONS.EXE here. Does the problem always happen eventually or are there cases where it doesn't?

fmcoder

  • Posts: 485
Is BASSFLAC.DLL present? The NETRADIO example will use it if it is. To confirm, is the stream working with the NETRADIO example before the problem begins? If so, it would appear that the stream is becoming corrupt at some point (error code 41 is BASS_ERROR_FILEFORM).
Tried netradio.exe with bassflac.dll present. It's able to play the stream initially, before it's corrupted. It's no longer able to play it after a while.

The server-addons.exe instance (latest version) also failed just now - stream is unplayable.

All failed FLAC streams result in error 41 from netradio.exe.

The problem still hasn't happened yet with SERVER-ADDONS.EXE here. Does the problem always happen eventually or are there cases where it doesn't?
Yes, it happens eventually, one time it ran for several days before it happened.

fmcoder

  • Posts: 485
I'd like to note that link actually continues working, in a sense: wget is able to "download" it. At quick glance, looks like "broken" stream doesn't have FLAC part in the header. This could possibly explain error 41.

Please see the result of stream downloads here (zip file with good and bad stream):
https://we.tl/t-CLgqfL4GKx?utm_campaign=TRN_TDL_05&utm_source=sendgrid&utm_medium=email&trk=TRN_TDL_05
« Last Edit: 26 May '23 - 20:36 by fmcoder »

Ian @ un4seen

  • Administrator
  • Posts: 25287
Yes, it looks like the header is missing in the "broken" file. It also looks like the 2 files are from different streams? If so, was the broken stream working initially? Once a header is detected by BASSenc, it should be sent to all clients until a new header is detected, so it's strange if there was a header initially and then not later.

If you're able to reproduce the problem with the latest SERVER-ADDONS.EXE example, please try using its metadata option to set a title and then see if the stream starts working shortly after. That option will cause the encoder to generate a new header (via BASS_Encode_FLAC_NewStream). Don't use that option before the problem happens, to ensure that it isn't causing the problem.

The problem still hasn't happened here yet. Perhaps it only happens after a number connections and/or multiple simultaneous connections? I'm currently only connecting to the server to check it occasionally, perhaps once an hour on average.

fmcoder

  • Posts: 485
Yes, it looks like the header is missing in the "broken" file. It also looks like the 2 files are from different streams? If so, was the broken stream working initially? Once a header is detected by BASSenc, it should be sent to all clients until a new header is detected, so it's strange if there was a header initially and then not later.
Yes, the stream worked initially. The recording was from two different streams, the second stream also failed, after about 1.5 days :)

If you're able to reproduce the problem with the latest SERVER-ADDONS.EXE example, please try using its metadata option to set a title and then see if the stream starts working shortly after. That option will cause the encoder to generate a new header (via BASS_Encode_FLAC_NewStream). Don't use that option before the problem happens, to ensure that it isn't causing the problem.
OK, will do.

The problem still hasn't happened here yet. Perhaps it only happens after a number connections and/or multiple simultaneous connections? I'm currently only connecting to the server to check it occasionally, perhaps once an hour on average.
I only check it 1-3 times per day, usually it's only one connection, not simultaneous, I let it play for a couple of seconds if it connects.

Ian @ un4seen

  • Administrator
  • Posts: 25287
The problem did eventually happen here. I didn't see the data that triggered it, but from the server's state there appears to have been some data that looked like the start of a new OGG bitstream (with new headers) but wasn't actually, and that left empty headers being sent to new clients. It was playable again after starting a new bitstream with BASS_Encode_FLAC_NewStream.

I'll look into tweaking the OGG bitstream detection and then come back with an update for you to try (probably tomorrow).

Ian @ un4seen

  • Administrator
  • Posts: 25287
Here's the update for you to try:

   www.un4seen.com/stuff/bassenc.zip

Please let me know if you ever still see the problem happening.

fmcoder

  • Posts: 485
Thank you!