Does the problem only happen when you have got the other encoding going (calling ExtAudioFileWrite/etc) at the same ti me? If you haven't tried without that other encoding, then please do so to confirm whether it does have anything to do with the problem.
I have tried without that other encoding but error continuing

When you say you're checking the handle, do you mean via BASS_Encode_IsActive? If not, please check what that says and/or use BASS_Encode_SetNotify to receive notification of when the encoder dies.
I am checking handle with BASS_Encode_IsActive and BASS_Encode_SetNotify. In a random moment i am receiving BASS_ENCODE_NOTIFY_ENCODER in ENCODENOTIFYPROC notify and handle is stopped. According to doc this mean 'encoder died' but there is no clarification about it. What can be causing the encoder die?
Are you freeing the encoder (eg. via BASS_Encode_Stop) before you try to open the written file? If not, there may be some data that hasn't been flushed to the file yet.
Yes, you were right, BASS_Encode_Stop was being called before opening the file. Sorry...