Yes, you need to provide a certificate and private key for the server to use, like you would for a normal web server. If you would just like to quickly try things, test files are included with the SERVER.C example in the BASSenc package (C\SERVER folder), which are used like this:
// example SSL certificate for HTTPS
BASS_SetConfigPtr(BASS_CONFIG_ENCODE_SERVER_CERT, "testcert.crt");
BASS_SetConfigPtr(BASS_CONFIG_ENCODE_SERVER_KEY, "testcert.key");