1, is the ONLY bassmod using free? Or must I buy it's licence?
BASSMOD is free for non-money making use. Otherwise, it's license fee is half that of the full BASS.
2, how do I set the volume of BASS and BASSMOD separatly?
Why would you want to use both together? Why not just use BASS?

3, at volume settings, what are the intervallums? Can I define a custom intervall? Maybe 0..255
BASSMOD's volume range is 0-100. BASS defaults to that too, but it can be changed via the BASS_CONFIG_MAXVOL option.