Yes, the handles do increment so you could do that, but you would then need to at least retain the latest handle to know where to stop checking - you can't just stop when BASS_ChannelGetDevice fails because there may be gaps in the sequence, eg. from freed channels but also from DSP/FX or sync handles in amongst them. You would also need to ignore the internal handles I mentioned, perhaps using BASS_ATTRIB_USER to mark yours.
I really do think it'd be better to just retain the created handles, eg. in an array/vector. It shouldn't need much extra code, basically an extra line after each channel creation. You could also use BASS_SYNC_FREE syncs to remove freed handles from the list, or just remove them when you check the list and find invalid/freed ones in it.