By default, the Winamp DSP wrapper will pass 16-bit sample data to the Winamp DSP plugin, as that should be supported by all of them. If a plugin supports higher resolutions, you can set it to 24/32-bit in the wrapper config, for higher quality. Note that if you have multiple DSP plugins active, the final quality will be limited to the lowest resolution amongst them, so there's not much point setting one to 32-bit while another is 16-bit, for example.
If you happen to have source code for what you want to do, the best thing (for quality) would be to create a native XMPlay DSP plugin
