Thank you. It was the apply-to-all-file-formats that did it. (Sample rate and such was already set as in Windows, and changing it didn't affect the audio distortion.)
I couldn't make the connection since it didn't seem to show a clear pattern of affecting only specific file formats. And disabling the setting has no effect with Direct Sound output.
(And I was really tired, thus didn't experiment much. - I have to say, though, the fact that XMPlay immediately applies settings to playback once you hit the button, without having to restart it, is very convenient.)