The compression of samples is done with either a built in lossless method or lossy methods, that are provided by external encoders. Namely, lame.exe and oggenc.exe. You can get those from
http:/www.rarewares.org/. Additionally, mo3enc.exe and unmo3.exe are command line applications and they exit right away because you haven't told them to do anything. Command line programs need parameters, which can be inserted into a shortcut to the program but that is meant for automizing repeated actions. To really use these two programs, you need to descend to the command line by clicking Start > Run, typing in "cmd" without the quotes and hitting Run.
You don't need DosBox, these are all native Win32 programs but only one of them (mo3.exe) has a graphical user interface.
There's a fundamental error in what you are saying. MO3 files are modules that contain (compressed) WAVE files as instruments and pattern data (similar to MIDI information) which tells a player how to play those instruments over and over again at different pitches to produce music. Decoding MO3 files means decompressing the instruments and the pattern data, after that the module is either played or rendered to Wav. So, decoding MO3 to WAV is not only possible but easy but the resulting WAV is only what's generated by the MO3 and the player. All the pattern data is lost, and there's no way of rebuilding a MO3 out of a WAV file.
