Topic: Mo3 file format specs petition

[Laurent]

And before Ian might tell there is no format description ;-)

http://lclevy.free.fr/mo3/mo3_format.txt
(still incomplete)

Laurent

[Ian @ un4seen]

For the 16-bit versions, besides the obvious "check overflow" stuff/etc differences, simply change "dh" to 8 and "DECODE_VAL" to...

Code: [Select]

#define DECODE_VAL16 { \
   if (cl>=5) DECODE_VAL \
   else do { \
     READ_CTRL_BIT; \
     val = (val<<1) + carry; \
     READ_CTRL_BIT; \
     val = (val<<1) + carry; \
     READ_CTRL_BIT; \
   }while(carry); \
 }
Anyway, that's a pretty good effort so far, but I think the pattern data will perhaps be the trickiest bit to fully work out. You could continue working on it if you want, but I'd suggest just waiting for the "official" specs. The plan is that Milkytracker will be a proving ground for the specs, after which it'll be ready for release. Btw, if someone had actually offered a project for MO3 support before (instead of just demanding specs for no real purpose ), I think this whole thing could've been done and dusted a long time ago.

[Laurent]

Yes, I said that 16bits unpackers are not yet ready.
And it has been improved since the version you quoted.

Yes, the pattern compression seems pretty smart. I take it as a challenge.

MilkyTracker seems a very good tracker, and multi-platform. But not open source ?
Your goal is to compress modules (even more) without altering the replay result, which is not so easy : MilkyTracker authors may help you to achieve this.
My goal is only to document the storage of MO3.

Are you going to release code source with the specs ? And when ?

Laurent

[Ian @ un4seen]

Yes, I said that 16bits unpackers are not yet ready.
And it has been improved since the version you quoted.

Well, you did ask me to release the 16bit decoder code, and the version I quoted is what you posted just yesterday. Hiding cards?

MilkyTracker seems a very good tracker, and multi-platform. But not open source ?
Your goal is to compress modules (even more) without altering the replay result, which is not so easy : MilkyTracker authors may help you to achieve this.
My goal is only to document the storage of MO3.

For me, the thing about MO3 support in Milkytracker is that it's actually a reason to write the specs. I don't see the point in bothering to write specs if no-one is going to use them.

Are you going to release code source with the specs ? And when ?

Sure, it'll include code. I'm not sure when it'll be ready. Kmuland tells me the Milk-man is currently busy with exams, so we've not actually started yet.

[Laurent]

I have only released the piece of code which is stable. I'm going on with 16bits decoders, and the parser.
Currently the parser does this :

$ ./unmo3 -d 1 -p 3 dannyelf_ll.mo3
Uncompressed size = 53448 (0xd0c8)
Offset in compressed data after decompression = 5216
Songname = Danny elfmania
Nb channels = 8
Song length = 60
Song restart = 0
Nb pattern = 41
Nb unique voice = 123
Nb instr = 31
Nb sample = 31
Ticks/row = 6
Initial tempo = 125
Song sequence = 0 0 1 2 3 4 5 6 [...]
Unique voice number for each pattern = 0 1 2 3 4 4 4 4 [...]
Pattern size table = 64 64 64 64 [...]
Voice data len table = 0xf9 0x5b 0xc5 [...]
Instr table =
 Instr#0 name = Assembled by -moby-
 Instr#1 name = Original simpsons
[...]

Why releasing the code and specs if nobody needs it ? To share your know-how of module compression.
So that every tracker authors can respect this standard and some others can improve it.

Example: why it was so difficult to write a modern tracker compatible with the de-facto standard FT2 ?
Because (to my knowledge) no source was available, and very few documentation : only the format of the module, and not how to play it. Take a look here (http://lclevy.free.fr/exotica/), I have helped to gather and re(dis)cover information about module and replay algorithms.

I have made the same thing years ago about the Amiga filesystem : writing a solid specification and some code as a demonstration (http://lclevy.free.fr/adflib/). It was designed to work with a first application : ADFopus. Then years ago, without any contact with me, ADFview adopted it (because of the documented API). More recently a 3rd application surprisely adopted it.
And of course each time, the authors send me bugs fixes, and even a C++ wrapper.

I think the tracker communauty has more to win to use open and standard fileformat, instead of closed and proprietary ones, like MO3 -today-. That's the idea behind my work on it.

Now you have the whole ideas behind my motivation : sharing knowledge, and improve know-how on a given topic.

I'm continuing my work here : http://lclevy.free.fr/mo3

Laurent

[Laurent]

Anyway, that's a pretty good effort so far, but I think the pattern data will perhaps be the trickiest bit to fully work out.

Well, I finished tonight to understand pattern compression (for .mod only). Again, you did a very good job to compress this specific kind of data.

Question : why do you encode the empty voice for mod : "7, 0x10, 0xf0, 0xf0, 0xf0, 0xf0, 0x30, 00" ?
"6, 0xf0, 0xf0, 0xf0, 0xf0, 0x40, 00" is equivalent and is 1 byte shorter no ?

Laurent

[Ian @ un4seen]

Question : why do you encode the empty voice for mod : "7, 0x10, 0xf0, 0xf0, 0xf0, 0xf0, 0x30, 00" ?
"6, 0xf0, 0xf0, 0xf0, 0xf0, 0x40, 00" is equivalent and is 1 byte shorter no ?

Yep, it should produce the 2nd result. What encoder version are you using?

Btw, the current unreleased encoder stops at the last non-empty row, so that would now be encoded as a single 0 byte. The update has been sitting around for a while, as I was considering switching to LZMA for the structure compression. But I should probably go ahead and release the 2.2 update soon, as any compression changes would need to be 2.3 anyway (the MO3 version matches the BASS version that supports it).

[Laurent]

I'm using 1.8. But I saw 2.1 compress "the structures" the same way.

What I suggest for future compression is to combine your "semantic" compression:
-detect voice repetition
-detect and encode row repetition
-note/instr/effect efficient encoding 
-detect unused sample and sample repetition
-delta encoding for sample
-delta slope encoding for sample

1. apply this first pass to the module,
2. then apply as a second pass a standard and efficient compression (that can be chosen) like LZMA or bzip2

Why storing 8bits samples in 16bits inside MO3 ? because of BASS replaying ?

Laurent

[Ian @ un4seen]

Why storing 8bits samples in 16bits inside MO3 ?

8-bit samples aren't stored as 16-bit in MO3, but they are converted to 16-bit during decoding, so that all samples are 16-bit, which simplifies sample handling/processing.

[Laurent]

Yep, I was wrong about 8bits samples.

You don't answer my question about 2 phases compression I propose for mo3enc > 2.2

Are patterns played by BASS "in place" in RAM after MO3 decoding ? Or another transformation is required ?
Amiga protracker packers (see Exotica replay pages) were designed to by replayable "in place". I not 100% for MO3 decoded structure.

Laurent

[Ian @ un4seen]

You don't answer my question about 2 phases compression I propose for mo3enc > 2.2

I didn't realize you were asking a question. You basically described what the MO3 encoder already does

Are patterns played by BASS "in place" in RAM after MO3 decoding ? Or another transformation is required ?
Amiga protracker packers (see Exotica replay pages) were designed to by replayable "in place". I not 100% for MO3 decoded structure.

Sure, MO3 can be played "in place", but I don't know about "designed" to be so - that to me suggests consideration for things like alignment and padding for run-time variables, which is not the case in MO3.

[Laurent]

hi Ian,

that to me suggests consideration for things like alignment and padding for run-time variables, which is not the case in MO3.

no, but you can have a format very very compact, but hard to "replay". forget this.

another thing : I think there is another kind of repetition not yet exploited for compression by mo3enc : sequence of row repetion in a voice. Ex: imagine a given voice is like that (the number is one row data) : 1 2 3 0 1 2 3 0 1 2 3 0, why not coding this repetition ? for example : 1 2 3 0 (already done today) then "the 4 last row is repeated 3 times" (encoded 83 03, "-4 , 3 times").
And this kind of repetition is -very- common in module, removing it can save a lot of room, no ?

I'm still working at http://lclevy.free.fr/mo3, if you could tell me if there is errors...

Laurent

[Ian @ un4seen]

another thing : I think there is another kind of repetition not yet exploited for compression by mo3enc : sequence of row repetion in a voice. Ex: imagine a given voice is like that (the number is one row data) : 1 2 3 0 1 2 3 0 1 2 3 0, why not coding this repetition ? for example : 1 2 3 0 (already done today) then "the 4 last row is repeated 3 times" (encoded 83 03, "-4 , 3 times").
And this kind of repetition is -very- common in module, removing it can save a lot of room, no ?

The compression layer takes care of block repetition. Adding extra opcodes for it may well actually increase the size of the end result.

I'm still working at http://lclevy.free.fr/mo3, if you could tell me if there is errors...

Seems to be coming along nicely. One thing though, it's not only MPEG 2 layer 3 that's supported - MPEG 1 and MPEG 2.5 are also supported

[Laurent]

The compression layer takes care of block repetition. Adding extra opcodes for it may well actually increase the size of the end result.

OK so I should have missed it...

Seems to be coming along nicely. One thing though, it's not only MPEG 2 layer 3 that's supported - MPEG 1 and MPEG 2.5 are also supported

OK, I'm going to fix that.
The 'DeltaSlope' unpacker for 16 bits is working now. Do you have a better name for this kind of compression ?

I could not find a module which one of the 16bits sample is compressed with the 'delta' compression : I can not test yet my unpacker.

Laurent

[Laurent]

Well, the full code to unpack (5 function) is ready and released on the web page.

More code to come in the following week.

Laurent

[Ian @ un4seen]

The 'DeltaSlope' unpacker for 16 bits is working now. Do you have a better name for this kind of compression ?

I haven't named it, but it basically predicts the samples and encodes the error/difference, so maybe "prediction error" encoding? I believe most lossless audio compressors use the same principle, with varying prediction and encoding schemes.

Btw, I think you could change/remove the "an empty voice is encoded len=7, 10 f0 f0 f0 f0 30 00" pattern data example. That less than optimal output is an encoder bug (rather than part of the specs), and it'll be just "00" with the 2.2 encoder anyway

[Laurent]

I haven't named it, but it basically predicts the samples and encodes the error/difference, so maybe "prediction error" encoding? I believe most lossless audio compressors use the same principle, with varying prediction and encoding schemes.

OK for "ErrorPrediction". And if you look in the 'unofficial spec' I have already talked about that.

Btw, I think you could change/remove the "an empty voice is encoded len=7, 10 f0 f0 f0 f0 30 00" pattern data example. That less than optimal output is an encoder bug (rather than part of the specs), and it'll be just "00" with the 2.2 encoder anyway

OK, I going to change that. So you have read the doc until the 2.3.2 at least, hum ?
You talked about 'block repetition detection done by the encoder'. You're talking about the whole structure encoder, right ? I have seen no specific -pattern- (voice in fact) block repetition encoding... unless for blocks with only 1 row.

Another questions:
-how do you choose the best 'compression' between Delta/ErrorPrediction/NotCompressed ? by testing the 3 ones ?
-why not choosing FLAC or shorten ? except for short samples of course. ?
-or do you want to keep your compact decoder to use it in embedded devices with low memory ?

Other opinion :
- on a marketing point of view, for musicians, "MO3" seems not such a good name, since mp3 is associated with loss, and loss of quality. I would instead someting like choose 'Tracker Module Compression', to highlight the fact that the know-how is the compression of music data, rather than using MP3, which is only -one- of the compression method.

Laurent
 

[raina]

About the marketing, I think tracker musicians pretty much know MO3 already. MIDI musicians also need to learn to track before worrying about compression. It's the computer music newbies who have no idea and who repeatedly ask how to convert MP3s to MO3. They think MO3 is a more efficient stream compression method because the filesizes are so small, some just want to make the conversion to substitute a game's original music with their favourites. "Tracker Module Compression" is informative, but not a good name on its own IMHO. Combination of the two like this: "MO3 - Tracker Module Compression" (with or without the dash) could be used to refer the technology but MO3 would still be short for that. Also, "MO3 Tracker Module Compression" would be better than "MO3, mod format" in the MO3 page.

[Laurent]

I see marketing the way to explain 'simply' things to -real world- people (not only technical ones), including newbies 

Laurent

[raina]

In my view, MO3 offers nothing to newbies no matter what it was called.

The way I would explain MO3 to layman would be: "Yes, it's a tiny music format that sounds good. No, you cannot make your MP3s smaller by converting them to MO3." To further questions: "You can only make MO3s out of module formats such as MOD/S3M/XM/IT." To even further questions: "Modules are music files you write yourself using a tracker program or download from the Internet. You won't find any contemporary hits though. Although commercial music and tracker music meet sometimes, they still come from different worlds."

[Ian @ un4seen]

You talked about 'block repetition detection done by the encoder'. You're talking about the whole structure encoder, right ?

Yep, that is what I meant by "compression layer". It will efficiently compress any repeated blocks - repeated blocks is what general purpose compressors do best.

-how do you choose the best 'compression' between Delta/ErrorPrediction/NotCompressed ? by testing the 3 ones ?

The encoder uses 2 passes... the 1st pass just checks what the size of each method would be, and then the 2nd pass does the actual encoding using the smallest method.

-why not choosing FLAC or shorten ? except for short samples of course. ?

I don't really want to bloat the decoding side of things, especially as the lossless compression is primarily a backup plan for when MP3/OGG compression doesn't work well, eg. short samples. I think there's probably more to be squeezed out of the current methods, though. I'll look into it (for 2.3).

- on a marketing point of view, for musicians, "MO3" seems not such a good name, since mp3 is associated with loss, and loss of quality.

Personally, I like "MO3". It doesn't necessarily have to stand for "MOD with MP3", it could mean "MOD with 3 sample compression forms" (MP3/OGG/lossless). Although it does actually stand for "MOD with MP3"

[Laurent]

I don't really want to bloat the decoding side of things, especially as the lossless compression is primarily a backup plan for when MP3/OGG compression doesn't work well, eg. short samples.

So why not offer to use FAAC ? mpeg4 audio ?

I think there's probably more to be squeezed out of the current methods, though. I'll look into it (for 2.3).

And it could have for you (but not for the unofficial spec) an interesting -temporary- side effect.

Laurent

[Ian @ un4seen]

I don't really want to bloat the decoding side of things, especially as the lossless compression is primarily a backup plan for when MP3/OGG compression doesn't work well, eg. short samples.

So why not offer to use FAAC ? mpeg4 audio ?

I don't see how that would help lossless compression, or avoid bloatage.

I think there's probably more to be squeezed out of the current methods, though. I'll look into it (for 2.3).

And it could have for you (but not for the unofficial spec) an interesting -temporary- side effect.

I'm not sure what you mean. What side effect?

[Laurent]

I'll have to adapt my C code to new unpack methods...

Laurent

[Laurent]

$ ./unmo3 -s 4 -v 2 1 6500-ast_ogg.mo3
unmo3 v0.5 (opensource version)

Songname = "A Step to the Stars"
Nb channels = 61
Song length = 24
Song restart = 0
Nb pattern = 24
Nb unique voice = 661
Nb instr = 99
Nb sample = 99
Ticks/row = 12
Initial tempo = 125
Initial format is IT (0x00020f69)

Displaying channel #1 of pattern #2 (length 64): unique voice #33
[3, 2 3 1 3c f 15] 0*15 [3, 2 3 1 46 f 15] 0*15 [3, 2 3 1 4b f 15] 0*15 [3, 2 3 1 3e f 15] 0*15

saving sample004.dat (offset 0x9e2e, length 4009/2109 bytes, compression LosslessDeltaPred8, resolution 8bits)...

laurent@becane-21d7cd1a /cygdrive/e/mo3
$ ./unmo3 -s 81 -v 1 1 6500-ast_ogg.mo3
unmo3 v0.5 (opensource version)

Songname = "A Step to the Stars"
Nb channels = 61
Song length = 24
Song restart = 0
Nb pattern = 24
Nb unique voice = 661
Nb instr = 99
Nb sample = 99
Ticks/row = 12
Initial tempo = 125
Initial format is IT (0x00020f69)

Displaying channel #1 of pattern #1 (length 64): unique voice #9
[3, 2 0 1 30 b 0] [1, 22 f1] [6*1, 22 0] [1, 22 f2] [3*1, 22 0] [1, 22 f3] [3*1, 22 0] [3, 2 0 1 30 b ff] [1, 22 f1] [6*
1, 22 0] [1, 22 f2] [3*1, 22 0] [1, 22 f3] [3*1, 22 0] [3, 2 0 1 32 b 0] [1, 22 f1] [6*1, 22 0] [1, 22 f2] [3*1, 22 0] [
1, 22 f3] [3*1, 22 0] 0*11 [1, b 42] [3, 2 3 1 46 f 18] [4, 2 3 1 3c f 17 6 5] [2, f 16 6 3] [2, f 15 6 1]

saving sample081.ogg (offset 0x563b2, length 33600/6874 bytes, compression ogg, resolution 8bits)...

next step is explicit pattern displaying and why not .mod output...

Laurent