next »

[JohnyDog]

For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

I don't think he wants to keep it secret. The linux version of mo3 unpacker seems to be upxed just for size reduction, as the unpacked file contains full debugging symbols, function names etc. Should i have the reason (and time ) to go for it, it should be pretty easy to reverse-engineered it.

However i'd like to see the spec too - relying on external proprietary program to do the decompressing really isn't good, and it hurts twice as much if you're developing FOSS software.

[Lau]

For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

I don't think he wants to keep it secret. The linux version of mo3 unpacker seems to be upxed just for size reduction, as the unpacked file contains full debugging symbols, function names etc. Should i have the reason (and time ) to go for it, it should be pretty easy to reverse-engineered it.

However i'd like to see the spec too - relying on external proprietary program to do the decompressing really isn't good, and it hurts twice as much if you're developing FOSS software.

Reverse is 90% done on Win32 binary, do not waste your time. I can explain you the 3 compression methods (1 for header, 2 lossless kind for samples). Format description and source code are done in paralell.

The remaining 10% is the pattern compression method, plus some details.

For the release I would like Ian opinion : official release with his agreement and checking, or unofficial release (might cause wrong implementation).

Lau

[Ian @ un4seen]

For the release I would like Ian opinion : official release with his agreement and checking, or unofficial release (might cause wrong implementation).

Before I can answer that, I'd like to know who you are and what your motives are?

You must've known that the way you've gone about things so far wasn't likely to leave me all warm and fuzzy at the prospect of working with you

[Laurent]

Hello Ian,

My motivation is only to open the mo3 format : having a documentation and source code to handle the format. "Handle the format" means being handle to decode and parse the header, and unpack the samples 'lossless methods'.

Then I'm going to release the doc and source code publically, that's all.

Yes, putting some x86 code on this forum wasn't the good way, I agree. First I did not think of writing a C version.

I do not want to write a player, or an encoder, Bass and MO3enc are very good for that.

I just want to 'open' the format, nothing more. I had reversed viruses in 68000 in the past. MO3 compression is my first experience in x86 and packing. I must admit your code is very compact, and the 'delta slope' compression very smart.

I think the 'delta' approach can be combined with 'best' freely available compression like lzo or bzip2. Today there's no need to optimize code at the asm level. I think the important is the compression ratio.

Nice to start a discussion with you.

Laurent.

[Ian @ un4seen]

Ok, that sounds fair enough. Send (by email rather than forum ) the stuff when you're done, and I'll let you know if there are any mistakes.

Btw, I wouldn't mind if you were planning to create a player or encoder with the code/specs, so long as it's free. I just don't really want people using it for commercial gain.

[Laurent]

Hi Ian,

The license will be GPL, and not BSD (which is business friendly). I do not want to make money with that.

Here is the status of my work : the header unpack routine (you call it MO3_decode) is translated in C,
as well as the 8 bits version of the 2 sample unpacking methods. I'm going to send you that.
All this code has a lot of comments : algorithms are explained.

The 16bits version of the sample unpackers are not debugged yet. And the content of the format is not 100% described yet (about 80%), just by lack of time. Could you release publically your 16bits decoders (C versions) and collaborate with me to finish the specification ?

Laurent

[Ian @ un4seen]

The 16-bit decoders are basically the same as the 8-bit (only very small differences). Send what you've done so far, and I'll see what's what.

[Laurent]

Hum, you're thinking I'm bluffing and you want to see my cards...
I'm going to send the proof i'm not bluffing tonight,when back at home.

Still no public commitment on this forum that your gonna collaborate to open MO3 ?

Laurent

[Ian @ un4seen]

It's not about cards, I was just thinking I could simply show how to modify your 8-bit code to support 16-bit.

Regarding collaborating, you can't really expect me to commit to that without seeing what you've done. For all I know, it could be a mess

[Laurent]

It's not about cards, I was just thinking I could simply show how to modify your 8-bit code to support 16-bit.

Regarding collaborating, you can't really expect me to commit to that without seeing what you've done. For all I know, it could be a mess

See by yourself, Ian :

http://lclevy.free.fr/mo3/


Laurent

[Laurent]

And before Ian might tell there is no format description ;-)

http://lclevy.free.fr/mo3/mo3_format.txt
(still incomplete)

Laurent

[Ian @ un4seen]

For the 16-bit versions, besides the obvious "check overflow" stuff/etc differences, simply change "dh" to 8 and "DECODE_VAL" to...

#define DECODE_VAL16 { \
   if (cl>=5) DECODE_VAL \
   else do { \
     READ_CTRL_BIT; \
     val = (val<<1) + carry; \
     READ_CTRL_BIT; \
     val = (val<<1) + carry; \
     READ_CTRL_BIT; \
   }while(carry); \
 }

Anyway, that's a pretty good effort so far, but I think the pattern data will perhaps be the trickiest bit to fully work out. You could continue working on it if you want, but I'd suggest just waiting for the "official" specs. The plan is that Milkytracker will be a proving ground for the specs, after which it'll be ready for release. Btw, if someone had actually offered a project for MO3 support before (instead of just demanding specs for no real purpose ), I think this whole thing could've been done and dusted a long time ago.

[Laurent]

Yes, I said that 16bits unpackers are not yet ready.
And it has been improved since the version you quoted.

Yes, the pattern compression seems pretty smart. I take it as a challenge.

MilkyTracker seems a very good tracker, and multi-platform. But not open source ?
Your goal is to compress modules (even more) without altering the replay result, which is not so easy : MilkyTracker authors may help you to achieve this.
My goal is only to document the storage of MO3.

Are you going to release code source with the specs ? And when ?

Laurent

[Ian @ un4seen]

Yes, I said that 16bits unpackers are not yet ready.
And it has been improved since the version you quoted.

Well, you did ask me to release the 16bit decoder code, and the version I quoted is what you posted just yesterday. Hiding cards?

MilkyTracker seems a very good tracker, and multi-platform. But not open source ?
Your goal is to compress modules (even more) without altering the replay result, which is not so easy : MilkyTracker authors may help you to achieve this.
My goal is only to document the storage of MO3.

For me, the thing about MO3 support in Milkytracker is that it's actually a reason to write the specs. I don't see the point in bothering to write specs if no-one is going to use them.

Are you going to release code source with the specs ? And when ?

Sure, it'll include code. I'm not sure when it'll be ready. Kmuland tells me the Milk-man is currently busy with exams, so we've not actually started yet.

[Laurent]

I have only released the piece of code which is stable. I'm going on with 16bits decoders, and the parser.
Currently the parser does this :

$ ./unmo3 -d 1 -p 3 dannyelf_ll.mo3
Uncompressed size = 53448 (0xd0c8)
Offset in compressed data after decompression = 5216
Songname = Danny elfmania
Nb channels = 8
Song length = 60
Song restart = 0
Nb pattern = 41
Nb unique voice = 123
Nb instr = 31
Nb sample = 31
Ticks/row = 6
Initial tempo = 125
Song sequence = 0 0 1 2 3 4 5 6 [...]
Unique voice number for each pattern = 0 1 2 3 4 4 4 4 [...]
Pattern size table = 64 64 64 64 [...]
Voice data len table = 0xf9 0x5b 0xc5 [...]
Instr table =
 Instr#0 name = Assembled by -moby-
 Instr#1 name = Original simpsons
[...]

Why releasing the code and specs if nobody needs it ? To share your know-how of module compression.
So that every tracker authors can respect this standard and some others can improve it.

Example: why it was so difficult to write a modern tracker compatible with the de-facto standard FT2 ?
Because (to my knowledge) no source was available, and very few documentation : only the format of the module, and not how to play it. Take a look here (http://lclevy.free.fr/exotica/), I have helped to gather and re(dis)cover information about module and replay algorithms.

I have made the same thing years ago about the Amiga filesystem : writing a solid specification and some code as a demonstration (http://lclevy.free.fr/adflib/). It was designed to work with a first application : ADFopus. Then years ago, without any contact with me, ADFview adopted it (because of the documented API). More recently a 3rd application surprisely adopted it.
And of course each time, the authors send me bugs fixes, and even a C++ wrapper.

I think the tracker communauty has more to win to use open and standard fileformat, instead of closed and proprietary ones, like MO3 -today-. That's the idea behind my work on it.

Now you have the whole ideas behind my motivation : sharing knowledge, and improve know-how on a given topic.

I'm continuing my work here : http://lclevy.free.fr/mo3

Laurent

[Laurent]

Anyway, that's a pretty good effort so far, but I think the pattern data will perhaps be the trickiest bit to fully work out.

Well, I finished tonight to understand pattern compression (for .mod only). Again, you did a very good job to compress this specific kind of data.

Question : why do you encode the empty voice for mod : "7, 0x10, 0xf0, 0xf0, 0xf0, 0xf0, 0x30, 00" ?
"6, 0xf0, 0xf0, 0xf0, 0xf0, 0x40, 00" is equivalent and is 1 byte shorter no ?

Laurent

[Ian @ un4seen]

Question : why do you encode the empty voice for mod : "7, 0x10, 0xf0, 0xf0, 0xf0, 0xf0, 0x30, 00" ?
"6, 0xf0, 0xf0, 0xf0, 0xf0, 0x40, 00" is equivalent and is 1 byte shorter no ?

Yep, it should produce the 2nd result. What encoder version are you using?

Btw, the current unreleased encoder stops at the last non-empty row, so that would now be encoded as a single 0 byte. The update has been sitting around for a while, as I was considering switching to LZMA for the structure compression. But I should probably go ahead and release the 2.2 update soon, as any compression changes would need to be 2.3 anyway (the MO3 version matches the BASS version that supports it).

[Laurent]

I'm using 1.8. But I saw 2.1 compress "the structures" the same way.

What I suggest for future compression is to combine your "semantic" compression:
-detect voice repetition
-detect and encode row repetition
-note/instr/effect efficient encoding 
-detect unused sample and sample repetition
-delta encoding for sample
-delta slope encoding for sample

1. apply this first pass to the module,
2. then apply as a second pass a standard and efficient compression (that can be chosen) like LZMA or bzip2

Why storing 8bits samples in 16bits inside MO3 ? because of BASS replaying ?

Laurent

[Ian @ un4seen]

Why storing 8bits samples in 16bits inside MO3 ?

8-bit samples aren't stored as 16-bit in MO3, but they are converted to 16-bit during decoding, so that all samples are 16-bit, which simplifies sample handling/processing.

[Laurent]

Yep, I was wrong about 8bits samples.

You don't answer my question about 2 phases compression I propose for mo3enc > 2.2

Are patterns played by BASS "in place" in RAM after MO3 decoding ? Or another transformation is required ?
Amiga protracker packers (see Exotica replay pages) were designed to by replayable "in place". I not 100% for MO3 decoded structure.

Laurent