Author Topic: Mo3 file format specs petition (Read 152033 times)

Ian @ un4seen · « **Reply #25 on:** 17 Apr '05 - 17:10 »

Yep, I think the code is pretty portable now (MO3 2.1 with OSX support was released recently). No progress on writing specs/documentation (surprise surprise

), but I'll look into the possibility of releasing the UNMO3 source sometime soon - it'll need some tidying/stripping first (much of the source is shared with BASS/XMPlay, and isn't required).

Lau · « **Reply #26 on:** 3 May '05 - 22:03 »

Lau · « **Reply #27 on:** 3 May '05 - 22:06 »

header of an MO3 file :

hexa offset
length type value or comments
----------------------------------------------
00000 3 char "MO3"
00003 1 byte 0
00004 4 long uncompressed length (little endian)

00008 1 byte[] compressed data

see decompression routine1 provided

Lau · « **Reply #28 on:** 3 May '05 - 22:22 »

Zarggg · « **Reply #29 on:** 4 May '05 - 00:57 »

And what about those of us who prefer not to use that eDonkey crap?

Lau · « **Reply #30 on:** 6 May '05 - 21:25 »

unpack_mo3_routine1.txt =

seg000:0040E470
seg000:0040E470 ; ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦ S U B R O U T I N E ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
seg000:0040E470
seg000:0040E470
seg000:0040E470 read_ctrl_bit proc near ; CODE XREF: decode_ctrl_bits:carry_is_setp
seg000:0040E470 ; decode_ctrl_bits+8p ...
seg000:0040E470 add dl, dl
seg000:0040E472 jnz short not_zero
seg000:0040E474 mov dl, [esi]
seg000:0040E476 sub esi, 0FFFFFFFFh
seg000:0040E479 adc dl, dl
seg000:0040E47B
seg000:0040E47B not_zero: ; CODE XREF: read_ctrl_bit+2j
seg000:0040E47B retn
seg000:0040E47B read_ctrl_bit endp
seg000:0040E47B
seg000:0040E47C
seg000:0040E47C ; ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦ S U B R O U T I N E ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
seg000:0040E47C
seg000:0040E47C
seg000:0040E47C decode_ctrl_bits proc near ; CODE XREF: unpack+2Ap
seg000:0040E47C ; unpack+68p
seg000:0040E47C inc ecx
seg000:0040E47D
seg000:0040E47D carry_is_set: ; CODE XREF: decode_ctrl_bits+Dj
seg000:0040E47D call read_ctrl_bit
seg000:0040E482 adc ecx, ecx
seg000:0040E484 call read_ctrl_bit
seg000:0040E489 jb short carry_is_set ; 10/01/11 or encoded length in 1st bit or bit pairs, until 'n1'
seg000:0040E48B retn
seg000:0040E48B decode_ctrl_bits endp
seg000:0040E48B
seg000:0040E48C ; ---------------------------------------------------------------------------
seg000:0040E48C ; START OF FUNCTION CHUNK FOR unpack
seg000:0040E48C
seg000:0040E48C unpack_end: ; CODE XREF: unpack+1Bj
seg000:0040E48C ; unpack+74j ...
seg000:0040E48C pop eax
seg000:0040E48D mov eax, esi
seg000:0040E48F pop ebp
seg000:0040E490 pop ebx
seg000:0040E491 pop edi
seg000:0040E492 pop esi
seg000:0040E493 retn
seg000:0040E493 ; END OF FUNCTION CHUNK FOR unpack
seg000:0040E494
seg000:0040E494 ; ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦ S U B R O U T I N E ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
seg000:0040E494
seg000:0040E494
seg000:0040E494 unpack proc near ; CODE XREF: process+58p
seg000:0040E494
seg000:0040E494 compr_ptr = dword ptr 28h
seg000:0040E494 uncompr_ptr = dword ptr 2Ch
seg000:0040E494 uncompr_size = dword ptr 30h
seg000:0040E494
seg000:0040E494 ; FUNCTION CHUNK AT seg000:0040E48C SIZE 00000008 BYTES
seg000:0040E494
seg000:0040E494 push esi
seg000:0040E495 push edi
seg000:0040E496 push ebx
seg000:0040E497 push ebp
seg000:0040E498 mov esi, [esp-14h+compr_ptr]
seg000:0040E49C mov edi, [esp-14h+uncompr_ptr]
seg000:0040E4A0 mov ebx, [esp-14h+uncompr_size]
seg000:0040E4A4 push 0
seg000:0040E4A6 xor dl, dl
seg000:0040E4A8 dec ebx ; uncompressed_size--;
seg000:0040E4A9 movsb ; move byte at ds:(e)si to ds:(e)di
seg000:0040E4AA xor ecx, ecx
seg000:0040E4AC
seg000:0040E4AC unpack_loop: ; CODE XREF: unpack+26j
seg000:0040E4AC ; unpack+7Dj
seg000:0040E4AC cmp ebx, 0
seg000:0040E4AF jle short unpack_end ; end of decompression
seg000:0040E4B1 call read_ctrl_bit
seg000:0040E4B6 jb short compressed_data
seg000:0040E4B8 movsb ; if bit in dl is 0 then copy the byte,
seg000:0040E4B8 ; else it is compressed
seg000:0040E4B9 dec ebx
seg000:0040E4BA jmp short unpack_loop
seg000:0040E4BC ; ---------------------------------------------------------------------------
seg000:0040E4BC
seg000:0040E4BC compressed_data: ; CODE XREF: unpack+22j
seg000:0040E4BC xor ebp, ebp
seg000:0040E4BE call decode_ctrl_bits
seg000:0040E4C3 sub ecx, 3
seg000:0040E4C6 jnb short lz_ptr_in_ctrl_stream
seg000:0040E4C8 mov eax, [esp+0] ; '00' = LZ ptr with same previous relative LZ ptr (from [esp+0])
seg000:0040E4CB inc ecx
seg000:0040E4CC jmp short previous_lz_ptr ; length in 2 bits, unless ecx==0
seg000:0040E4CE ; ---------------------------------------------------------------------------
seg000:0040E4CE
seg000:0040E4CE lz_ptr_in_ctrl_stream: ; CODE XREF: unpack+32j
seg000:0040E4CE mov eax, ecx
seg000:0040E4D0 xor ecx, ecx
seg000:0040E4D2 shl eax, 8
seg000:0040E4D5 lodsb ; load byte in esi into al
seg000:0040E4D6 xor eax, 0FFFFFFFFh ; not
seg000:0040E4D9 cmp eax, 0FFFFFB00h
seg000:0040E4DE adc ebp, 1 ; if eax < -500 then ebp++; ebp++;
seg000:0040E4E1 cmp eax, 0FFFF8300h
seg000:0040E4E6 adc ebp, 0 ; if eax < -32000 then ebp++
seg000:0040E4E9 mov [esp+0], eax ; offset to previous string
seg000:0040E4EC
seg000:0040E4EC previous_lz_ptr: ; CODE XREF: unpack+38j
seg000:0040E4EC call read_ctrl_bit ; length in 2 bits, unless ecx==0
seg000:0040E4F1 adc ecx, ecx
seg000:0040E4F3 call read_ctrl_bit
seg000:0040E4F8 adc ecx, ecx
seg000:0040E4FA jnz short len_in_2bits ; ecx = length of previous string
seg000:0040E4FC call decode_ctrl_bits ; decode length, first bit of bits pairs (n0), until (n1)
seg000:0040E501 add ecx, 2
seg000:0040E504
seg000:0040E504 len_in_2bits: ; CODE XREF: unpack+66j
seg000:0040E504 add ecx, ebp ; ecx = length of previous string
seg000:0040E506 sub ebx, ecx ; decrease remaining bytes to decompress
seg000:0040E508 jb short unpack_end
seg000:0040E50A push esi ; save esi
seg000:0040E50B lea esi, [eax+edi] ; pointer to previous string
seg000:0040E50E rep movsb ; copy ecx bytes from esi to edi
seg000:0040E510 pop esi ; restore esi
seg000:0040E511 jmp short unpack_loop
seg000:0040E511 unpack endp
seg000:0040E511

unpack_mo3.txt =

(big thank you to Matt for the x86 code analysis)

The first byte is always uncompressed. After that, you've got two interleaved streams
of control bytes and data bytes. The control bytes are read by the shift_dl routine.
In the unpack routine, the control bits are read most-significant first.
A zero bit indicates "uncompressed byte". A one bit indicates compressed data.
The next two control bits control which kind of compression
-- if they are '00' it's LZ with the same (relative) pointer as a previous LZ.
The next two bits of the control stream are the length, unless they are both zero.

If they are both zero, the true length minus 2 is encoded in the control stream, two bits per bit.
The first bit in each pair is the actual data, the second bit is 0 on the last pair.

If the first control bits are '11', '10' or '01', then the LZ pointer is in the control stream.
The most significant bit of the pointer is a '1', then the next most significant bits
of the pointer are read from the control stream two bits at a time as described above
(including the initial 11 or 01 or 10). Then 3 is subtracted from that value
and it is shifted left by 8 bits, and the 8 least significant bits
f the pointer are taken from the data stream. The one's-complement of the result is taken.
The length adjustment for -500 and -32000 is saved and added back in later
(it's always at least one). Then it goes into the same LZ as before,
with the next two bits of the control stream being the length unless they are both zero, etc.

Example:

64 6d 08 69 61

64 = 01100100
0 = next byte is literal 0x6d
1 = compressed data
10 = LZ with MSB of pointer zero after subtracting 3

08 -- byte from data stream, pointer to -9 bytes back (points to the 'a' in Danny)

01 -- from control stream, a length of 1, plus the adjustment 1 from earlier = 2.
0 -- indicates a literal 69
0 -- indicates a literal 61.

Irrational86 · « **Reply #31 on:** 7 May '05 - 02:14 »

Hey Lau, will you cut the crap out and let the guy finish releasing the damn source? It is not fair nor correct to do this kind of thing.

Lau · « **Reply #32 on:** 7 May '05 - 08:49 »

I think that Ian will not release a usable code source : I mean with comments and usable as a spec for the format and the compression used. Why the executable were packed to produce a obfuscated executable ? To protect against reverse engineering...

I just trying to help people which want to write a portable and well documented unmo3, that's all.

I hope i'll wrong with Ian intention : I hope he will release the C source code of mo3 2.1 which is code in C, I think.

I just do not like proprietary formats, I have nothing specific againt Ian.

Lau.

Dotpitch · « **Reply #33 on:** 7 May '05 - 10:30 »

well, Ian said he's working on the specs... it's nice that you've cracked the code, but what are you going to do with it? you still need a mod-player to play the modules that come out of it. why take all this effort, when there's a perfect player around (BASS

) and Ian's going to release the specs anyway? (ok, maybe not soon enough in your opinion, that's a point

)

John Marwin (guest mode) · « **Reply #34 on:** 16 May '05 - 06:18 »

Hmm, I'll throw this code to the people I hope will implement mo3 onto the modarchive, and see what happens.

Lau · « **Reply #35 on:** 9 Jan '06 - 11:48 »

I'm going to release the spec and a C source code that unpack the header and parse it.

Be patient

Laurent

raina · « **Reply #36 on:** 9 Jan '06 - 12:28 »

Quote from: Lau on 9 Jan '06 - 11:48

Be patient

That's pretty rich coming from a guy who can't wait for the official release of the specs.

Lau · « **Reply #37 on:** 9 Jan '06 - 17:48 »

For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

raina · « **Reply #38 on:** 9 Jan '06 - 18:30 »

I'm expecting progress in the first quarter, as both XMPlay 3.3 and Christmas are now out of the way. Erm.. Are you suggesting that Ian should opensource everything he's done so that people could benefit from selling software that uses _his_ technologies and he'd never see a dime?

Dotpitch · « **Reply #39 on:** 10 Jan '06 - 06:49 »

Quote from: Lau on 9 Jan '06 - 17:48

For how long are you going to wait ? "maybe", "If I have time"...
It is a proof that Ian want to keep the secret no?

Wouldn't you? It works, anyone could use it, and any player based on BASS can play it. Why release the source if you've had a lot of work making it and want some credit?

JohnyDog · « **Reply #40 on:** 23 Jan '06 - 05:43 »

Quote from: Lau on 9 Jan '06 - 17:48

For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

I don't think he wants to keep it secret. The linux version of mo3 unpacker seems to be upxed just for size reduction, as the unpacked file contains full debugging symbols, function names etc. Should i have the reason (and time

) to go for it, it should be pretty easy to reverse-engineered it.

However i'd like to see the spec too - relying on external proprietary program to do the decompressing really isn't good, and it hurts twice as much if you're developing FOSS software.

Lau · « **Reply #41 on:** 23 Jan '06 - 10:16 »

Quote from: JohnyDog on 23 Jan '06 - 05:43

Quote from: Lau on 9 Jan '06 - 17:48
For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

I don't think he wants to keep it secret. The linux version of mo3 unpacker seems to be upxed just for size reduction, as the unpacked file contains full debugging symbols, function names etc. Should i have the reason (and time ) to go for it, it should be pretty easy to reverse-engineered it.

However i'd like to see the spec too - relying on external proprietary program to do the decompressing really isn't good, and it hurts twice as much if you're developing FOSS software.

Reverse is 90% done on Win32 binary, do not waste your time. I can explain you the 3 compression methods (1 for header, 2 lossless kind for samples). Format description and source code are done in paralell.

The remaining 10% is the pattern compression method, plus some details.

For the release I would like Ian opinion : official release with his agreement and checking, or unofficial release (might cause wrong implementation).

Lau

Ian @ un4seen · « **Reply #42 on:** 25 Jan '06 - 14:14 »

Quote from: Lau on 23 Jan '06 - 10:16

For the release I would like Ian opinion : official release with his agreement and checking, or unofficial release (might cause wrong implementation).

Before I can answer that, I'd like to know who you are and what your motives are?

You must've known that the way you've gone about things so far wasn't likely to leave me all warm and fuzzy at the prospect of working with you

Laurent · « **Reply #43 on:** 26 Jan '06 - 13:54 »

Hello Ian,

My motivation is only to open the mo3 format : having a documentation and source code to handle the format. "Handle the format" means being handle to decode and parse the header, and unpack the samples 'lossless methods'.

Then I'm going to release the doc and source code publically, that's all.

Yes, putting some x86 code on this forum wasn't the good way, I agree. First I did not think of writing a C version.

I do not want to write a player, or an encoder, Bass and MO3enc are very good for that.

I just want to 'open' the format, nothing more. I had reversed viruses in 68000 in the past. MO3 compression is my first experience in x86 and packing. I must admit your code is very compact, and the 'delta slope' compression very smart.

I think the 'delta' approach can be combined with 'best' freely available compression like lzo or bzip2. Today there's no need to optimize code at the asm level. I think the important is the compression ratio.

Nice to start a discussion with you.

Laurent.

Ian @ un4seen · « **Reply #44 on:** 26 Jan '06 - 17:50 »

Ok, that sounds fair enough. Send (by email rather than forum

) the stuff when you're done, and I'll let you know if there are any mistakes.

Btw, I wouldn't mind if you were planning to create a player or encoder with the code/specs, so long as it's free. I just don't really want people using it for commercial gain.

Laurent · « **Reply #45 on:** 27 Jan '06 - 08:48 »

Hi Ian,

The license will be GPL, and not BSD (which is business friendly). I do not want to make money with that.

Here is the status of my work : the header unpack routine (you call it MO3_decode) is translated in C,
as well as the 8 bits version of the 2 sample unpacking methods. I'm going to send you that.
All this code has a lot of comments : algorithms are explained.

The 16bits version of the sample unpackers are not debugged yet. And the content of the format is not 100% described yet (about 80%), just by lack of time. Could you release publically your 16bits decoders (C versions) and collaborate with me to finish the specification ?

Laurent

Ian @ un4seen · « **Reply #46 on:** 27 Jan '06 - 15:34 »

The 16-bit decoders are basically the same as the 8-bit (only very small differences). Send what you've done so far, and I'll see what's what.

Laurent · « **Reply #47 on:** 27 Jan '06 - 15:53 »

Hum, you're thinking I'm bluffing and you want to see my cards...
I'm going to send the proof i'm not bluffing tonight,when back at home.

Still no public commitment on this forum that your gonna collaborate to open MO3 ?

Laurent

Ian @ un4seen · « **Reply #48 on:** 27 Jan '06 - 18:03 »

It's not about cards, I was just thinking I could simply show how to modify your 8-bit code to support 16-bit.

Regarding collaborating, you can't really expect me to commit to that without seeing what you've done. For all I know, it could be a mess

Laurent · « **Reply #49 on:** 27 Jan '06 - 18:15 »

Quote from: Ian @ un4seen on 27 Jan '06 - 18:03

It's not about cards, I was just thinking I could simply show how to modify your 8-bit code to support 16-bit.

Regarding collaborating, you can't really expect me to commit to that without seeing what you've done. For all I know, it could be a mess

See by yourself, Ian :

http://lclevy.free.fr/mo3/

Laurent

Author Topic: Mo3 file format specs petition (Read 152033 times)

Lau

Lau

Lau

Lau

Lau

John Marwin (guest mode)

Lau

Lau

JohnyDog

Lau

Laurent

Laurent

Laurent

Laurent