Author Topic: Mo3 file format specs petition  (Read 124586 times)

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Mo3 file format specs petition
« Reply #25 on: 17 Apr '05 - 17:10 »
Yep, I think the code is pretty portable now (MO3 2.1 with OSX support was released recently). No progress on writing specs/documentation (surprise surprise :)), but I'll look into the possibility of releasing the UNMO3 source sometime soon - it'll need some tidying/stripping first (much of the source is shared with BASS/XMPlay, and isn't required).

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #26 on: 3 May '05 - 22:03 »
for x86 readers that understand compression algorithms like lzss ...

ed2k://|file|unpack_mo3_routine1.txt|7136|51935E94B9ABAFA199CCD537BFC05C78|h=IVF36DXXCII25A7A66SJGLBW5B2XDCXE|/

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #27 on: 3 May '05 - 22:06 »
header of an MO3 file :

hexa offset
          length  type    value or comments
----------------------------------------------
00000     3       char   "MO3"
00003     1       byte    0
00004     4       long    uncompressed length (little endian)
                           
00008     1       byte[]  compressed data

          see decompression routine1 provided

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #28 on: 3 May '05 - 22:22 »

the compression explained

ed2k://|file|unpack_mo3.txt|1965|F6126C2E0588351798EF72C81B4CF6CD|h=BSUMEVD4FYW3VKB5GXNPIEHKANPP6SOL|/

Zarggg

  • Posts: 1242
Re: Mo3 file format specs petition
« Reply #29 on: 4 May '05 - 00:57 »
And what about those of us who prefer not to use that eDonkey crap? ;D

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #30 on: 6 May '05 - 21:25 »
unpack_mo3_routine1.txt =

seg000:0040E470
seg000:0040E470 ; S U B R O U T I N E
seg000:0040E470
seg000:0040E470
seg000:0040E470 read_ctrl_bit   proc near               ; CODE XREF: decode_ctrl_bits:carry_is_setp
seg000:0040E470                                         ; decode_ctrl_bits+8p ...
seg000:0040E470                 add     dl, dl
seg000:0040E472                 jnz     short not_zero
seg000:0040E474                 mov     dl, [esi]
seg000:0040E476                 sub     esi, 0FFFFFFFFh
seg000:0040E479                 adc     dl, dl
seg000:0040E47B
seg000:0040E47B not_zero:                               ; CODE XREF: read_ctrl_bit+2j
seg000:0040E47B                 retn
seg000:0040E47B read_ctrl_bit   endp
seg000:0040E47B
seg000:0040E47C
seg000:0040E47C ; S U B R O U T I N E
seg000:0040E47C
seg000:0040E47C
seg000:0040E47C decode_ctrl_bits proc near              ; CODE XREF: unpack+2Ap
seg000:0040E47C                                         ; unpack+68p
seg000:0040E47C                 inc     ecx
seg000:0040E47D
seg000:0040E47D carry_is_set:                           ; CODE XREF: decode_ctrl_bits+Dj
seg000:0040E47D                 call    read_ctrl_bit
seg000:0040E482                 adc     ecx, ecx
seg000:0040E484                 call    read_ctrl_bit
seg000:0040E489                 jb      short carry_is_set ; 10/01/11 or encoded length in 1st bit or bit pairs, until 'n1'
seg000:0040E48B                 retn
seg000:0040E48B decode_ctrl_bits endp
seg000:0040E48B
seg000:0040E48C ; ---------------------------------------------------------------------------
seg000:0040E48C ; START OF FUNCTION CHUNK FOR unpack
seg000:0040E48C
seg000:0040E48C unpack_end:                             ; CODE XREF: unpack+1Bj
seg000:0040E48C                                         ; unpack+74j ...
seg000:0040E48C                 pop     eax
seg000:0040E48D                 mov     eax, esi
seg000:0040E48F                 pop     ebp
seg000:0040E490                 pop     ebx
seg000:0040E491                 pop     edi
seg000:0040E492                 pop     esi
seg000:0040E493                 retn
seg000:0040E493 ; END OF FUNCTION CHUNK FOR unpack
seg000:0040E494
seg000:0040E494 ; S U B R O U T I N E
seg000:0040E494
seg000:0040E494
seg000:0040E494 unpack          proc near               ; CODE XREF: process+58p
seg000:0040E494
seg000:0040E494 compr_ptr       = dword ptr  28h
seg000:0040E494 uncompr_ptr     = dword ptr  2Ch
seg000:0040E494 uncompr_size    = dword ptr  30h
seg000:0040E494
seg000:0040E494 ; FUNCTION CHUNK AT seg000:0040E48C SIZE 00000008 BYTES
seg000:0040E494
seg000:0040E494                 push    esi
seg000:0040E495                 push    edi
seg000:0040E496                 push    ebx
seg000:0040E497                 push    ebp
seg000:0040E498                 mov     esi, [esp-14h+compr_ptr]
seg000:0040E49C                 mov     edi, [esp-14h+uncompr_ptr]
seg000:0040E4A0                 mov     ebx, [esp-14h+uncompr_size]
seg000:0040E4A4                 push    0
seg000:0040E4A6                 xor     dl, dl
seg000:0040E4A8                 dec     ebx             ; uncompressed_size--;
seg000:0040E4A9                 movsb                   ; move byte at ds:(e)si to ds:(e)di
seg000:0040E4AA                 xor     ecx, ecx
seg000:0040E4AC
seg000:0040E4AC unpack_loop:                            ; CODE XREF: unpack+26j
seg000:0040E4AC                                         ; unpack+7Dj
seg000:0040E4AC                 cmp     ebx, 0
seg000:0040E4AF                 jle     short unpack_end ; end of decompression
seg000:0040E4B1                 call    read_ctrl_bit
seg000:0040E4B6                 jb      short compressed_data
seg000:0040E4B8                 movsb                   ; if bit in dl is 0 then copy the byte,
seg000:0040E4B8                                         ; else it is compressed
seg000:0040E4B9                 dec     ebx
seg000:0040E4BA                 jmp     short unpack_loop
seg000:0040E4BC ; ---------------------------------------------------------------------------
seg000:0040E4BC
seg000:0040E4BC compressed_data:                        ; CODE XREF: unpack+22j
seg000:0040E4BC                 xor     ebp, ebp
seg000:0040E4BE                 call    decode_ctrl_bits
seg000:0040E4C3                 sub     ecx, 3
seg000:0040E4C6                 jnb     short lz_ptr_in_ctrl_stream
seg000:0040E4C8                 mov     eax, [esp+0]    ; '00' = LZ ptr with same previous relative LZ ptr (from [esp+0])
seg000:0040E4CB                 inc     ecx
seg000:0040E4CC                 jmp     short previous_lz_ptr ; length in 2 bits, unless ecx==0
seg000:0040E4CE ; ---------------------------------------------------------------------------
seg000:0040E4CE
seg000:0040E4CE lz_ptr_in_ctrl_stream:                  ; CODE XREF: unpack+32j
seg000:0040E4CE                 mov     eax, ecx
seg000:0040E4D0                 xor     ecx, ecx
seg000:0040E4D2                 shl     eax, 8
seg000:0040E4D5                 lodsb                   ; load byte in esi into al
seg000:0040E4D6                 xor     eax, 0FFFFFFFFh ; not
seg000:0040E4D9                 cmp     eax, 0FFFFFB00h
seg000:0040E4DE                 adc     ebp, 1          ; if eax < -500 then ebp++; ebp++;
seg000:0040E4E1                 cmp     eax, 0FFFF8300h
seg000:0040E4E6                 adc     ebp, 0          ; if eax < -32000 then ebp++
seg000:0040E4E9                 mov     [esp+0], eax    ; offset to previous string
seg000:0040E4EC
seg000:0040E4EC previous_lz_ptr:                        ; CODE XREF: unpack+38j
seg000:0040E4EC                 call    read_ctrl_bit   ; length in 2 bits, unless ecx==0
seg000:0040E4F1                 adc     ecx, ecx
seg000:0040E4F3                 call    read_ctrl_bit
seg000:0040E4F8                 adc     ecx, ecx
seg000:0040E4FA                 jnz     short len_in_2bits ; ecx = length of previous string
seg000:0040E4FC                 call    decode_ctrl_bits ; decode length, first bit of bits pairs (n0), until (n1)
seg000:0040E501                 add     ecx, 2
seg000:0040E504
seg000:0040E504 len_in_2bits:                           ; CODE XREF: unpack+66j
seg000:0040E504                 add     ecx, ebp        ; ecx = length of previous string
seg000:0040E506                 sub     ebx, ecx        ; decrease remaining bytes to decompress
seg000:0040E508                 jb      short unpack_end
seg000:0040E50A                 push    esi             ; save esi
seg000:0040E50B                 lea     esi, [eax+edi]  ; pointer to previous string
seg000:0040E50E                 rep movsb               ; copy ecx bytes from esi to edi
seg000:0040E510                 pop     esi             ; restore esi
seg000:0040E511                 jmp     short unpack_loop
seg000:0040E511 unpack          endp
seg000:0040E511



unpack_mo3.txt =


(big thank you to Matt for the x86 code analysis)


The first byte is always uncompressed.  After that, you've got two interleaved streams
of control bytes and data bytes.  The control bytes are read by the shift_dl routine. 
In the unpack routine, the control bits are read most-significant first. 
A zero bit indicates "uncompressed byte".  A one bit indicates compressed data. 
The next two control bits control which kind of compression
-- if they are '00' it's LZ with the same (relative) pointer as a previous LZ. 
The next two bits of the control stream are the length, unless they are both zero. 

If they are both zero, the true length minus 2 is encoded in the control stream, two bits per bit. 
The first bit in each pair is the actual data, the second bit is 0 on the last pair.

If the first control bits are '11', '10' or '01', then the LZ pointer is in the control stream. 
The most significant bit of the pointer is a '1', then the next most significant bits
of the pointer are read from the control stream two bits at a time as described above
(including the initial 11 or 01 or 10).  Then 3 is subtracted from that value
and it is shifted left by 8 bits, and the 8 least significant bits
f the pointer are taken from the data stream.  The one's-complement of the result is taken.   
The length adjustment for -500 and -32000 is saved and added back in later
(it's always at least one). Then it goes into the same LZ as before,
with the next two bits of the control stream being the length unless they are both zero, etc.


Example:

64 6d 08 69 61

64 = 01100100
0 = next byte is literal 0x6d
1 = compressed data
10 = LZ with MSB of pointer zero after subtracting 3

08 -- byte from data stream, pointer to -9 bytes back (points to the 'a' in Danny)

01 -- from control stream, a length of 1, plus the adjustment 1 from earlier = 2.
0 -- indicates a literal 69
0 -- indicates a literal 61.

Irrational86

  • Posts: 960
Re: Mo3 file format specs petition
« Reply #31 on: 7 May '05 - 02:14 »
Hey Lau, will you cut the crap out and let the guy finish releasing the damn source? It is not fair nor correct to do this kind of thing.

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #32 on: 7 May '05 - 08:49 »
I think that Ian will not release a usable code source : I mean with comments and usable as a spec for the format and the compression used. Why the executable were packed to produce a obfuscated executable ? To protect against reverse engineering...

I just trying to help people which want to write a portable and well documented unmo3, that's all.

I hope i'll wrong with Ian intention : I hope he will release the C source code of mo3 2.1 which is code in C, I think.

I just do not like proprietary formats, I have nothing specific againt Ian.

Lau.

Dotpitch

  • Posts: 2871
Re: Mo3 file format specs petition
« Reply #33 on: 7 May '05 - 10:30 »
well, Ian said he's working on the specs... it's nice that you've cracked the code, but what are you going to do with it? you still need a mod-player to play the modules that come out of it. why take all this effort, when there's a perfect player around (BASS ;)) and Ian's going to release the specs anyway? (ok, maybe not soon enough in your opinion, that's a point :P)

John Marwin (guest mode)

  • Guest
Re: Mo3 file format specs petition
« Reply #34 on: 16 May '05 - 06:18 »
Hmm, I'll throw this code to the people I hope will implement mo3 onto the modarchive, and see what happens.

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #35 on: 9 Jan '06 - 11:48 »

I'm going to release the spec and a C source code that unpack the header and parse it.

Be patient

Laurent

raina

  • Posts: 1163
Re: Mo3 file format specs petition
« Reply #36 on: 9 Jan '06 - 12:28 »
Be patient

That's pretty rich coming from a guy who can't wait for the official release of the specs.

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #37 on: 9 Jan '06 - 17:48 »
For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

raina

  • Posts: 1163
Re: Mo3 file format specs petition
« Reply #38 on: 9 Jan '06 - 18:30 »
I'm expecting progress in the first quarter, as both XMPlay 3.3 and Christmas are now out of the way. Erm.. Are you suggesting that Ian should opensource everything he's done so that people could benefit from selling software that uses _his_ technologies and he'd never see a dime?

Dotpitch

  • Posts: 2871
Re: Mo3 file format specs petition
« Reply #39 on: 10 Jan '06 - 06:49 »
For how long are you going to wait ? "maybe", "If I have time"...
It is a proof that Ian want to keep the secret no?
Wouldn't you? It works, anyone could use it, and any player based on BASS can play it. Why release the source if you've had a lot of work making it and want some credit? ;)

JohnyDog

  • Guest
Re: Mo3 file format specs petition
« Reply #40 on: 23 Jan '06 - 05:43 »
For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

I don't think he wants to keep it secret. The linux version of mo3 unpacker seems to be upxed just for size reduction, as the unpacked file contains full debugging symbols, function names etc. Should i have the reason (and time :( ) to go for it, it should be pretty easy to reverse-engineered it.

However i'd like to see the spec too - relying on external proprietary program to do the decompressing really isn't good, and it hurts twice as much if you're developing FOSS software.

Lau

  • Guest
Re: Mo3 file format specs petition
« Reply #41 on: 23 Jan '06 - 10:16 »
For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

I don't think he wants to keep it secret. The linux version of mo3 unpacker seems to be upxed just for size reduction, as the unpacked file contains full debugging symbols, function names etc. Should i have the reason (and time :( ) to go for it, it should be pretty easy to reverse-engineered it.

However i'd like to see the spec too - relying on external proprietary program to do the decompressing really isn't good, and it hurts twice as much if you're developing FOSS software.

Reverse is 90% done on Win32 binary, do not waste your time. I can explain you the 3 compression methods (1 for header, 2 lossless kind for samples). Format description and source code are done in paralell.

The remaining 10% is the pattern compression method, plus some details.

For the release I would like Ian opinion : official release with his agreement and checking, or unofficial release (might cause wrong implementation).

Lau

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Mo3 file format specs petition
« Reply #42 on: 25 Jan '06 - 14:14 »
For the release I would like Ian opinion : official release with his agreement and checking, or unofficial release (might cause wrong implementation).

Before I can answer that, I'd like to know who you are and what your motives are?

You must've known that the way you've gone about things so far wasn't likely to leave me all warm and fuzzy at the prospect of working with you :P

Laurent

  • Guest
Re: Mo3 file format specs petition
« Reply #43 on: 26 Jan '06 - 13:54 »
Hello Ian,

My motivation is only to open the mo3 format : having a documentation and source code to handle the format. "Handle the format" means being handle to decode and parse the header, and unpack the samples 'lossless methods'.

Then I'm going to release the doc and source code publically, that's all.

Yes, putting some x86 code on this forum wasn't the good way, I agree. First I did not think of writing a C version.

I do not want to write a player, or an encoder, Bass and MO3enc are very good for that.

I just want to 'open' the format, nothing more. I had reversed viruses in 68000 in the past. MO3 compression is my first experience in x86 and packing. I must admit your code is very compact, and the 'delta slope' compression very smart.

I think the 'delta' approach can be combined with 'best' freely available compression like lzo or bzip2. Today there's no need to optimize code at the asm level. I think the important is the compression ratio.

Nice to start a discussion with you.

Laurent.

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Mo3 file format specs petition
« Reply #44 on: 26 Jan '06 - 17:50 »
Ok, that sounds fair enough. Send (by email rather than forum ;)) the stuff when you're done, and I'll let you know if there are any mistakes.

Btw, I wouldn't mind if you were planning to create a player or encoder with the code/specs, so long as it's free. I just don't really want people using it for commercial gain.

Laurent

  • Guest
Re: Mo3 file format specs petition
« Reply #45 on: 27 Jan '06 - 08:48 »
Hi Ian,

The license will be GPL, and not BSD (which is business friendly). I do not want to make money with that.

Here is the status of my work : the header unpack routine (you call it MO3_decode) is translated in C,
as well as the 8 bits version of the 2 sample unpacking methods. I'm going to send you that.
All this code has a lot of comments : algorithms are explained.

The 16bits version of the sample unpackers are not debugged yet. And the content of the format is not 100% described yet (about 80%), just by lack of time. Could you release publically your 16bits decoders (C versions) and collaborate with me to finish the specification ?

Laurent

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Mo3 file format specs petition
« Reply #46 on: 27 Jan '06 - 15:34 »
The 16-bit decoders are basically the same as the 8-bit (only very small differences). Send what you've done so far, and I'll see what's what.

Laurent

  • Guest
Re: Mo3 file format specs petition
« Reply #47 on: 27 Jan '06 - 15:53 »
Hum, you're thinking I'm bluffing and you want to see my cards...
I'm going to send the proof i'm not bluffing tonight,when back at home.

Still no public commitment on this forum that your gonna collaborate to open MO3 ?

Laurent

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Mo3 file format specs petition
« Reply #48 on: 27 Jan '06 - 18:03 »
It's not about cards, I was just thinking I could simply show how to modify your 8-bit code to support 16-bit.

Regarding collaborating, you can't really expect me to commit to that without seeing what you've done. For all I know, it could be a mess :)

Laurent

  • Guest
Re: Mo3 file format specs petition
« Reply #49 on: 27 Jan '06 - 18:15 »
It's not about cards, I was just thinking I could simply show how to modify your 8-bit code to support 16-bit.

Regarding collaborating, you can't really expect me to commit to that without seeing what you've done. For all I know, it could be a mess :)

See by yourself, Ian :

http://lclevy.free.fr/mo3/


Laurent