Mo3 file format specs petition

Started by discontinued,

Reply Print
Go Down Pages 1 2 3 4 5

amigarulez

#20
Ian, did you seen this?
http://exotica.fix.no/tunes/formats/index.html

Could be very nice if you add support for more file formats and being able to "encapsulate" more of them, do you add the features of the mods or are encapsulated like in sc68? Amiga mods rulez!

Did you seen this player?
http://xmp.sourceforge.net/

And here some info about amiga editors...



This player seems interesting, and his format too! ;)
Symphonie Pro v3.x
Author(s): Patrick Meng    Updated: 01/01/1998
Requirements: OS 3.0, 68020+
Description: Symphinie Pro is the most powerful tracker available.
Features:

    * 256 Physical Audiochannels, 16 Bit Software DSP, 16 Bit Samples MODs have a resolution of at least 20 Bit if you use 16 Bit Samples hires frequency resolution, hires volume resolution
    * Midi
    * Digital output (Audio Rendering) in WAV, AIFF, MAESTRO or RAW, 8/16 Bit, 5-500 kHz
    * Realtime Timestretching, Realtime Pitchshifting, an independent realtime resonant filter for every Audiochannel
    * 3 Phase Oversampling, 500 kHz Mixrate, 3D Surround Sound
    * 256 Independent Digital 16 Bit Outputchannels. Non destructive sample manipulation
    * DSP PLUG IN SYSTEM: Flanger, LP/BP/HP Filters, Echo, Delay...
    * Platform and Hardware independent Fileformat
    * Running on Amiga, MAC (via UAE), DOS/WIN95 (via UAE), Linux (via UAE)
    * Samples supported: IFF-8SVX, AIFF, WAV, MAESTRO, MAUD, 8/16Bit resolution, 5 - 599 kHz sample rate

http://exotica.fix.no/websites/dreamland/amiga/editors.html

mo3-rulez

#21
An interesting changelog from XMP:

- added more module format specs
        - added MO3 unpacking support
        - added file detection to the XMMS plugin
        - added Beep Media Player support to the XMMS plugin
        - added Epic Megagames PSM module support
        - added Epic Megagames old PSM (Silverball) module support
        - added DSMI/DMP Advanced Module Format support
        - added support to Ultimate Soundtracker modules
        - added ALSA 0.9/1.0 sound output support
        - fixed recursive decrunching of module files
        - fixed QNX6 portability issues (by Mike Gorchak)
        - fixed heavy memory leak in the XMMS plugin
        - fixed --time command-line parameter
        - fixed portamento-after-keyoff bug (Jeronen Tel's "Nine One
One"
          now plays correctly)
        - fixed IFF file loading to avoid data alignment errors
        - fixed endianism issues in MDL loader
        - updated OPL emulation (by Mike Gorchak)
        - default verbosity level changed to 1
        - default sound mode set to stereo
        - disabled MED loader (nonportable, didn't work well)


Soon... http://xmp.sourceforge.net and on his CVS...

Ian @ un4seen

#22
Quote from: amigarulezCould be very nice if you add support for more file formats and being able to "encapsulate" more of them, do you add the features of the mods or are encapsulated like in sc68? Amiga mods rulez!

I'm not familiar with sc68, but MO3 has it's own structures/etc - the other formats (XM/IT/etc) are converted to it. The channel/sample/effect/etc limits are probably large enough to support any other existing formats, but I'm not sure I'll be adding others myself. Maybe someone else can if the source is released.

Quote from: mo3-rulezAn interesting changelog from XMP:

- added more module format specs
        - added MO3 unpacking support

I remember adding MO3 support to XMP when testing the Linux UNMO3 - as I recall, it was something like 1 line of code :)

lau

#23
I doubt Ian will release the specs.
Its linux unmo3 executable is packed with http://upx.sourceforge.net/,
and the unmo3 win32 executable packed with http://www.un4seen.com/petite/ and
obfuscated. Anyway it can be disassembled, for education purpose.

Lau.

mo3rulez

#24
Any news about MO3 specs or portable source code?

Ian @ un4seen

#25
Yep, I think the code is pretty portable now (MO3 2.1 with OSX support was released recently). No progress on writing specs/documentation (surprise surprise :)), but I'll look into the possibility of releasing the UNMO3 source sometime soon - it'll need some tidying/stripping first (much of the source is shared with BASS/XMPlay, and isn't required).

Lau

#26
for x86 readers that understand compression algorithms like lzss ...

ed2k://|file|unpack_mo3_routine1.txt|7136|51935E94B9ABAFA199CCD537BFC05C78|h=IVF36DXXCII25A7A66SJGLBW5B2XDCXE|/

Lau

#27
header of an MO3 file :

hexa offset
          length  type    value or comments
----------------------------------------------
00000     3       char   "MO3"
00003     1       byte    0
00004     4       long    uncompressed length (little endian)
                           
00008     1       byte[]  compressed data

          see decompression routine1 provided

Lau

#28

the compression explained

ed2k://|file|unpack_mo3.txt|1965|F6126C2E0588351798EF72C81B4CF6CD|h=BSUMEVD4FYW3VKB5GXNPIEHKANPP6SOL|/

Zarggg

#29
And what about those of us who prefer not to use that eDonkey crap? ;D

Lau

#30
unpack_mo3_routine1.txt =

seg000:0040E470
seg000:0040E470 ; ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦ S U B R O U T I N E ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
seg000:0040E470
seg000:0040E470
seg000:0040E470 read_ctrl_bit   proc near               ; CODE XREF: decode_ctrl_bits:carry_is_setp
seg000:0040E470                                         ; decode_ctrl_bits+8p ...
seg000:0040E470                 add     dl, dl
seg000:0040E472                 jnz     short not_zero
seg000:0040E474                 mov     dl, [esi]
seg000:0040E476                 sub     esi, 0FFFFFFFFh
seg000:0040E479                 adc     dl, dl
seg000:0040E47B
seg000:0040E47B not_zero:                               ; CODE XREF: read_ctrl_bit+2j
seg000:0040E47B                 retn
seg000:0040E47B read_ctrl_bit   endp
seg000:0040E47B
seg000:0040E47C
seg000:0040E47C ; ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦ S U B R O U T I N E ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
seg000:0040E47C
seg000:0040E47C
seg000:0040E47C decode_ctrl_bits proc near              ; CODE XREF: unpack+2Ap
seg000:0040E47C                                         ; unpack+68p
seg000:0040E47C                 inc     ecx
seg000:0040E47D
seg000:0040E47D carry_is_set:                           ; CODE XREF: decode_ctrl_bits+Dj
seg000:0040E47D                 call    read_ctrl_bit
seg000:0040E482                 adc     ecx, ecx
seg000:0040E484                 call    read_ctrl_bit
seg000:0040E489                 jb      short carry_is_set ; 10/01/11 or encoded length in 1st bit or bit pairs, until 'n1'
seg000:0040E48B                 retn
seg000:0040E48B decode_ctrl_bits endp
seg000:0040E48B
seg000:0040E48C ; ---------------------------------------------------------------------------
seg000:0040E48C ; START OF FUNCTION CHUNK FOR unpack
seg000:0040E48C
seg000:0040E48C unpack_end:                             ; CODE XREF: unpack+1Bj
seg000:0040E48C                                         ; unpack+74j ...
seg000:0040E48C                 pop     eax
seg000:0040E48D                 mov     eax, esi
seg000:0040E48F                 pop     ebp
seg000:0040E490                 pop     ebx
seg000:0040E491                 pop     edi
seg000:0040E492                 pop     esi
seg000:0040E493                 retn
seg000:0040E493 ; END OF FUNCTION CHUNK FOR unpack
seg000:0040E494
seg000:0040E494 ; ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦ S U B R O U T I N E ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
seg000:0040E494
seg000:0040E494
seg000:0040E494 unpack          proc near               ; CODE XREF: process+58p
seg000:0040E494
seg000:0040E494 compr_ptr       = dword ptr  28h
seg000:0040E494 uncompr_ptr     = dword ptr  2Ch
seg000:0040E494 uncompr_size    = dword ptr  30h
seg000:0040E494
seg000:0040E494 ; FUNCTION CHUNK AT seg000:0040E48C SIZE 00000008 BYTES
seg000:0040E494
seg000:0040E494                 push    esi
seg000:0040E495                 push    edi
seg000:0040E496                 push    ebx
seg000:0040E497                 push    ebp
seg000:0040E498                 mov     esi, [esp-14h+compr_ptr]
seg000:0040E49C                 mov     edi, [esp-14h+uncompr_ptr]
seg000:0040E4A0                 mov     ebx, [esp-14h+uncompr_size]
seg000:0040E4A4                 push    0
seg000:0040E4A6                 xor     dl, dl
seg000:0040E4A8                 dec     ebx             ; uncompressed_size--;
seg000:0040E4A9                 movsb                   ; move byte at ds:(e)si to ds:(e)di
seg000:0040E4AA                 xor     ecx, ecx
seg000:0040E4AC
seg000:0040E4AC unpack_loop:                            ; CODE XREF: unpack+26j
seg000:0040E4AC                                         ; unpack+7Dj
seg000:0040E4AC                 cmp     ebx, 0
seg000:0040E4AF                 jle     short unpack_end ; end of decompression
seg000:0040E4B1                 call    read_ctrl_bit
seg000:0040E4B6                 jb      short compressed_data
seg000:0040E4B8                 movsb                   ; if bit in dl is 0 then copy the byte,
seg000:0040E4B8                                         ; else it is compressed
seg000:0040E4B9                 dec     ebx
seg000:0040E4BA                 jmp     short unpack_loop
seg000:0040E4BC ; ---------------------------------------------------------------------------
seg000:0040E4BC
seg000:0040E4BC compressed_data:                        ; CODE XREF: unpack+22j
seg000:0040E4BC                 xor     ebp, ebp
seg000:0040E4BE                 call    decode_ctrl_bits
seg000:0040E4C3                 sub     ecx, 3
seg000:0040E4C6                 jnb     short lz_ptr_in_ctrl_stream
seg000:0040E4C8                 mov     eax, [esp+0]    ; '00' = LZ ptr with same previous relative LZ ptr (from [esp+0])
seg000:0040E4CB                 inc     ecx
seg000:0040E4CC                 jmp     short previous_lz_ptr ; length in 2 bits, unless ecx==0
seg000:0040E4CE ; ---------------------------------------------------------------------------
seg000:0040E4CE
seg000:0040E4CE lz_ptr_in_ctrl_stream:                  ; CODE XREF: unpack+32j
seg000:0040E4CE                 mov     eax, ecx
seg000:0040E4D0                 xor     ecx, ecx
seg000:0040E4D2                 shl     eax, 8
seg000:0040E4D5                 lodsb                   ; load byte in esi into al
seg000:0040E4D6                 xor     eax, 0FFFFFFFFh ; not
seg000:0040E4D9                 cmp     eax, 0FFFFFB00h
seg000:0040E4DE                 adc     ebp, 1          ; if eax < -500 then ebp++; ebp++;
seg000:0040E4E1                 cmp     eax, 0FFFF8300h
seg000:0040E4E6                 adc     ebp, 0          ; if eax < -32000 then ebp++
seg000:0040E4E9                 mov     [esp+0], eax    ; offset to previous string
seg000:0040E4EC
seg000:0040E4EC previous_lz_ptr:                        ; CODE XREF: unpack+38j
seg000:0040E4EC                 call    read_ctrl_bit   ; length in 2 bits, unless ecx==0
seg000:0040E4F1                 adc     ecx, ecx
seg000:0040E4F3                 call    read_ctrl_bit
seg000:0040E4F8                 adc     ecx, ecx
seg000:0040E4FA                 jnz     short len_in_2bits ; ecx = length of previous string
seg000:0040E4FC                 call    decode_ctrl_bits ; decode length, first bit of bits pairs (n0), until (n1)
seg000:0040E501                 add     ecx, 2
seg000:0040E504
seg000:0040E504 len_in_2bits:                           ; CODE XREF: unpack+66j
seg000:0040E504                 add     ecx, ebp        ; ecx = length of previous string
seg000:0040E506                 sub     ebx, ecx        ; decrease remaining bytes to decompress
seg000:0040E508                 jb      short unpack_end
seg000:0040E50A                 push    esi             ; save esi
seg000:0040E50B                 lea     esi, [eax+edi]  ; pointer to previous string
seg000:0040E50E                 rep movsb               ; copy ecx bytes from esi to edi
seg000:0040E510                 pop     esi             ; restore esi
seg000:0040E511                 jmp     short unpack_loop
seg000:0040E511 unpack          endp
seg000:0040E511



unpack_mo3.txt =


(big thank you to Matt for the x86 code analysis)


The first byte is always uncompressed.  After that, you've got two interleaved streams
of control bytes and data bytes.  The control bytes are read by the shift_dl routine. 
In the unpack routine, the control bits are read most-significant first. 
A zero bit indicates "uncompressed byte".  A one bit indicates compressed data. 
The next two control bits control which kind of compression
-- if they are '00' it's LZ with the same (relative) pointer as a previous LZ. 
The next two bits of the control stream are the length, unless they are both zero. 

If they are both zero, the true length minus 2 is encoded in the control stream, two bits per bit. 
The first bit in each pair is the actual data, the second bit is 0 on the last pair.

If the first control bits are '11', '10' or '01', then the LZ pointer is in the control stream. 
The most significant bit of the pointer is a '1', then the next most significant bits
of the pointer are read from the control stream two bits at a time as described above
(including the initial 11 or 01 or 10).  Then 3 is subtracted from that value
and it is shifted left by 8 bits, and the 8 least significant bits
f the pointer are taken from the data stream.  The one's-complement of the result is taken.   
The length adjustment for -500 and -32000 is saved and added back in later
(it's always at least one). Then it goes into the same LZ as before,
with the next two bits of the control stream being the length unless they are both zero, etc.


Example:

64 6d 08 69 61

64 = 01100100
0 = next byte is literal 0x6d
1 = compressed data
10 = LZ with MSB of pointer zero after subtracting 3

08 -- byte from data stream, pointer to -9 bytes back (points to the 'a' in Danny)

01 -- from control stream, a length of 1, plus the adjustment 1 from earlier = 2.
0 -- indicates a literal 69
0 -- indicates a literal 61.

Irrational86

#31
Hey Lau, will you cut the crap out and let the guy finish releasing the damn source? It is not fair nor correct to do this kind of thing.

Lau

#32
I think that Ian will not release a usable code source : I mean with comments and usable as a spec for the format and the compression used. Why the executable were packed to produce a obfuscated executable ? To protect against reverse engineering...

I just trying to help people which want to write a portable and well documented unmo3, that's all.

I hope i'll wrong with Ian intention : I hope he will release the C source code of mo3 2.1 which is code in C, I think.

I just do not like proprietary formats, I have nothing specific againt Ian.

Lau.

Dotpitch

#33
well, Ian said he's working on the specs... it's nice that you've cracked the code, but what are you going to do with it? you still need a mod-player to play the modules that come out of it. why take all this effort, when there's a perfect player around (BASS ;)) and Ian's going to release the specs anyway? (ok, maybe not soon enough in your opinion, that's a point :P)

John Marwin (guest mode)

#34
Hmm, I'll throw this code to the people I hope will implement mo3 onto the modarchive, and see what happens.

Lau

#35

I'm going to release the spec and a C source code that unpack the header and parse it.

Be patient

Laurent

raina

#36
Quote from: LauBe patient

That's pretty rich coming from a guy who can't wait for the official release of the specs.

Lau

#37
For how long are you going to wait ? "maybe", "If I have time"...
Ian's tools are protected against disassembling, and packed with PEtite or UPX. It is a proof that Ian want to keep the secret no ?

raina

#38
I'm expecting progress in the first quarter, as both XMPlay 3.3 and Christmas are now out of the way. Erm.. Are you suggesting that Ian should opensource everything he's done so that people could benefit from selling software that uses _his_ technologies and he'd never see a dime?

Dotpitch

#39
Quote from: LauFor how long are you going to wait ? "maybe", "If I have time"...
It is a proof that Ian want to keep the secret no?
Wouldn't you? It works, anyone could use it, and any player based on BASS can play it. Why release the source if you've had a lot of work making it and want some credit? ;)
Reply Print
Go Up Pages 1 2 3 4 5