Topic: Old UNMO3 IT header issues

[saga]

This is kind of a follow-up to a thread from a few years back...

What I observed back then: UNMO3, at the time, wrote the number of edit history entries (which was 0, since it didn't write an actual edit history), even though the "embed edit history" special flag 0x02 was not set. The only data following the edit history are MIDI macros (Edit: and possible MPT extensions), so this is only relevant when the original file contained any non-standard macros or plugin stuff (which would then not be read correctly).

I no longer have this specific version of UNMO3, so I tried to reproduce this with an older version from April 2008, and - surprise! - that older version writes the "embed edit history" flag, and it writes 6 extra zero bytes before the MIDI macros! Given the low number of IT files that have non-standard MIDI macros, combined with the low number of IT files that have been uncompressed using UNMO3, I am not entirely sure if this would be worth a workaround, but it surely could be used for fingerprinting such files as UNMO3-made files.

So, now for the interesting part: When did this behaviour change from writing those extra bytes and the edit history flags to writing a few less extra bytes and not setting the history flag? Can you dig up an UNMO3 version from between 2008 and 2011 that would show the behaviour from my original post? Do you still have the old source code that could possibly explain why UNMO3 wrote those 6 (or 4, depending on how you look at it) extra bytes?

[Ian @ un4seen]

Looking at the UNMO3 source code, the only difference I can see in the MIDI stuff is that "special" bits 2+3 (6) were set prior to the update posted in your linked thread. I don't see any difference in the number of bytes written before the MIDI config. Are you sure there is a difference?

[saga]

Looking at various UNMO3-ed files, the difference does not seem to come from different UNMO3 version, but probably from different module settings.
Using UNMO3 2.4 from April 2008, I do not see those four extra bytes (i.e. only two zero bytes in front of the FX00 chunk) when converting codename_silver.mo3 to IT, but they are present when decoding the two MO3 files found in the attachment (the decoded IT files are also in the archive for convenience).

EDIT: I think I figured it out! Even in sample mode, UNMO3 2.4 from 2008 seems to reserve some space for instrument parapointers in the header (as many as there are samples). This is where the extra four zeros come from. It doesn't actually write out any instrument parapointers, so there are 4 * smpNum zero bytes after the pattern parapointers. Since Codename Silver is in instrument mode, I cannot see the extra bytes there.
Having written my own MO3 decoder I can somehow see where this is coming from (the MO3 file still contains instrument headers for each sample), but if you still have the source code for that old version, it would be good if you can confirm that this is indeed what UNMO3 was doing wrong.

[Ian @ un4seen]

UNMO3 was indeed reserving space for the instrument parapointers in 2008. It stopped doing that in early 2009, although the next release (2.4.1) wasn't actually until 2016!

[saga]

Okay, I think with this information I can now build fairly reliable fingerprinting of IT files saved with UNMO3, and automatically fix IT files saved with UNMO3 older than 2.4.0.1 ("stuff" update from January 2011). They are probably rare but they do exist here and there.

For anyone else trying to identify IT files saved with UNMO3, here's the fingerprint (feel free to correct if anything is wrong ):
- cwtv and cmwt are both 0x0214
- row highlights are both 0
- pwd is 0
- reserved is 0
- MIDI pitch controller and MIDI config flags are never set
- edit history has a length of 0

The only applications I am aware of that are similar to this fingerprint are:
- CheeseTracker, but it always seems to write non-zero row highlights
- IT 2.14 without any patches, but it always writes a non-empty edit history (and is unlikely to write a reserved value of 0)

In addition to this fingerprint, for detecting old (broken) UNMO3 versions:
- If instrument mode is disabled but 4*smpNum zero bytes follow after the pattern parapointers, an UNMO3 version older than 2.4.0.1 was used.
- If only the "message" extra bit is set and two zero bytes are read after the parapointers, an UNMO3 version older than 2.4.0.1 was used.