Missing IT header data: Row highlights, pitch wheel depth

[saga]

Given that an effort was made to put some of the missing metadata (such as "made with tracker" information) into MO3 files, it would be sensible to also do this for IT row highlights and pitch wheel depth. In particular, this information can be required for MPTM files that use modern tempo mode (where the row highlights determine how long a row lasts) or instrument plugins (where the pitch wheel depth is used to translate portamentos into MIDI pitch wheel commands).

Pitch wheel information can be stored in a backwards-compatible way by using the per-instrument pitch wheel information that is already used for XM instruments, so no extra chunk needs to be introduced. For row highlights, I guess a new chunk could be added at the end of the song data.

I have attached an example IT file with row highlights set to 16/32 (rather than the default 4/16), and pitch wheel depth set to 13.

[Ian @ un4seen]

Here's a MO3 encoder update that adds a "PRHI" chunk for the "PHiliht" IT header value:

   www.un4seen.com/stuff/mo3.exe

The chunk will not be included if it has the default 4/16 value (or 0/0). This update will also store the "PWD" IT header value (when enabled in "Flags") in each instrument's pitch wheel depth field (if the instrument has a MIDI channel set), but I guess that isn't really needed because OpenMPT's "DWPM" chunk (containing per-instrument pitch wheel depths) will be included in the MO3 file anyway.

[saga]

Thanks for the quick update.

This update will also store the "PWD" IT header value (when enabled in "Flags")

Which flag is this based on? From what I can see IT will unconditionally read this value from the file, i.e. it does not depend on the flag for embedding the Zxx MIDI configuration.
While it is indeed less important for OpenMPT due to the per-instrument PWD setting, an accurate representation of a compressed IT file should still contain the PWD information in any case (although the number of MIDI-based IT files outside of OpenMPT is pretty low of course).

If it is possible, I would also suggest to place the PRHI chunk before the OMPT chunk, as the OMPT chunk may contain values that override PRHI chunk's signature if required. This is only ever the case if either of the two PRHI values is greater than 255, i.e. pretty rare, so simplifying the decoding logic by expecting PRHI to come first would be nice.

[Ian @ un4seen]

This update will also store the "PWD" IT header value (when enabled in "Flags")

Which flag is this based on?

In the ITTECH.TXT file, it says this under "Flags":

Bit 6: Use MIDI pitch controller, Pitch depth given by PWD

It sounds like pitch wheel depth should be ignored without that flag set but I've never used this stuff in Impulse Tracker myself, so perhaps not?

If it is possible, I would also suggest to place the PRHI chunk before the OMPT chunk, as the OMPT chunk may contain values that override PRHI chunk's signature if required. This is only ever the case if either of the two PRHI values is greater than 255, i.e. pretty rare, so simplifying the decoding logic by expecting PRHI to come first would be nice.

OK, I will change that in the next update.

[saga]

Right, OpenMPT always sets that flag, which is why I forgot about it. The reason I was wondering is that with the example file provided above ran through the new mo3.exe, I cannot see the pitch wheel value popping up in the generated MO3 instruments. Does it maybe check for the wrong flag?

[Ian @ un4seen]

The header "Flags" check looks OK but perhaps it's the instrument MIDI channel ("MCh") check that's causing the problem? The MO3 encoder ignores the instrument's other MIDI settings if that's set to 0/none. That's because MO3 doesn't have a "none" setting (0 = channel 1), so all MIDI settings being 0 is used to indicate "none".

[saga]

Ah yes, that explains it - there was no MIDI channel set.

[saga]

As I was testing the latest unmo3.exe from the OPL thread, I noticed that it doesn't seem to use the PRHI chunk and pitch wheel data from instruments when writing IT/MPTM files, so currently only Open. Would it be possible to write those values back to the IT header?

[Ian @ un4seen]

Here's an update that should restore those things in IT/MPTM files:

   www.un4seen.com/stuff/unmo3.exe

[saga]

It almost works. The highlights are written to the file indeed, but the "special" bit for enabling them (0x04) is not being set, so they will be ignored by a compliant IT player. I didn't check if it was written, but I think if the pitch wheel depth information is present, the "flags" bit 0x40 should be set as well.

Since settings these flags and values changes UNMO3's "fingerprint", this update will make it harder to detect such files (which can be useful for various compatibility hacks). Various trackers already use the 4-byte "reserved" field in IT's header (which in fact contains an encrypted edit timer when saved with IT) to identify themselves, so I think it would be sensible of UNMO3 e.g. wrote "MO3 " in that field.

[Ian @ un4seen]

It almost works. The highlights are written to the file indeed, but the "special" bit for enabling them (0x04) is not being set, so they will be ignored by a compliant IT player. I didn't check if it was written, but I think if the pitch wheel depth information is present, the "flags" bit 0x40 should be set as well.

Oh, I wasn't aware of the "special" bit. In that case, the MO3 encoder should probably also check that bit before retaining the "PHiliht" value. Here are updates for that:

   www.un4seen.com/stuff/mo3.exe
   www.un4seen.com/stuff/unmo3.exe

Flag 0x40 should already be set when "pwd" is set. Let me know if you find it isn't.

Since settings these flags and values changes UNMO3's "fingerprint", this update will make it harder to detect such files (which can be useful for various compatibility hacks). Various trackers already use the 4-byte "reserved" field in IT's header (which in fact contains an encrypted edit timer when saved with IT) to identify themselves, so I think it would be sensible of UNMO3 e.g. wrote "MO3 " in that field.

Are you sure there's a need to detect an UNMO3'd file? For playback quirk activation, it seems like it would be more important to know what the original tracker was.

[saga]

Yup, I can confirm that the two flags are set correctly in this build.

Are you sure there's a need to detect an UNMO3'd file? For playback quirk activation, it seems like it would be more important to know what the original tracker was.

In fact, both can be useful. I'm specifically thinking of stuff like this thread where we figured out that some UNMO3 versions wrote some bogus parapointers in IT headers. In this case, having some indication that the file was saved with MO3 can make it easier to develop such workarounds to keep those files playable. For the actual playback, it is indeed not relevant.