Topic: unmo3 as a library?

[kode54]

I pointed it out to you on IRC, but I can tell you here as well, in case anyone else missed it. You need to set the size variable you pass in by pointer, to the correct size of the compressed data. And in case you didn't notice, it does not decompress in place, because it assumes your buffer is neither writable nor large enough for the uncompressed data. So the data pointer will be modified to point to the newly allocated buffer on success, which will need to be freed with UNMO3_Free when you're done with it.

Furthermore, it is unlikely that a decoded file will be 100% identical to the output of unmo3.exe, unless you compare against unmo3.dll for x86 32 bit. Floating point decoding of lossy samples like MP3 or Ogg Vorbis will vary by which platform the library is compiled for.

[haspor]

Hi,

yeah it works now but there are few differences still (not sample data related), see the attached screenshot.

First there is 1 byte change, then later those dword sized 00 00 00 00 differences about 100 of them before the sample data starts.

Thanks to kode54 for the guidance. I wasn't aware that i need to pass the encoded filesize to the decoder function, i was expecting the decoder to only return the decoded size to the variable.

[saga]

The first difference shows that you are using an old version of UNMO3. Comparing the two files further stops making sense at this point.

[Ian @ un4seen]

An iOS version been added to the UNMO3 library package, which is now up on the MO3 page ("other stuff" section).

[Duc]

Hi, I would like to ask a question.
I'm currently modding PVZ, including music. I had a lot of Mp3 music files, so how to turn all those files into one mo3 file as PVZ music (1 mo3 file that has a lot of songs compressed in it)?

[kode54]

Hi, I would like to ask a question.
I'm currently modding PVZ, including music. I had a lot of Mp3 music files, so how to turn all those files into one mo3 file as PVZ music (1 mo3 file that has a lot of songs compressed in it)?

It'll be a dirty trick.

And I suggest you start with lossless or high quality compressed music to start with.

Basically, you want your music in 16 bit PCM WAV format, stereo if you want simple, mono pairs if you want to assemble them into the tracker.

What you'll be doing is loading these PCM files as samples into Open Modplug Tracker.

You'll want to make note of the original order list of the original MO3, just in case the game uses raw orders for song numbers.

You'll be recreating that order list with empty patterns, with your songs loaded as samples. Then you'll be using note on commands at the start of each subsong's order list (first pattern there) to start each song from its sample numbers, then use a combination of break to order in channel 0 and break to row in channel 1 to jump back to that exact order/row, while your music sample will either play out completely, or loop according to whatever loop settings you have applied to it.

Once you are satisfied that you have a working .XM or .IT with all of your PCM songs shoved into it, you'll be packing this with the MO3 compressor, and using Vorbis for the samples you supplied. (This is why you should start off with uncompressed high quality music.)

I would say use existing MP3 or Ogg Vorbis files, but I do not know of any way to insert existing compressed samples into an MO3.

Even if that were possible, it still doesn't handle the need to create a module template that is nearly identical in order structure to the file you'll be replacing.

We are talking about Plants vs. Zombies here, right?

[haspor]

the latest unmo3 library seems to have a decoding problem on Android v4.3 (at least on x86). Uploaded 3 binaries to your FTP.

Filenames:

 * BeyondTheNetwork.mo3 (original)
 * BeyondTheNetwork_dumped_new.bin (dumped using the latest unmo3 library)
 * BeyondTheNetwork_dumped_old.bin (dumped using the unmo3 library from 2015 february or so)

[Ian @ un4seen]

Oh yes, there is indeed a problem with the Android version. It's actually an issue that has come up before with other code and is related to the GCC versions (4.8/9) included in recent Android NDK releases. It affects code like this:

Code: [Select]

do *dest++=dest[offset]; while (--c);

The expected result is that the dest pointer is incremented after copying from dest+offset, which is what happened with GCC 4.6 from older NDK (and Clang from newer NDK), but that isn't the case with GCC 4.8/9; it's incrementing first, so it ends up copying from dest+offset+1. An updated Android build to get around that is up now in the UNMO3 library package.

[manx]

Oh yes, there is indeed a problem with the Android version. It's actually an issue that has come up before with other code and is related to the GCC versions (4.8/9) included in recent Android NDK releases. It affects code like this:

Code: [Select]

do *dest++=dest[offset]; while (--c);

The expected result is that the dest pointer is incremented after copying from dest+offset, which is what happened with GCC 4.6 from older NDK (and Clang from newer NDK), but that isn't the case with GCC 4.8/9; it's incrementing first, so it ends up copying from dest+offset+1. An updated Android build to get around that is up now in the UNMO3 library package.

GCC is actually free to do so, quoting C99:

5.1.2.3 Program execution 2:
Accessing a volatile object, modifying an object, modifying a file, or calling a function that does any of those operations are all side effects, 11) which are changes in the state of the execution environment. Evaluation of an expression may produce side effects. At certain specified points in the execution sequence called sequence points, all side effects of previous evaluations shall be complete and no side effects of subsequent evaluations shall have taken place. (A summary of the sequence points is given in annex C.)

Annex C does not list the assignment operator as a sequence point, only the full expression as a whole.

And in particular:

6.5 Expressions 2:
Between the previous and next sequence point an object shall have its stored value modified at most once by the evaluation of an expression. 72) Furthermore, the prior value shall be read only to determine the value to be stored. 73)
73) This paragraph renders undefined statement expressions such as

Code: [Select]

i = ++i + 1;
a[i++] = i;
while allowing

Code: [Select]

i = i + 1;
a[i] = i;


[haspor]

Tested the new version of the library. Now there is another kind of problem, it sounds like it is skipping some parts all the time.  Same testsong and same test environment.

[kode54]

Subsongs have nothing to do with the formats that mo3/unmo3 support compression for. Subsongs are detected and supported through heuristic checks against the pattern and order list, to see if there are playable orders which are not used by the first order in the list. I'd love to know how it scans so quickly, since I'd love to incorporate that into DUMB, or use it with BASS itself, because it's way faster than my primitive full song scan, mainly because my full song scan does a null mixing pass that at the very least advances all of the samples.

[saga]

I think XMPlay's and OpenMPT's subsong scan speed are pretty comparable, and there is no black magic involved in OpenMPT - except maybe that the pattern scanner is a separate implementation from the normal module renderer and thus needs to re-implement several effects. Ugly, but works. As a bonus, pattern loops are implemented differently (only scanned once) and cannot get stuck in an infinite loop. Feel free to get inspired by this idea.

[Ian @ un4seen]

The expected result is that the dest pointer is incremented after copying from dest+offset...

GCC is actually free to do so, quoting C99:
...

I should have said that's the result I expected (eg. given GCC 4.6's behaviour), not necessarily what's expected by the standards

Tested the new version of the library. Now there is another kind of problem, it sounds like it is skipping some parts all the time.  Same testsong and same test environment.

Oops. There was another instance similar to the one above, which was affecting the pattern data. It should be all sorted now. Another Android update is up in the UNMO3 library package now.

Subsongs have nothing to do with the formats that mo3/unmo3 support compression for. Subsongs are detected and supported through heuristic checks against the pattern and order list, to see if there are playable orders which are not used by the first order in the list. I'd love to know how it scans so quickly, since I'd love to incorporate that into DUMB, or use it with BASS itself, because it's way faster than my primitive full song scan, mainly because my full song scan does a null mixing pass that at the very least advances all of the samples.

XMPlay basically plays the file without doing any mixing, and marks the positions that it visits in the order list. Once it detects an end or loop (that's the trickiest part), it then checks for any unvisited positions and starts the process again from the first of those (if there are any). If you can't skip the mixing step, then you could use a very low sample rate (eg. 1000 Hz) to minimize its impact on speed.

[kode54]

The Win32 version appears to have a memory leak. Feeding the same input module in the same input buffer to UNMO3_Decode and UNMO3_Free in rapid succession starts to increase memory usage, according to the friend who tested it. I need to try this myself as well. According to them, 3MB at the start to 18MB at the one minute mark, and 97MB at the 7 minute mark. Using Vorbis for the sample compression.

[mudlord]

Does unmo3 use any threads? I think I found a odd bug in Windows 10 Anniversary Edition, or it could be a kernel bug related to that OS.

Sometimes my demo would crash on UNMO3_Decode inside a kernel function called NtWaitforAlertByThreadId.


edit -- no i was wrong, crashes inside the module player.

[Ian @ un4seen]

The Win32 version appears to have a memory leak. Feeding the same input module in the same input buffer to UNMO3_Decode and UNMO3_Free in rapid succession starts to increase memory usage, according to the friend who tested it. I need to try this myself as well. According to them, 3MB at the start to 18MB at the one minute mark, and 97MB at the 7 minute mark. Using Vorbis for the sample compression.

Oops! There was indeed a little leak: the memory used for sample filenames wasn't being released. Here's an update that should fix the problem:

   www.un4seen.com/stuff/unmo3.dll

Let me know if you/they still see any leaks.

[kode54]

Does this affect other platforms as well? I am also distributing what is possibly an old version for macOS, with Cog.

[Ian @ un4seen]

Yes, it does affect the other platforms too. I will post an update for all platforms shortly.

[Ian @ un4seen]

The UNMO3 library update (2.4.1.1) for all platforms is now up on the MO3 page.

[Ian @ un4seen]

I noticed that an S3M pattern parapointers bug was introduced in the 2.4.1 UNMO3 release, affecting both the standalone and library versions. So another update (2.4.1.2) of both is up now on the MO3 page. The OSX and iOS versions have also switched to using the OS's MP3 decoder (own decoder is still used on other platforms).

[kode54]

So long as you're aware of iOS and macOS decoder behavior regarding chopping off the decoder delay. Or does it also chop off the encoder delay? I just know that it chops off some standard delay before sending the data to the calling program.

[Ian @ un4seen]

Yes, the UNMO3 update reduces its own delay chopping to account for the decoder's delay chopping, so that should be fine. Let me know if you do encounter any problems though.