Author Topic: unmo3 as a library?  (Read 49288 times)

saga

  • Posts: 2152
unmo3 as a library?
« on: 26 Oct '08 - 22:31 »
Hey Ian,
I was wondering if you could provide the unmo3 thingy rather as library than an executable? That way, it would probably be easier to be implemented into other applications and you sure want this be done. :) I'm thinking about OpenModplug in this case, I joined the devteam some weeks ago. It would be a nice thing to have this as a library as this would be less confusing to users (what? 2 executables in the folder?) and prolly also easier to handle (call a function, pass the filename, get a pointer).

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #1 on: 27 Oct '08 - 14:52 »
That may be possible. Give it a few days, and I'll see what I can do :)

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #2 on: 15 Nov '08 - 13:44 »
any news on this? :)

what i forgot to mention above is that also the length of the uncompressed module has to be returned, so a function like this would be fine:

Code: [Select]
convertmo3(filename, pointer, length)and the library would fill in the pointer and length stuff.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #3 on: 17 Nov '08 - 15:06 »
OK, here's something to try :)

   www.un4seen.com/stuff/unmo3dll.zip

Let me know if there any problems with it.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #4 on: 17 Nov '08 - 17:57 »
i'll try it as soon as i can find some spare time and let you know if there's anything that's not clear. :)
I have one question in advance, though: is there a way to include the library into the program's exe?
« Last Edit: 17 Nov '08 - 17:59 by saga »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #5 on: 19 Nov '08 - 16:35 »
Would that be permitted by Modplug's GPL licence? I'm not sure.

Interestingly, there is a GPL UNMO3 package made from the unofficial specs, which should be fine for you to incorporate into Modplug...

   http://gnu.ethz.ch/debian/unmo3/

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #6 on: 19 Nov '08 - 19:43 »
i guess i'd have to talk with the other devs about gpl issues first then. although i have in mind that some license changes were planned / discussed some time ago, but i don't know exactly...

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #7 on: 18 Jan '09 - 19:36 »
okay, so today I've finally tried to work on the mo3 import again. :)
Everything is working nicely, but I've found that a little change in the .dll may be useful because of the lacking of a proper header in .MOD files.
Due to the way our code is structured, I need a little change in the parameters of UNMO3_Decode.
Instead of passing a filename, I would like to try to pass the pointer to the compressed module and the length and receive a new pointer and a new length. Could you please update the dll for this? :)
-saga

BTW: We've moved on to the BSD license now. :)

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #8 on: 19 Jan '09 - 15:30 »
Due to the way our code is structured, I need a little change in the parameters of UNMO3_Decode.
Instead of passing a filename, I would like to try to pass the pointer to the compressed module and the length and receive a new pointer and a new length. Could you please update the dll for this? :)

Yep, here's an update to try...

   www.un4seen.com/stuff/unmo3dll.zip

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #9 on: 19 Jan '09 - 17:45 »
Thanks a lot, that did the trick :)

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #10 on: 11 Oct '09 - 04:23 »
What is the license on this and is it okay to redistribute this? I'm thinking of using this instead of unmo3.exe for foo_dumb. It would be a lot more convenient and possibly more secure than piping the module out of unmo3.exe, as I don't really have many security checks on the pipe process. I also tried piping both in and out simultaneously, which failed, and probably not because I was trying to do it all in a single thread. I found this while digging in relation to Open ModPlug Tracker, heh.

(Scatterbrained post brought to you by lack of sleep, a first for me in months.)

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #11 on: 12 Oct '09 - 15:38 »
What is the license on this and is it okay to redistribute this? I'm thinking of using this instead of unmo3.exe for foo_dumb.

The UNMO3.DLL is free for non-commercial use (like UNMO3.EXE), so that sounds fine.

Zarggg

  • Posts: 1242
Re: unmo3 as a library?
« Reply #12 on: 14 Oct '09 - 15:36 »
Good to see you here, kode54. :)

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #13 on: 25 Oct '13 - 04:35 »
Any chance of Linux 32/64 bit .so or OS X fat .dylib versions of this?

EDIT: Having it signed would also be nice as well, but I bet Petite is incompatible with code signing.
« Last Edit: 25 Oct '13 - 05:14 by kode54 »

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #14 on: 25 Oct '13 - 08:35 »
And since OpenMPT might go 64-bit soon-ish, a Win64 build would be nice as well, while we're at it ;) A Linux .so would also be interesting for libopenmpt, I guess.
« Last Edit: 25 Oct '13 - 17:15 by saga »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #15 on: 25 Oct '13 - 17:12 »
I think that should be possible. I'll try to get something ready next week.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #16 on: 1 Nov '13 - 16:54 »
OK. Here's an updated UNMO3 library package for you to try...

   www.un4seen.com/stuff/unmo3lib.zip

Support is included for x86/x64 on Windows and Linux, and x86/x64/ppc on OSX. This update also adds a "no sample processing" option, which could be useful (faster) for when you just want to get info/texts without playing.

Let me know if you encounter any problems (it hasn't been tested much yet).

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #17 on: 1 Nov '13 - 19:32 »
OS X x86_64 verified working, and it seems to be quite a bit faster at decoding PvZ.mo3 (Plants vs. Zombies soundtrack) than the Windows 32 bit version. Of course, it could just be placebo.

One minor problem, I had to typedef or define DWORD myself. I decided to import stdint.h and typedef it from uint32_t.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #18 on: 1 Nov '13 - 19:50 »
Windows x64 tested and confirmed to be working. Thanks a lot! :)
It's kinda unfortunate that old library versions will crash when using the new UNMO3_Decode specification, though, not sure if anything in the interface could be changed to avoid that. Otherwise the host programm will have to check the DLL version, I guess. :)
« Last Edit: 1 Nov '13 - 20:25 by saga »

PSXGamerPro1

  • Posts: 258
Re: unmo3 as a library?
« Reply #19 on: 1 Nov '13 - 21:25 »
Or maybe It can check the MD5 of the dll. I know of a program that does that. And if the MD5 does not match the program will close saying "'<file name>.dll' Invalid."

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #20 on: 1 Nov '13 - 21:28 »
...which would result in way more code than importing UNMO3_GetVersion and you'd still have to differentiate between several DLL versions, so is a worse solution. Please go away with your senseless suggestions, you are not being helpful at all.
« Last Edit: 1 Nov '13 - 21:30 by saga »

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #21 on: 1 Nov '13 - 23:06 »
Or you could supply your own copy of the library so you always link against the version you developed for, assuming redistribution is compatible with the licenses of unmo3 and your program or library.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #22 on: 2 Nov '13 - 12:35 »
One minor problem, I had to typedef or define DWORD myself. I decided to import stdint.h and typedef it from uint32_t.

Oops. I had intentionally avoided using Windows types in the UNMO3.H header previously, but forgot to so in this update. I have now changed it to use "unsigned" instead of "DWORD".

It's kinda unfortunate that old library versions will crash when using the new UNMO3_Decode specification, though, not sure if anything in the interface could be changed to avoid that. Otherwise the host programm will have to check the DLL version, I guess. :)

Yep, you could use UNMO3_GetVersion to check that the appropriate DLL version is loaded. You don't necessarily need to call it though; it was introduced at the same time as the UNMO3_Decode "flags" parameter, so its presence alone is enough to confirm the UNMO3_Decode signature :)

Anyway, good to hear that the update is working well so far. Let me know if any problems do crop up.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #23 on: 2 Nov '13 - 14:01 »
Yeah, I guess that's the most sensible approach for now. Luckily, calling the new DLL from an old program expecting the two-parameter signature (without flags) still works, so that's at least something.

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #24 on: 12 Dec '13 - 04:09 »
A fairly minor problem I've discovered. Or at least it's minor because I can work around it rather than letting my users crash frequently.

Namely, the library does not appear to be thread safe. Unpacking more than one MO3 at a time from different threads results in crashes.

I can work around this by dynamically loading the module and copying it around to random names in the temporary folder when I need more than one instance at a time.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #25 on: 12 Dec '13 - 15:35 »
That's strange, as the library should be thread-safe. I quickly tried reproducing the problem with the following code, but it never happened.

Code: [Select]
#include <windows.h>
#include <stdlib.h>
#include <stdio.h>
#include "unmo3.h"

void *buf;
unsigned len;

DWORD WINAPI threadproc(void *p)
{
while (1) {
void *buf0=buf;
unsigned len0=len;
int r=UNMO3_Decode(&buf0, &len0, 0);
printf("%d: unmo3=%d\n", (int)p, r);
if (!r) UNMO3_Free(buf0);
}
return 0;
}

int main(int argc, char **argv)
{
FILE *f;
if (argc<2) return 0;
f=fopen(argv[1], "rb");
if (!f) return 0;
len=filelength(fileno(f));
buf=malloc(len);
fread(buf, len, 1, f);
fclose(f);
CreateThread(0, 0, threadproc, (void*)0, 0, 0);
CreateThread(0, 0, threadproc, (void*)1, 0, 0);
getchar();
return 0;
}

Can you produce the problem with something that?

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #26 on: 13 Dec '13 - 04:51 »
I am using version 2.4.0.3, and I am also calling LoadLibrary/GetProcAddress/FreeLibrary from each thread which unpacks an MO3. It was the original way I dealt with bundling the library in my component's directory before the player started adding the components' own directories to the DLL search path in turn while each component is being loaded. I suppose I could change that now.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #27 on: 13 Dec '13 - 13:44 »
I am using version 2.4.0.3

Ah, I think that would explain it. Thread-safety was added in 2.4.0.4. To ensure that the correct DLL version is loaded, you could check for the presence of the UNMO3_GetVersion function, which was also added in 2.4.0.4.

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #28 on: 14 Dec '13 - 01:53 »
The problem isn't that someone is providing the wrong version, since I bundle my own copy. I just neglected to update the component. Let me take care of that right now.

Dr. Fiemost

  • Posts: 13
Re: unmo3 as a library?
« Reply #29 on: 2 Jan '14 - 21:55 »
Is this library supposed to support mp3 samples? I get mostly silent output from the mo3 I've tried.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #30 on: 2 Jan '14 - 22:18 »
You're right, MP3 sample support seems to be broken in the latest version indeed!
Here's an example file to demonstrate the problem.
I also noticed that a previous version of unmo3.dll (dated 2011) had a problem with this particular MO3 file; the belltree sample had some silence in the loop part of the sample. This doesn't happen in the latest XMPlay however, so it's probably already fixed.
« Last Edit: 2 Jan '14 - 22:28 by saga »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #31 on: 3 Jan '14 - 16:12 »
Oops! There were some changes in the MP3 decoder, which the UNMO3 code hadn't been updated for yet. Here's an update that should sort it...

   www.un4seen.com/stuff/unmo3lib.zip

Let me know if it still gives you any trouble.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #32 on: 3 Jan '14 - 16:16 »
Yup, that seems to fix the problem. Thanks for the quick fix!

Dr. Fiemost

  • Posts: 13
Re: unmo3 as a library?
« Reply #33 on: 3 Jan '14 - 20:59 »
Thanks for the fix, works fine so far.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #34 on: 6 Feb '15 - 22:04 »
Since libopenmpt can use the unmo3 library as a "plug-in" to decode MO3 files, a couple of Android libopenmpt users have asked if this feature could be made on Android as well. Ian, would it be possible to port the library to Android as well?

CasualBoy

  • Posts: 5
Re: unmo3 as a library?
« Reply #35 on: 7 Feb '15 - 14:08 »
I am second to Saga proposal.
I would like to see Mo3 supported in my Android phone. ;D

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #36 on: 9 Feb '15 - 17:40 »
Since libopenmpt can use the unmo3 library as a "plug-in" to decode MO3 files, a couple of Android libopenmpt users have asked if this feature could be made on Android as well. Ian, would it be possible to port the library to Android as well?

I'll look into building an Android version this week. Have you already implemented the dynamic library loading for Android in your code? I recall that is a bit tricky/messy because the app's "lib" folder isn't in Android's library search path, so you need to provide the full path, which is tricky/messy to get in native code.

I would like to see Mo3 supported in my Android phone. ;D

If you happen to just want to play MO3 files, that is possible with the BASS library.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #37 on: 9 Feb '15 - 18:27 »
We didn't have a look yet, but I'll try to ask the people who're interested in the feature if they can help out. My own experience with Android is rather limited.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #38 on: 12 Feb '15 - 17:09 »
OK. An Android version of the unmo3 library has now been added...

   www.un4seen.com/stuff/unmo3lib.zip

It's untested so far, so let me know if you have any trouble with it. As I mentioned, loading it in native code may be tricky due to the issue of getting the full path. It'll probably be simplest to load it from Java first, and then the native code shouldn't need the full path to access it.

Code: [Select]
System.loadLibrary("unmo3");

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #39 on: 14 Feb '15 - 16:57 »
Hi,

I tested the library and it seems to be produce too much decoded data. Decode function returned success but the filesize became over 15MB (encoded size is 473484 bytes). The real decoded size is around 4.7MB. Also the filesize and decoded contents change if you decode the same data several times in a row. Tested on Android x86.
« Last Edit: 14 Feb '15 - 17:14 by haspor »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #40 on: 16 Feb '15 - 14:06 »
That's strange. I tried unmo3'ing a few files just now (on an ARMv7 device), and that seemed to go fine. In case the problem is something file-specific, please upload a MO3 file that you're having the problem with to have a look at here...

   ftp.un4seen.com/incoming/

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #41 on: 16 Feb '15 - 17:14 »
Hi,

this is the file: /incoming/BeyondNetwork.mo3

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #42 on: 17 Feb '15 - 20:29 »
Hi,

I uploaded Android ARMv7 device decoded file of BeyondNetwork.mo3:

 * ftp.un4seen.com/incoming/BeyondNetwork_AndroidARM_decoded.it

The decoded filesize is at least correct.

I decoded the BeyondNetwork.mo3 with unmo3.exe and compared it to the ARM decoded file. The beginning seems to be almost correct but starting from offset 0x46da2 its just zeroes.
« Last Edit: 17 Feb '15 - 20:54 by haspor »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #43 on: 18 Feb '15 - 13:47 »
I was able to reproduce the problem in an x86 Android emulator. Here's an update that seems to be working properly...

   www.un4seen.com/stuff/unmo3lib.zip

Let me know if it's still not working properly there.

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #44 on: 18 Feb '15 - 17:31 »
Hi,

I will test. I noticed the ARMv7 version is still the same. Could you provide some simple example how to get it working on ARMv7? Android.mk file + how to properly access the library functions.


Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #45 on: 19 Feb '15 - 14:35 »
In my tests, I was just reading the MO3 file to memory, passing that memory block to UNMO3_Decode, and then writing the returned memory block to a new file. That's all in native code, using fopen/etc for the file stuff.

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #46 on: 19 Feb '15 - 15:06 »
Hi,

that's exactly what I did.

fopen, fread, decode and write back to a new file.

I suppose there is just one way to load the .so file and call its functions.

How is it even possible to fail then...I'll paste my code here later

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #47 on: 19 Feb '15 - 19:59 »
Hi,

tried the new version, little improvement. The decoded size is now correct but the file data is now equal to data I get when decoding with Android ARMv7a device -> at the beginning its ok but later it gets corrupted. I used the beyondnetwork.mo3 file.

How did you verify the decoded data is ok? Did you try to play the decoded file? Do verify that the data is equal to unmo3.exe decoded content please. Also check the offset i mentioned above.
« Last Edit: 19 Feb '15 - 21:09 by haspor »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #48 on: 20 Feb '15 - 17:24 »
Yep, the unmo3'd file seems to be fine here, ie. it sounds fine when played and there isn't the block of 0s that's in the file you uploaded. Do you get the block of 0s with other MO3 files too? I wonder if the problem is that the new file content hasn't been fully flushed to disk; are you calling fclose to close the file? If that's not it, you could check whether the memory block actually contains 0 before it is written. For example...

Code: [Select]
for (int a=0x46da2; a<length; a++) {
if (buffer[a]!=0) {
__android_log_print(ANDROID_LOG_DEBUG, "unmo3", "not 0 at %x", a);
break;
}
}

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #49 on: 20 Feb '15 - 19:38 »
Here is my Android.mk file:
Code: [Select]
LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := unmo3
LOCAL_SRC_FILES := unmo3lib/$(TARGET_ARCH_ABI)/libunmo3.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE    := unmo3unpacker
LOCAL_SRC_FILES := unmo3test.cpp
LOCAL_SHARED_LIBRARIES := unmo3
include $(BUILD_SHARED_LIBRARY)

Here's the C code that suppose to decode the .mo3 stuff:
Code: [Select]

#include <stdlib.h>
#include <stdio.h>
#include <jni.h>
#include "com_example_unmo3test_MainActivity.h"

extern "C" {
}

#define WINAPI
#include "unmo3.h"

JNIEXPORT jint JNICALL Java_com_example_unmo3test_MainActivity_load_1and_1unpack(JNIEnv *env, jobject obj)
{
    int result;
    unsigned int filesize = 0;
    int comp_size = 0;
    void * filedata;
    FILE * file = NULL;
    FILE * file2 = NULL;

    filedata = malloc(6000000);

    file = fopen("/mnt/sdcard/Music/BeyondNetwork.mo3", "rb");
    fseek(file, 0, SEEK_END);
    comp_size = ftell(file);
    fseek(file, 0, SEEK_SET);

    fread(filedata, 1, comp_size, file);
    result = UNMO3_Decode(&filedata, &filesize, 1); <----- 2nd param must be comp_size and 3rd set to 0
    file2 = fopen("/mnt/sdcard/Music/output.it", "wb");
    fwrite(filedata, 1, filesize, file2); <------ filesize has to be comp_size instead of filesize
    fclose(file2);
    fclose(file);
    UNMO3_Free(filedata);

    return 1;
}

That's it. Let me know what's wrong with it. Also before the offset 0x46da2 the data is also corrupted, random dword sized blocks are zero.

EDIT: the code problem is solved now. I marked the errors there.
« Last Edit: 21 Feb '15 - 08:55 by haspor »

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #50 on: 21 Feb '15 - 07:44 »
I pointed it out to you on IRC, but I can tell you here as well, in case anyone else missed it. You need to set the size variable you pass in by pointer, to the correct size of the compressed data. And in case you didn't notice, it does not decompress in place, because it assumes your buffer is neither writable nor large enough for the uncompressed data. So the data pointer will be modified to point to the newly allocated buffer on success, which will need to be freed with UNMO3_Free when you're done with it.

Furthermore, it is unlikely that a decoded file will be 100% identical to the output of unmo3.exe, unless you compare against unmo3.dll for x86 32 bit. Floating point decoding of lossy samples like MP3 or Ogg Vorbis will vary by which platform the library is compiled for.
« Last Edit: 21 Feb '15 - 08:11 by kode54 »

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #51 on: 21 Feb '15 - 08:46 »
Hi,

yeah it works now but there are few differences still (not sample data related), see the attached screenshot.

First there is 1 byte change, then later those dword sized 00 00 00 00 differences about 100 of them before the sample data starts.

Thanks to kode54 for the guidance. I wasn't aware that i need to pass the encoded filesize to the decoder function, i was expecting the decoder to only return the decoded size to the variable.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #52 on: 21 Feb '15 - 11:42 »
The first difference shows that you are using an old version of UNMO3. Comparing the two files further stops making sense at this point.
« Last Edit: 21 Feb '15 - 12:02 by saga »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #53 on: 10 Jun '15 - 18:06 »
An iOS version been added to the UNMO3 library package, which is now up on the MO3 page ("other stuff" section).

Duc

  • Guest
Re: unmo3 as a library?
« Reply #54 on: 12 Aug '15 - 11:46 »
Hi, I would like to ask a question.
I'm currently modding PVZ, including music. I had a lot of Mp3 music files, so how to turn all those files into one mo3 file as PVZ music (1 mo3 file that has a lot of songs compressed in it)?

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #55 on: 13 Aug '15 - 07:44 »
Hi, I would like to ask a question.
I'm currently modding PVZ, including music. I had a lot of Mp3 music files, so how to turn all those files into one mo3 file as PVZ music (1 mo3 file that has a lot of songs compressed in it)?

It'll be a dirty trick.

And I suggest you start with lossless or high quality compressed music to start with.

Basically, you want your music in 16 bit PCM WAV format, stereo if you want simple, mono pairs if you want to assemble them into the tracker.

What you'll be doing is loading these PCM files as samples into Open Modplug Tracker.

You'll want to make note of the original order list of the original MO3, just in case the game uses raw orders for song numbers.

You'll be recreating that order list with empty patterns, with your songs loaded as samples. Then you'll be using note on commands at the start of each subsong's order list (first pattern there) to start each song from its sample numbers, then use a combination of break to order in channel 0 and break to row in channel 1 to jump back to that exact order/row, while your music sample will either play out completely, or loop according to whatever loop settings you have applied to it.

Once you are satisfied that you have a working .XM or .IT with all of your PCM songs shoved into it, you'll be packing this with the MO3 compressor, and using Vorbis for the samples you supplied. (This is why you should start off with uncompressed high quality music.)

I would say use existing MP3 or Ogg Vorbis files, but I do not know of any way to insert existing compressed samples into an MO3.

Even if that were possible, it still doesn't handle the need to create a module template that is nearly identical in order structure to the file you'll be replacing.

We are talking about Plants vs. Zombies here, right?

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #56 on: 2 Apr '16 - 20:30 »
the latest unmo3 library seems to have a decoding problem on Android v4.3 (at least on x86). Uploaded 3 binaries to your FTP.

Filenames:

 * BeyondTheNetwork.mo3 (original)
 * BeyondTheNetwork_dumped_new.bin (dumped using the latest unmo3 library)
 * BeyondTheNetwork_dumped_old.bin (dumped using the unmo3 library from 2015 february or so)


Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #57 on: 4 Apr '16 - 18:23 »
Oh yes, there is indeed a problem with the Android version. It's actually an issue that has come up before with other code and is related to the GCC versions (4.8/9) included in recent Android NDK releases. It affects code like this:

Code: [Select]
do *dest++=dest[offset]; while (--c);

The expected result is that the dest pointer is incremented after copying from dest+offset, which is what happened with GCC 4.6 from older NDK (and Clang from newer NDK), but that isn't the case with GCC 4.8/9; it's incrementing first, so it ends up copying from dest+offset+1. An updated Android build to get around that is up now in the UNMO3 library package.

manx

  • Posts: 17
Re: unmo3 as a library?
« Reply #58 on: 4 Apr '16 - 18:57 »
Oh yes, there is indeed a problem with the Android version. It's actually an issue that has come up before with other code and is related to the GCC versions (4.8/9) included in recent Android NDK releases. It affects code like this:

Code: [Select]
do *dest++=dest[offset]; while (--c);

The expected result is that the dest pointer is incremented after copying from dest+offset, which is what happened with GCC 4.6 from older NDK (and Clang from newer NDK), but that isn't the case with GCC 4.8/9; it's incrementing first, so it ends up copying from dest+offset+1. An updated Android build to get around that is up now in the UNMO3 library package.


GCC is actually free to do so, quoting C99:

Quote
5.1.2.3 Program execution 2:
Accessing a volatile object, modifying an object, modifying a file, or calling a function that does any of those operations are all side effects, 11) which are changes in the state of the execution environment. Evaluation of an expression may produce side effects. At certain specified points in the execution sequence called sequence points, all side effects of previous evaluations shall be complete and no side effects of subsequent evaluations shall have taken place. (A summary of the sequence points is given in annex C.)

Annex C does not list the assignment operator as a sequence point, only the full expression as a whole.

And in particular:

Quote
6.5 Expressions 2:
Between the previous and next sequence point an object shall have its stored value modified at most once by the evaluation of an expression. 72) Furthermore, the prior value shall be read only to determine the value to be stored. 73)
73) This paragraph renders undefined statement expressions such as
Code: [Select]
i = ++i + 1;
a[i++] = i;
while allowing
Code: [Select]
i = i + 1;
a[i] = i;

haspor

  • Posts: 19
Re: unmo3 as a library?
« Reply #59 on: 4 Apr '16 - 20:43 »
Tested the new version of the library. Now there is another kind of problem, it sounds like it is skipping some parts all the time.  Same testsong and same test environment.

« Last Edit: 5 Apr '16 - 07:40 by haspor »

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #60 on: 5 Apr '16 - 04:41 »
Subsongs have nothing to do with the formats that mo3/unmo3 support compression for. Subsongs are detected and supported through heuristic checks against the pattern and order list, to see if there are playable orders which are not used by the first order in the list. I'd love to know how it scans so quickly, since I'd love to incorporate that into DUMB, or use it with BASS itself, because it's way faster than my primitive full song scan, mainly because my full song scan does a null mixing pass that at the very least advances all of the samples.

saga

  • Posts: 2152
Re: unmo3 as a library?
« Reply #61 on: 5 Apr '16 - 11:57 »
I think XMPlay's and OpenMPT's subsong scan speed are pretty comparable, and there is no black magic involved in OpenMPT - except maybe that the pattern scanner is a separate implementation from the normal module renderer and thus needs to re-implement several effects. Ugly, but works. As a bonus, pattern loops are implemented differently (only scanned once) and cannot get stuck in an infinite loop. Feel free to get inspired by this idea. ;)

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #62 on: 5 Apr '16 - 13:25 »
The expected result is that the dest pointer is incremented after copying from dest+offset...


GCC is actually free to do so, quoting C99:
...

I should have said that's the result I expected (eg. given GCC 4.6's behaviour), not necessarily what's expected by the standards :)

Tested the new version of the library. Now there is another kind of problem, it sounds like it is skipping some parts all the time.  Same testsong and same test environment.

Oops. There was another instance similar to the one above, which was affecting the pattern data. It should be all sorted now. Another Android update is up in the UNMO3 library package now.

Subsongs have nothing to do with the formats that mo3/unmo3 support compression for. Subsongs are detected and supported through heuristic checks against the pattern and order list, to see if there are playable orders which are not used by the first order in the list. I'd love to know how it scans so quickly, since I'd love to incorporate that into DUMB, or use it with BASS itself, because it's way faster than my primitive full song scan, mainly because my full song scan does a null mixing pass that at the very least advances all of the samples.

XMPlay basically plays the file without doing any mixing, and marks the positions that it visits in the order list. Once it detects an end or loop (that's the trickiest part), it then checks for any unvisited positions and starts the process again from the first of those (if there are any). If you can't skip the mixing step, then you could use a very low sample rate (eg. 1000 Hz) to minimize its impact on speed.

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #63 on: 2 Nov '16 - 05:45 »
The Win32 version appears to have a memory leak. Feeding the same input module in the same input buffer to UNMO3_Decode and UNMO3_Free in rapid succession starts to increase memory usage, according to the friend who tested it. I need to try this myself as well. According to them, 3MB at the start to 18MB at the one minute mark, and 97MB at the 7 minute mark. Using Vorbis for the sample compression.

mudlord

  • Posts: 7
Re: unmo3 as a library?
« Reply #64 on: 2 Nov '16 - 07:39 »
Does unmo3 use any threads? I think I found a odd bug in Windows 10 Anniversary Edition, or it could be a kernel bug related to that OS.

Sometimes my demo would crash on UNMO3_Decode inside a kernel function called NtWaitforAlertByThreadId.


edit -- no i was wrong, crashes inside the module player.
« Last Edit: 2 Nov '16 - 07:45 by mudlord »

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #65 on: 2 Nov '16 - 17:20 »
The Win32 version appears to have a memory leak. Feeding the same input module in the same input buffer to UNMO3_Decode and UNMO3_Free in rapid succession starts to increase memory usage, according to the friend who tested it. I need to try this myself as well. According to them, 3MB at the start to 18MB at the one minute mark, and 97MB at the 7 minute mark. Using Vorbis for the sample compression.

Oops! There was indeed a little leak: the memory used for sample filenames wasn't being released. Here's an update that should fix the problem:

   www.un4seen.com/stuff/unmo3.dll

Let me know if you/they still see any leaks.

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #66 on: 3 Nov '16 - 02:25 »
Does this affect other platforms as well? I am also distributing what is possibly an old version for macOS, with Cog.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #67 on: 3 Nov '16 - 15:17 »
Yes, it does affect the other platforms too. I will post an update for all platforms shortly.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #68 on: 4 Nov '16 - 13:19 »
The UNMO3 library update (2.4.1.1) for all platforms is now up on the MO3 page.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #69 on: 4 Nov '16 - 16:14 »
I noticed that an S3M pattern parapointers bug was introduced in the 2.4.1 UNMO3 release, affecting both the standalone and library versions. So another update (2.4.1.2) of both is up now on the MO3 page. The OSX and iOS versions have also switched to using the OS's MP3 decoder (own decoder is still used on other platforms).

kode54

  • Posts: 100
Re: unmo3 as a library?
« Reply #70 on: 4 Nov '16 - 17:07 »
So long as you're aware of iOS and macOS decoder behavior regarding chopping off the decoder delay. Or does it also chop off the encoder delay? I just know that it chops off some standard delay before sending the data to the calling program.

Ian @ un4seen

  • Administrator
  • Posts: 20210
Re: unmo3 as a library?
« Reply #71 on: 4 Nov '16 - 17:55 »
Yes, the UNMO3 update reduces its own delay chopping to account for the decoder's delay chopping, so that should be fine. Let me know if you do encounter any problems though.