Author Topic: Playlist shows empty rows, library borked?  (Read 1071 times)

Dhry

  • Posts: 64
Playlist shows empty rows, library borked?
« on: 10 Jun '17 - 03:03 »
Does XMPlay have a maximum number of items in the library? I might have exceeded it.

Using XMPlay 3.8.2.27. I downloaded the entire Modland archive just for fun. I've added several large directories to the library and they all seem to be added incrementally and properly. But after I got about 190,000 files, weird stuff is happening. Removing entries from the options/playlist/library area doesn't shrink down the list of entries shown at the top right of the library window. And if I scroll all the way down to the bottom of the list, there is a huge amount of completely blank rows. If I click on any of the blank rows I get an error and XMPlay crashes. However, if I use the scrollbar to get to just where the entries end, and then start paging down via keyboard, the rows light up with valid entries on the way down, all the way to the end.

If I delete the xmplay.library file I can zero everything out but all the folders I added to the library also disappear.

How do I force a complete reread/refresh of the library without manually deleting and readding all of the folders to monitor, so the number of tracks accurately is accurately represented?

Dhry

saga

  • Posts: 2183
Re: Playlist shows empty rows, library borked?
« Reply #1 on: 10 Jun '17 - 13:58 »
Just to be sure, maybe check XMPlay's memory usage while adding/scanning the files. 190,000 files doesn't sound like it should be a problem for XMPlay itself but maybe some memory is being leaked (e.g. by plugins) and it eventually fails to allocate memory for all playlist entries.
For refreshing, have you tried "Refresh info from file" when right-clicking a playlist entry (or a selection of entries)?

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #2 on: 10 Jun '17 - 16:34 »
Quote
check XMPlay's memory usage while adding/scanning the files. 190,000 files doesn't sound like it should be a problem for XMPlay itself but maybe some memory is being leaked (e.g. by plugins) and it eventually fails to allocate memory for all playlist entries.

I have 32Gb RAM on this box. XMPlay isn't using very much at all, see attachment.

The big chunk of empty space at the bottom of the library list appears if I use the scrollbar handle to drag down into that area. Again, if I drag up to where lib entries exist and then use the keyboard to page down, the entries in the blank area light up and I can scroll down to the bottom of the list that way. I selected a chunk of files (many of these are Nintendo .SPC files containing Japanese characters) and refreshed the file info, didn't do anything.

I'm going to try removing some plugins. I have seen a couple of crashes recently with Fraggie's Delix (using the latest 9.1), that could be the culprit. Even so, is there any way to just fully force a library refresh/rescan?

DRS




Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #3 on: 10 Jun '17 - 17:01 »
Renamed several plugins including Delix, didn't do anything. I still saw a stack of .SPC files in there despite not having anything that played them anymore. Went into the filesystem and renamed the dir they were contained in. Back to XMPlay and removed all dead entries. That got rid of thousands of files from the library list but there is still blank space at the bottom if I scroll there with the scrollbar handle. All filenames appear fine up if I page down or cursor down into that area and there now aren't any filenames that look particularly concerning.

I just renamed three other massive folders from Modland - Gameboy Sound Format, Playstation Sound Format, and Nintendo DS Sound Format, all of which were still listed in the library despite there being no players for the modules. Then did a remove dead entries. That blew away about 70,000 files from the library and NOW I can scroll to the bottom without any blank space (79,664 entries currently). That's still too high because according to the entries value for all monitored folders, there should only be 58,756 entries listed. XMPlay's folder monitoring only seems to be additive, not subtractive. Is this correct?

DRS

saga

  • Posts: 2183
Re: Playlist shows empty rows, library borked?
« Reply #4 on: 10 Jun '17 - 17:05 »
Quote
I have 32Gb RAM on this box.
That doesn't matter since XMPlay is a 32-bit application. Its memory consumption is capped at either 2GB (without high address aware flag) or 4GB.
Are those screenshots taken while the entries are being added/scanned? As I said it might have been a problem while this was being done, and not when reloading the library afterwards.

Quote
Even so, is there any way to just fully force a library refresh/rescan?
Did you try my suggestion with "Refresh info from file"? Are those "empty" playlist entries actually still associated with a file or are they really empty? You can verify this by right-clicking them and choosing "Track info".

Quote
I still saw a stack of .SPC files in there despite not having anything that played them anymore.
XMPlay only knows if it can play a file after it has been scanned. If it cannot be played, it will be drawn with a strike-through line. Until the scan is completed, the file just remains "potentially playable".

Anyway, at this point it's probably only Ian who can give any further recommendations

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Playlist shows empty rows, library borked?
« Reply #5 on: 12 Jun '17 - 17:23 »
Does XMPlay have a maximum number of items in the library? I might have exceeded it.

There isn't a fixed limit, but there will be some limit due to available memory. XMPlay enables Window's IMAGE_FILE_LARGE_ADDRESS_AWARE option, so the memory limit will be up to 4GB.

Using XMPlay 3.8.2.27. I downloaded the entire Modland archive just for fun. I've added several large directories to the library and they all seem to be added incrementally and properly. But after I got about 190,000 files, weird stuff is happening. Removing entries from the options/playlist/library area doesn't shrink down the list of entries shown at the top right of the library window. And if I scroll all the way down to the bottom of the list, there is a huge amount of completely blank rows. If I click on any of the blank rows I get an error and XMPlay crashes. However, if I use the scrollbar to get to just where the entries end, and then start paging down via keyboard, the rows light up with valid entries on the way down, all the way to the end.

Please upload your XMPLAY.LIBRARY file to have a look at here:

   ftp.un4seen.com/incoming/

How do I force a complete reread/refresh of the library without manually deleting and readding all of the folders to monitor, so the number of tracks accurately is accurately represented?

So long as you haven't disabled dead track checking (via "NoCheckDead"), then XMPlay will check the existence of library (and playlist) entries upon launch. You can remove any missing files via the "Remove dead" option (right-click the '-' button).

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #6 on: 12 Jun '17 - 21:54 »
Cheers Ian. I'll attach the library file when I get home tonight.

Quote
Did you try my suggestion with "Refresh info from file"? Are those "empty" playlist entries actually still associated with a file or are they really empty? You can verify this by right-clicking them and choosing "Track info".

I had mentioned in my original post that when I clicked on any blank row in the playlist, it immediately crashes XMPlay. That was still true for right as well as left clicks. However per my more recent posts I have got it to the point where there are no more blank rows in the playlist, although the number of items in the playlist mismatches with the number of items shown in the item count of all monitored folders. I will have to try re-borking the filelist and see if I can trigger this condition again, likely possible by readding the massive folders from modland.

Quote
You can remove any missing files via the "Remove dead" option

Understood. I guess I was wondering if there is a process not just for removing dead entries, but for removing ALL entries from the playlist and then rebuilding it from scratch and zero items. In other words, the same effect as physically deleting the xmplay.library file except without also deleting the details of all the monitored folders. I didn't realise that the list of monitored folders AND their contents was all in the same file. It would be great if Remove Dead had an option to be fully automated rather than a user manually having to click a button to trigger the action, so if the network drive (or virtual drive) that hosted the tracks went away, the library would automatically and quietly reduce in content.

Quote
Are those screenshots taken while the entries are being added/scanned? As I said it might have been a problem while this was being done, and not when reloading the library afterwards.

No, the blank playlist screenshot was taken well after scanning had completed. Verified by waiting several minutes and checking that no CPU was being used by XMPlay nor any changes occurring in memory usage.

Last question: My Delix (rev 9) plugin (from here https://ssz.bitbucket.io/input_plugins.html#delix) supposedly plays 140 filetypes, but the "supported filetypes" box in Plugins -> Input when you highlight the plugin, is completely empty. Same with the Extended Module Player Library plugin (libXMP 4.3.3). Every other plugin I have shows file extensions except those two. Is that.. a problem? I'm trying to figure out whether these plugins are the cause of potential issues with the library.

DRS

saga

  • Posts: 2183
Re: Playlist shows empty rows, library borked?
« Reply #7 on: 12 Jun '17 - 22:45 »
Quote
Every other plugin I have shows file extensions except those two. Is that.. a problem?
No, it's not a problem and this fact itself is not related to the issue (but of course the plugins may still be responsible for some of the issues, that can never be ruled out).
A plugin can optionally report which file extensions it supports, but XMPlay will try all plugins until it finds one that can decode a file. Neither XMPlay nor most plugins blindly trust file extensions.
The "problem" with Delix in particular is that the player engines it uses for decoding do not really report any file extensions (I think), or the module formats they support do not have a file extension to begin with.

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #8 on: 13 Jun '17 - 05:45 »
Thanks Saga.

Ian, I uploaded the file as requested. The playlist is showing more files than the total amount of files in the monitored folders (I provided screenshots) including files that were contained in folders that WERE in monitored folders but that I deleted. No amount of waiting or clicking "remove dead" will shrink the playlist down. I have not as yet been able to bork the library to the point where blank rows are showing but will keep working on that.

XMPlay 3.8.2.27.

Cheers
DRS

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #9 on: 13 Jun '17 - 07:55 »
So XMPlay's "remove dead" option only appears to work if a file in the library no longer exists in the filesystem. If the file is in the library but the folder that it was pulled from is removed from "monitored folders", then the file doesn't vanish from the library (because it still exists in the filesystem). Basically meaning the only way to get the files removed from the library is either via manually deleting them or by completely renaming the folders the files are in, forcing XMPlay to consider them "dead".

If the monitored folders option could work both ways - both adding to and subtracting from the library automatically as folders are added and removed from the monitored list, that would be awesome.

I just noticed something else. If I manually delete all the files from the library, it disables the folder monitoring. Even restarting XMPlay, the library will never autopopulate out of the existing monitored folders list until you manually remove and the readd those folders in the config. Ugh.

DRS

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Playlist shows empty rows, library borked?
« Reply #10 on: 13 Jun '17 - 16:27 »
Ian, I uploaded the file as requested. The playlist is showing more files than the total amount of files in the monitored folders (I provided screenshots) including files that were contained in folders that WERE in monitored folders but that I deleted. No amount of waiting or clicking "remove dead" will shrink the playlist down. I have not as yet been able to bork the library to the point where blank rows are showing but will keep working on that.

Did you see red lines appear over the library entries? The dead track scanning is done in the background at low priority, so it may take a while for all of the entries to be checked. Trying your library here, all entries were eventually detected as dead, and the "Remove dead" option removed them all.

Let me know if you do get the blank rows issue to happen again. That seems like the more puzzling issue.

I just noticed something else. If I manually delete all the files from the library, it disables the folder monitoring. Even restarting XMPlay, the library will never autopopulate out of the existing monitored folders list until you manually remove and the readd those folders in the config. Ugh.

The folder monitoring shouldn't get disabled in that scenario, but it will only add files that are new/modified since it last checked the folder. If you right-click on the monitored folder in the "Monitored folders" list and select "Rescan", then the folder's old entries should get added again.

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #11 on: 13 Jun '17 - 18:36 »
Quote
If you right-click on the monitored folder in the "Monitored folders" list and select "Rescan", then the folder's old entries should get added again.

Ah, that's basically just want I want (although automatic would be great, rather than manually doing this to twenty or more folders, or even a single button to rescan all).

Quote
Trying your library here, all entries were eventually detected as dead, and the "Remove dead" option removed them all.

Right, and that's what I expected. The reason would be that my library is pointing to files/folders that don't exist at all in your environment. They DO exist in my filesystem, therefore never get classified as dead. But they never disappear either. I could remove ALL the folders from the monitored folders list and the library would still contain ~100k files - until I delete them manually.

There's an app called Everything which monitors folders in realtime and can add the contents to a database and remove them in a split second. Foobar's folder monitoring also does the same thing. I'm not sure if there's an easy library you can call that can hook into the file allocation table of a drive somehow to make folder content addition/truncation silent and automatic?

Quote
it will only add files that are new/modified since it last checked the folder

Right, that appears to be the case. So somehow it saves a pointer or cache of previously read entries that isn't cleared out until the folder entry itself is removed from the monitored folders list. Or until a right-click-rescan.

Quote
Let me know if you do get the blank rows issue to happen again.

I think I did this morning, I left it scanning as I had to head to work but there were definitely blanks at the bottom. I'll check and see if it's borked when I get home this evening.

Cheers
DRS

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #12 on: 14 Jun '17 - 07:03 »
Borked playlist with blank rows at the bottom has been uploaded.

DRS

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #13 on: 14 Jun '17 - 17:16 »
The playlist definitely has blank rows at the bottom, for what I would estimate to be several hundred entries. They are from the Modland S98 folder (ftp://ftp.modland.com/pub/modules/S98/). I know this because as previously mentioned, the rows are blank only if I drag down to it using the scrollbar gadget. If I drag to where the final visible entries are, click one and then use PGDN on my keyboard the entries will all be visible.

I tried right-clicking on the S98 folder in the XMPlay configuration and selecting Rescan but it did nothing. At least nothing that affected the condition of the library in terms of making those entries visible.

Just noticed something weird. I was using the mousewheel to scroll into "no man's land" when I reached a point where the drag gadget in the scrollbar disappeared. See attached. I am using Windows Classic 2 by Amit. Scroll up and it reappears again. Seems there's a point at which the number of entries in the library overflows the ability of the scrollbar gadget to granularize? I just tried some other more recent skins such as Rated Alpha and it still happens.

DRS

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Playlist shows empty rows, library borked?
« Reply #14 on: 14 Jun '17 - 17:21 »
Right, and that's what I expected. The reason would be that my library is pointing to files/folders that don't exist at all in your environment. They DO exist in my filesystem, therefore never get classified as dead. But they never disappear either. I could remove ALL the folders from the monitored folders list and the library would still contain ~100k files - until I delete them manually.

Oh, I think I see what you want now. You would like all files from a monitored folder to be removed from the library when monitoring of that folder is stopped? I think that option can be added.

Borked playlist with blank rows at the bottom has been uploaded.

That's strange, I'm not seeing any empty lines in the library when I load it here. How many entries does XMPlay say the library has when you load it there? I'm seeing 147348 here. What do you see if you right-click on an empty line and select "Track info"? In case it's making a difference, please upload your XMPLAY.INI file too.

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #15 on: 14 Jun '17 - 17:36 »
Quote
How many entries does XMPlay say the library has when you load it there? I'm seeing 147348 here.

Correct, same here.

Quote
You would like all files from a monitored folder to be removed from the library when monitoring of that folder is stopped? I think that option can be added.

That would be outstanding! Yes, exactly what I was asking for. Also, with the rescan, possibly an improvement in that all files from the folder are removed from the library first before the rescan occurs, since that didn't look to be happening when I did it. (Unless the latter is covered by your potential change.)

Quote
What do you see if you right-click on an empty line and select "Track info"?

See attached. XMPlay basically crashes. I don't get any context menu popup at all.

From event viewer:
Faulting application name: xmplay.exe, version: 3.8.2.27, time stamp: 0x59396e19
Faulting module name: xmplay.exe, version: 3.8.2.27, time stamp: 0x59396e19
Exception code: 0xc0000005
Fault offset: 0x00039755
Faulting process id: 0x44dc
Faulting application start time: 0x01d2e5294c2c2ca0
Faulting application path: J:\Utils\xmplay.exe
Faulting module path: J:\Utils\xmplay.exe
Report Id: b35f8c8c-511d-11e7-bcbc-408d5cbfdd98

Attaching ini as requested.

DRS


Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Playlist shows empty rows, library borked?
« Reply #16 on: 15 Jun '17 - 17:33 »
Oh yes, I see the problem now. There was an overflow in the scroller position calculation. Here's an update to fix that:

   www.un4seen.com/stuff/xmplay.exe

This update also adds the option of whether to keep the folder's files in the library when stopping monitoring a folder. It will also remove the files upon launch when a monitored folder has been removed from the filesystem since last time (the files are already removed if the folder is removed while XMPlay is monitoring it).

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #17 on: 15 Jun '17 - 18:05 »
You are the MAN. Thanks Ian. I'll download the exe and check it out when I get back from the office today.

Cheers
DRS

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #18 on: 15 Jun '17 - 19:03 »
Hi Ian. I tried it out here in the office, I have my tracks on a virtual drive.

1) Enabled drive, set up some folders to monitor. Library populated as expected.
2) Unmounted drive. Library entries persisted. Expected result: Library unpopulates in realtime due to drive with monitored folders disappearing.
3) Quit and restart XMPlay with drive still unmounted. Library is now empty, as expected.
4) Remount virtual drive so original monitored folders are accessible again, while XMPlay is running. Library does not autopopulate. Expected result: Library should autopopulate since the original monitored folders are once again available.
5) Exit and reenter XMPlay. Library still does not autopopulate despite original folders once again being present.

Edits:
6) Added subfolders with additional tracks under a monitored folder. Library contents did not update. Expected result: Since the subfolders are under a monitored folder their contents should have been picked up and added to the library automatically.

Also noticed that when the library is zeroed out and then repopulates a subsequent time, the speed is exactly the same as the first time. I suppose this is normal, but if 60k files go away and then come back again it will take another 5-6 minutes to reload the library. I wonder if there's any way to populate the library at ultra high speed so at least the files are in there quickly to be doubleclicked on, with the "evaluation" thread that reads times, titles, info etc run afterward? Or some kind of permanent file cache, separate to the library, that can be called upon for files that were already evaluated once before so they don't have to be reread again. Here's a thought: if XMPlay built a .library file in the root folder of every folder added for monitoring, those files would be there permanently and used for instant library population at startup, and then automatically lazy-refreshed later, in the event that they no longer match with the contents of the folder or its subs. Would massively cut down the lib population time for subsequent startups.

DRS
« Last Edit: 15 Jun '17 - 21:24 by Dhry »

Ian @ un4seen

  • Administrator
  • Posts: 20437
Re: Playlist shows empty rows, library borked?
« Reply #19 on: 16 Jun '17 - 18:06 »
1) Enabled drive, set up some folders to monitor. Library populated as expected.
2) Unmounted drive. Library entries persisted. Expected result: Library unpopulates in realtime due to drive with monitored folders disappearing.

I just tried this with a USB stick, and it looks like Windows doesn't give notification of a folder disappearing when the host drive is unmounted/ejected. Windows did give a warning message that the drive was in use though (presumably XMPlay monitoring it), so some dodginess may be expected when doing that.

3) Quit and restart XMPlay with drive still unmounted. Library is now empty, as expected.
4) Remount virtual drive so original monitored folders are accessible again, while XMPlay is running. Library does not autopopulate. Expected result: Library should autopopulate since the original monitored folders are once again available.
5) Exit and reenter XMPlay. Library still does not autopopulate despite original folders once again being present.

XMPlay can't put a monitor on a folder that doesn't exist, so the contents won't reappear in the library if the drive is remounted while XMPlay is running (without using "Rescan"), but they should reappear if XMPlay is launched afterwards. The problem is that XMPlay isn't resetting its "last check" timestamp when the folder disappears from the filesystem, so it doesn't re-add the contents when the folder reappears with an older timestamp. Here's an update that should fix that:

   www.un4seen.com/stuff/xmplay.exe

6) Added subfolders with additional tracks under a monitored folder. Library contents did not update. Expected result: Since the subfolders are under a monitored folder their contents should have been picked up and added to the library automatically.

This problem appears to be due to the folder and its contents keeping their old timestamps when moved into a monitored folder, so XMPlay thinks it has already been checked. I'm not sure what's the best way around that. I suppose XMPlay could keep a list of sub-folders that it finds, but it would be good if that isn't necessary. I'll look into it.

Also noticed that when the library is zeroed out and then repopulates a subsequent time, the speed is exactly the same as the first time. I suppose this is normal, but if 60k files go away and then come back again it will take another 5-6 minutes to reload the library. I wonder if there's any way to populate the library at ultra high speed so at least the files are in there quickly to be doubleclicked on, with the "evaluation" thread that reads times, titles, info etc run afterward? Or some kind of permanent file cache, separate to the library, that can be called upon for files that were already evaluated once before so they don't have to be reread again. Here's a thought: if XMPlay built a .library file in the root folder of every folder added for monitoring, those files would be there permanently and used for instant library population at startup, and then automatically lazy-refreshed later, in the event that they no longer match with the contents of the folder or its subs. Would massively cut down the lib population time for subsequent startups.

Tags/lengths/etc are read after the files are added to the library, but if there are archives (particularly nested archives) in the monitored folder, then it could take a while to check those for playable files.

Dhry

  • Posts: 64
Re: Playlist shows empty rows, library borked?
« Reply #20 on: 2 Jul '17 - 17:02 »
HI Ian. Sorry for the delayed response.

Quote
it looks like Windows doesn't give notification of a folder disappearing when the host drive is unmounted/ejected.

There's definitely a way to sense this event in Windows, but I'm not a programmer so I don't know whether it's available in the SDK you're using for example. Take a look at the (freeware) program Everything by Voidtools. It monitors the contents of all configured, available NTFS drives in your system and in Options -> Indexes -> NTFS there is an option to automatically remove offline volumes, which happens instantaneously, in addition to the instant (re)scanning of newly-added volumes. I think that this is an NTFS-only feature though, and of course it would only be for file information for NTFS, ie filesize, date etc but not module metadata.

Quote
This problem appears to be due to the folder and its contents keeping their old timestamps when moved into a monitored folder, so XMPlay thinks it has already been checked. I'm not sure what's the best way around that. I suppose XMPlay could keep a list of sub-folders that it finds, but it would be good if that isn't necessary. I'll look into it.

A quick and dirty solution might be an optional flag, globally or perhaps on a per-folder basis, to auto-rescan for changes every x seconds rather than opening options, manually navigating to a folder and selecting to rescan. If the user has an "incoming files" folder that is being monitored where music is added and deleted regularly this could be quite useful. (I sync with ftp.modland.com/pub/modules on a weekly basis.)

Quote
Tags/lengths/etc are read after the files are added to the library

So this is weird then, I don't know if I have something set wrong but my library has a huge amount of files - for example BRSTM files, played with the xmp-vgmstream plugin, or MDX (in_mdx.dll v1.14.4) - where they are all listed in the library as 3:00, the default playback time, but as soon as I doubleclick on a file to play it, the duration changes to what it actually is. Other files appear to have their duration identified via a secondary sweep that occurs after they are loaded. I wonder if this is a problem with the plugins themselves.

Cheers
DRS