RFE: VFS 'runtime' m3u overlay

[i30817]

Description

Thinking about how to make using multiple cd (or other medium) simpler, i had two thoughts:

1. Current RA is using m3u files for this purpose in several relevant cores, but the user has to create them manually.
2. VFS offer the opportunity to create 'ephemeral' files at runtime.

Yes this idea is a bit crazy and depends on VFS capabilities i'm not certain it has. And since only a few cores have m3u support it might be a false positive until every core that needs to use the RA method to change cds has m3u support.

The idea is the same as my script that creates a m3u file for cd formats it encounters on a dir and then applying that idea to a temporary VFS layer to hide the files that get into the new m3u, and show the new m3u.

The algorithm to create the m3u is very simple:

1. find files with applicable extensions in dir. This idea can duplicate m3u for mixed formats like '.cue/iso' vs 'iso'. I'd suggest trying to create m3u with the same name be a non-fatal error and try the 'more complete' fileformats first.
2. Sort output of 1. by filename (natural order so 1 isn't followed by 11) output named 'remaining list'
3. Iterate over 'remaining list', take 1 file from the list and add to a 'm3u list'.
4. Iterate over 'remaining list' again
   Peek at the first of the 'remaining list', find largest common substring between first file and peeked file (starting from 0 always).
   If that string ends in 'disc #*' (disc followed by 0 or more numbers) or 'cd #*' (cd followed by 0 or more numbers), we assume that the first thing changed on the string is the cd number - this is the heuristic this depends on, the dumping groups all name game cds '(disc 1)' or similar.
   Add it to the 'm3u list' if true and remove the first 'remaining list' element, break inner loop otherwise (to create a new m3u).
5. do 'something' with the 'm3u list you created before the outer loop repeats. Creating a VFS file could happen here. Suggested name for most dumping groups is just strip out the first filename on the list of '(.)|[.]' groupings and any extension if any, and use that + '.m3u'. Or just the name of the first file + '.m3u' if you don't care about it looking good.

I've actually tested this with a large (10% of USA total) redump psx collection and all m3u were created correctly.

This process can be tweaked so nothing happens if the m3u list size == 1 of course. In this case, for a VFS, and assuming that every RA core that wants multiple cds uses m3u and those that don't don't, this 'm3u' is useless (and even harmful if the VFS hides other files that exist on m3u) for cores that don't need m3u.

For VFS suggestions i only have speculations because i don't know what RA makes possible here. My main suggestion is:

1. have a setting that named 'use m3u VFS' that applies to the scanner, playlists and the filebrowser.
2. this setting applies a overlay filesystem over the native one that creates a m3u on dir listings and hides files in the m3u. Or (simpler alternative) creates the m3u and hides every other file extensions (if all cores that use those extensions can also use m3u). In the later case, you should also allow m3u with size 1 to appear.

The m3u wouldn't be persistent, but more like a 'runtime file'. That's the idea at least.

I have alternative ideas to VFS that would essentially do the same thing but would require intrusive modifications for the filechooser and scanner.

Expected behavior

It should be easy for the user to select a game as a single unit without using pbp, or creating new files. This would require some heuristics that depend on dumping groups naming habits, but since RA already requires 'correct' crcs and it would be a purely runtime optional feature, i don't think anyone would mind.

Actual behavior

This actually requires the user knowing about m3u and when to create them.

Steps to reproduce the bug

N/A

Bisect Results

N/A

Version/Commit

N/A

Environment information

N/A

[i30817]

I would also be interesting to 'extend' m3u support for floppies and platforms like x68000 that require two floppies at once. To be clear, i'm not suggesting these 'extended' m3u be automatically supported with the idea on the first comment.

The idea is that instead of '.cmd' files for the cores that need to load two files you could have a m3u that uses a path separator character.

so for example final fight.m3u first (and only line) would be:

ff1.fdi:ff2.fdi

This isn't very interesting until you find games that need to change disks sets, imagine one that always needs one boot file, one file for the intro, another for the game.

boot.fdi:intro.fdi
boot.fdi:game.fdi

These 'groups' would necessarily need to be manually figured out though, so this isn't a silver bullet. And pathseperator is different on many platforms unfortunately, so files like these won't be platform portable (but were m3u files already platform portable even if you used purely relative paths?)

Don't get me wrong, i'd be extremely happy if relative paths could be made platform portable. There is even a open issue for that here (make the scanner always create relative paths on playlists if it starts scanning from XDG_CONFIG_HOME).

BTW with this idea implemented relative paths on m3u wouldn't be a problem for 'platform portability'
except if the 'path separator' extension is adopted (and it doesn't always use ';' or recognize both ':' and ';' - if unquoted - or something like that) because the relative path created is always in the same dir (ie: it's assuming games cds are all in the same dir). The only problem is playlists directory separator if the 'always relative path inside RA home dir idea is accepted.

[Ferk]

I'm not from the dev team, so first take this with a grain of salt, but I'm not sure if I understand the request. It looks to me that this involves at the same time two different independent features.

1. Have the frontend generate m3u files by itself. I think what you suggest is doing it automatically based on filename patterns.

It you want to use filename patterns, would m3u files be needed at all? you could skip the m3u generation step completely and just use the filenames to figure out the multidisk images without needing any m3u. Like it's already being done in a few places with things like IPS patches or some core-specific things (like prboom matching deh files with wads). The frontend VFS (or even just the core) could look, for example, for ".iso.001" ".iso.002" extensions or so, and switch those files.

I agree that having something like this automated is a nice thing but on the other hand I think the reason for using m3u is giving the user flexibility so everyone can make its own scripts to create their m3u if so desires without enforcing a particular naming pattern.

2. Expose to the core, through VFS, only the file (or files, after your last comment) for a particular line of the m3u at a time. And have it switch from one to the other in some unspecified way (is it triggered by the user/frontend or by the core?)

I'm not sure if it would be wise to exploit the m3u format in unusual ways to add multiple files per line. Perhaps a different format would make more sense for something like this.

This would also probably require additional features in the frontend-core interface, beyond just VFS. I don't think the frontend knows whether the core supports this type of switching, and having the VFS add/remove files in such a way would be messy for cores that do not expect that. Probably a video/audio player core (#5070), for example, might expect the files in a m3u playlist to be available simultaneously (eg. precaching next song/movie, transition effects between songs, etc) and might want to handle a m3u file on its own, controlling when to switch to the next item, while a core that emulates a disk-based dosbox game might by design have to rely on some trigger from the user to switch disks, since the game does not tell the core when is it appropriate to do it.

The VFS only answers requests for files. It's likely that in many cases (like cores that need to load a new CD/DVD/FBI image by themselves) making a new file appear wouldn't have that much of an effect in a core unless the core is actively notified about the new file being now available. I don't think VFS supports this and at that point you might as well just don't hide anything and just notify to the core when to read the next already existing file. Then you don't need all the complexity and potential problems of making files (dis)appear.

Perhaps it would be wise dividing this into 2 different issues, for clarity.

[i30817]

You could, right, but there is already code in RA doing what's needed for m3u. I don't actually know if the scanner can work with m3u yet, but the playlist can certainly load them iirc. To be clear, i don't mind if the 'VFS m3u ephemeral file' part is skipped as long as the same effect happens (RA inserting the games as a collection in playlists instead of multiple cds and the disc change GUI being already prepared). But i thought this was much more elegant than creating new special cases in the scanner, playlists, browser, etc and could synergize with a file format that the user can already use manually (and manual use would override the automatic use per-case).

BTW, your 'simplification' of using 'new extension' would prevent the biggest gains of my proposed scheme:

1. doesn't make the user change filenames from the the dumping standard
2. still permits different games on the same dir - not all of them because there are some dumping projects that do dumb stuff like naming the tracks standard names (1.bin etc). Even this may work if m3u can load chd, zip or similar formats that 'hide' conflicting filenames.

Not involving work for the user is fundamental (otherwise they may as well create the file themselves).

The switch is the normal one that already occurs in RA if you have a m3u, which i assume is a collaboration between core and frontend.

For the second part, you know best about what each core can do, though i must say that i think the whole point of VFS is to do these kinds of manipulations before the core can see them. For instance, a softpatch VFS would present two files (source and patch) as a single patched file. So files 'suddenly' appearing to the core should be impossible.

Since you're worried about music etc players getting confused by this don't be, you can simply filter the 'kind' of files that trigger the creation of the m3u to accepted disc image formats (I think the precedence i lay out in the first post is necessary to support .cue+iso and plain iso though). Or you can also change the format extension, i'm not attached to m3u (it's just what mednafen core already uses). Hell if you extended the format, you could add a 'type-of-core' marker when you create the file and make the format truly core independent. (this is actually too much trouble for no gain, see 3 posts down).

So i don't think a VFS presenting a file that never existed is too out there. Hiding the other files may be too much (in order to prevent the scanner from adding 'single cds' to the playlist instead of the virtual m3u) because the core might need access to the 'main files' to correctly use the m3u, but i guess that's actually optional and can be resolved another way.

[i30817]

I'd be very sad if a feature for automatic disc change list happened... that required the user to rename all of their isos. RA already has faaar to much of these 'user intervention' features, like the scummvm core marker files, or the x68k core .cmd files (to load two floppies for game boot) etc.

I don't have a solution for the x68k issue actually, i just suggested that m3u 'could' be extended to solve the issue (manually) in order to be consistent with the fileformat of rom disc/floppy change.

To solve this for x68k i think the 'correct' nPr (r=2) permutations (maybe in the form of a bitset?) would have to be hardcoded in libretro-database (this is also a alternative route to solve this issue for single list of discs btw, if slightly less flexible since the user would required to have the right roms and the right names and the scanner would need to check for groups of files instead of single files).
But a database solution would be far too disruptive to the current code in RA i reckon. RDB is not a good format for this apparently and even the source .DATs that those files are created are simplistic about roms and don't specify the order (it's implicit though).

[i30817]

Also btw, there is another advantage to m3u (or any other equivalent 'content file collection' format for RA)... game config files can target 'all the cds' of a game if a m3u is the content file.

Though i'd like to change that slightly too (see my issue #6089).

[i30817]

Sorry for the double posts, but speaking of this part:

The switch is the normal one that already occurs in RA if you have a m3u, which i assume is a collaboration between core and frontend.

You're actually right there is a dangerous underspecification here: a 'ephemeral' m3u doesn't know what core it's for 'by itself'. To make this clear, the idea here is that the VFS is applied before a Scan and before a core content file from the playlists is loaded (always active if the user requests it really).

If you're worried about the user disabling the VFS layer during execution of a core (the only way it's possible to 'confuse' a core i can see), you can simply gray out the option when a core is loaded.

The scanner would add the m3u to one the playlists (because the files inside of it would be redirected to one of the systems from the md5sums/serials etc). Then when the user uses that m3u to load the game, the RA GUI first checks that the m3u file exists. If it does exist (either because of the VFS or because the user actually created one), it launches it for the configured core.

Pretty simple.

Now, i'd prefer if the Scanner part didn't add singular cds to playlists if a m3u with those files is going to be added, so i'd request a special case to filter that.

[Ferk]

BTW, your 'simplification' of using 'new extension' would prevent the biggest gains of my proposed scheme:

1. doesn't make the user change filenames from the the dumping standard

If you are refering to the ".001", ".002", it was just an example that intended to be generic and simple.

You could as well instead use some de facto standard if there's such a thing. Either way, users would have to be forced to use whatever pattern you want them to follow. I doubt every image dump out there follows the same naming standard, so renames would be inevitable either way.

It might not be ideal to introduce a form of automatic list generation if it requires the user to be forced to use a particular naming schema, but wasn't that what you were proposing?

Maybe I misunderstood, were you suggesting using the database to generate the list? From your script it looked to me that you used naming patterns.

2. still permits different games on the same dir

You are still referring to automatic m3u generation, right?
If that would permit different games in the same dir, why wouldn't it also permit it when doing the same but on content load, without saving an m3u?

The switch is the normal one that already occurs in RA if you have a m3u, which i assume is a collaboration between core and frontend.

As I understand, the normal switch that already occurs doesn't happen on the FS level. The file doesn't hide/show from your filesystem, it's just that the core starts reading the next file in the m3u. The VFS wouldn't need to do anything special here.

Maybe I'm misunderstanding again, but I just don't see the advantage of having the VFS hide/show files that are just disk images. Specially considering the VFS is not "mounting" them or so, the core still expects to read them as image files and it would still need to be aware that the new image file needs to be reloaded. I believe you need involvement from the core, just changing in the VFS without notifying the core would likely not work.

[i30817]

edit : read the next post.

[i30817]

Ok, i think i see what you mean now after thinking some more:

It should be possible (with less intrusive code) to make the playlist GUI display present the multiple cds as a single cd, by taking the core of the heuristic and applying it to the playlist data before displaying. It would in effect turn a list [isos] into a list [games [isos]] and apply each 'game' to a cds change structure ala m3u without the m3u before starting the game.

for instance:
[Arc the Lad III (USA) (Disc 1).cue, Arc the Lad III (USA) (Disc 2).cue, Bushido Blade (USA).cue]
would turn into:
[ {Arc the Lad III (USA), [Arc the Lad III (USA) (Disc 1).cue, Arc the Lad III (USA) (Disc 2).cue]}, {Bushido Blade (USA), [Bushido Blade (USA).cue]} ]

And starting the entry for "Arc the Lad III (USA)" would populate the disc change structure with the two 'actual game content files'.

This actually works because as i said, the heuristic works even with games on the same directory which is equivalent to the 'list', so it doesn't actually matter if they're all on the same 'list'.

I'd also request to change the core initialization code to make it possible to apply the 'condensed' naming as a per-game configuration hook (that works for all cds) when this mode is enabled (or maybe even always) and that would allow you to get rid of the VFS. Something like this is a major usecase to use m3u currently.

But you'd still need to care about the loading savestates that refer to a different disc from the loaded one problem, but since that's orthogonal to this, no problem.

About the ".001", ".002" naming and 'easier naming' to handle programmaticaly : Users are already 'strongly encouraged' to use redump/TOSEC etc naming because these are the files that the scanner recognizes metadata for. So, encouraging 'standards' that depend on renaming is very strongly approaching useless because one of the first things a experienced RA user does is to renaming the ROM files with a DAT and romcenter (or similar apps).
So, no it doesn't make it 'generic and simple'. It makes it complicated and full of user effort. It's true that the pattern is more variable than 'just cut off extension after ordering' but in order to use the names which the files have by default (ie: the dumper project names), more complicated regexes like that have to be used. They're still not complex mind because all projects of that kind use patterns like "(cd ##)" for metadata.

[i30817]

I think i'll open a new issue to make this newer, cleaner version of the same thing more attractive to 'actual RA coders'.