"New" launcher, bootstrap.xml compatible #277

Open
icex2 wants to merge 19 commits from icex2/launcher-refactoring into development
icex2 commented 2024-01-29 00:13:31 +03:00 (Migrated from github.com)

Fundamentally re-think how launcher operates and
bootstrapping the games is managed and configured.

This brings it significantly closer to how the original
bootstrap is doing the job: launcher now utilizes the
data (structures) provided by the bootstrap.xml configuration
file. This creates compatibility with vanilla data dumps
and original stock images. Note that bemanitools does not
include any code or means to run DRM'd data, only decrypted.

But, this allows users to keep decrypted dumps as stock as
possible which means:

  • No copying around of property files anymore
  • Keep the modules/ folder with the binaries
  • Have bemanitools binaries separate in the data
  • No need to edit/customize the original configuration files

A list of key features of the "new" launcher:

  • Boostrap games by following the configuration provided by
    stock game's bootstrap.xml files
  • Custom launcher.xml configuration file that adds further
    launcher configurable features, composability of
    bootstrap.xml configuration(s) as well as configuration
    overriding/stacking of selected types of configurations,
    e.g. eamuse config, avs-config. The latter eliminates
    the need for modifying stock config files in the prop/
    folder
  • Unified logging system: launcher and AVS logging uses
    the same logger, all output can now be in a single file
  • Original features such as various hook types still
    available

Move everything to new launcher.xml configuration
files. Adjust the bootstrapping of launcher in the
.bat files. Features such as copying the default
props/ files to nvram are now handled by launcher.

Using the PATH variable, bemanitools binaries can
live in their own dedicated bemanitools/ subfolder
next to props/ and modules/ now. All original
binaries are expected to be kept in a modules/
folder like on stock data.

Due to the significant architectural changes, this also
breaks with any backwards compatibility to existing
launcher setups. Thus, users need to migrate by re-applying
the new configuration format and migrating their config
parameters accordingly.

Further migration instructions and updated documentation
will be provided upon release.

Fundamentally re-think how launcher operates and bootstrapping the games is managed and configured. This brings it significantly closer to how the original bootstrap is doing the job: launcher now utilizes the data (structures) provided by the bootstrap.xml configuration file. This creates compatibility with vanilla data dumps and original stock images. Note that bemanitools does not include any code or means to run DRM'd data, only decrypted. But, this allows users to keep decrypted dumps as stock as possible which means: * No copying around of property files anymore * Keep the modules/ folder with the binaries * Have bemanitools binaries separate in the data * No need to edit/customize the original configuration files A list of key features of the "new" launcher: * Boostrap games by following the configuration provided by stock game's bootstrap.xml files * Custom launcher.xml configuration file that adds further launcher configurable features, composability of bootstrap.xml configuration(s) as well as configuration overriding/stacking of selected types of configurations, e.g. eamuse config, avs-config. The latter eliminates the need for modifying stock config files in the prop/ folder * Unified logging system: launcher and AVS logging uses the same logger, all output can now be in a single file * Original features such as various hook types still available Move everything to new launcher.xml configuration files. Adjust the bootstrapping of launcher in the .bat files. Features such as copying the default props/ files to nvram are now handled by launcher. Using the PATH variable, bemanitools binaries can live in their own dedicated bemanitools/ subfolder next to props/ and modules/ now. All original binaries are expected to be kept in a modules/ folder like on stock data. Due to the significant architectural changes, this also breaks with any backwards compatibility to existing launcher setups. Thus, users need to migrate by re-applying the new configuration format and migrating their config parameters accordingly. Further migration instructions and updated documentation will be provided upon release.
willxinc (Migrated from github.com) reviewed 2024-01-29 00:13:31 +03:00
icex2 commented 2024-01-30 00:42:38 +03:00 (Migrated from github.com)

Remark: This is currently not rebased to https://github.com/djhackersdev/bemanitools/pull/278 which is pre-work required for the major refactoring. Just the last two commits account for the actual refactoring work relevant in this PR. I will rebase accordingly to clean-up the history once https://github.com/djhackersdev/bemanitools/pull/278 is merged.

Remark: This is currently not rebased to https://github.com/djhackersdev/bemanitools/pull/278 which is pre-work required for the major refactoring. Just the last two commits account for the actual refactoring work relevant in this PR. I will rebase accordingly to clean-up the history once https://github.com/djhackersdev/bemanitools/pull/278 is merged.
icex2 commented 2024-01-30 00:46:03 +03:00 (Migrated from github.com)

These are still open TODOs and feedback that are not addressed by this PR but will be followed-up to once this is merged.

  • Complete addressing concerns of https://github.com/djhackersdev/bemanitools/issues/57
  • add development build container
  • rework logging system, issues with non-async logging and lagging the game (stdout and file must be written async),
    implement features such as log ration etc. as outlined by the bootstrap logging config
  • have some sort of migration guide to switch to the new launcher. part of that should be proper
    documentation about how to use the launcher the configuration xml file and what you can do
    with it with various use-case examples
  • align layout of distribution files, different for games such as iidx, jubeat and sdvx
  • config format with stacking and overriding doesn't work here:
    <fs>
        <nr_filesys __type="u16">16</nr_filesys>
        <nr_mountpoint __type="u16">1024</nr_mountpoint>
        <nr_mounttable __type="u16">32</nr_mounttable>
        <nr_filedesc __type="u16">1024</nr_filedesc>
        <link_limit __type="u16">8</link_limit>
        <root>
            <device __type="str">.</device>
        </root>
        <mounttable_selector __type="str">boot</mounttable_selector>
        <mounttable>
            <vfs name="boot" fstype="fs" src="D:/LDJ/contents/dev/raw" dst="/dev/raw" opt="vf=1,posix=1"/>
            <vfs name="boot" fstype="fs" src="D:/LDJ/contents/dev/nvram" dst="/dev/nvram2" opt="vf=0,posix=1"/>
            <vfs name="boot" fstype="nvram2" src="/dev/nvram2" dst="/dev/nvram" opt="nr_mirror=4"/>
        </mounttable>
    </fs>

vfs nodes are being replaced as they are not unique by node name -> add tag __replace="true" or something similar to

These are still open TODOs and feedback that are not addressed by this PR but will be followed-up to once this is merged. * Complete addressing concerns of https://github.com/djhackersdev/bemanitools/issues/57 * add development build container * rework logging system, issues with non-async logging and lagging the game (stdout and file must be written async), implement features such as log ration etc. as outlined by the bootstrap logging config * Address also current limitations of logging API (potentially breaking API compatibility), see also https://github.com/djhackersdev/bemanitools/issues/13 * have some sort of migration guide to switch to the new launcher. part of that should be proper documentation about how to use the launcher the configuration xml file and what you can do with it with various use-case examples * align layout of distribution files, different for games such as iidx, jubeat and sdvx * config format with stacking and overriding doesn't work here: ``` <fs> <nr_filesys __type="u16">16</nr_filesys> <nr_mountpoint __type="u16">1024</nr_mountpoint> <nr_mounttable __type="u16">32</nr_mounttable> <nr_filedesc __type="u16">1024</nr_filedesc> <link_limit __type="u16">8</link_limit> <root> <device __type="str">.</device> </root> <mounttable_selector __type="str">boot</mounttable_selector> <mounttable> <vfs name="boot" fstype="fs" src="D:/LDJ/contents/dev/raw" dst="/dev/raw" opt="vf=1,posix=1"/> <vfs name="boot" fstype="fs" src="D:/LDJ/contents/dev/nvram" dst="/dev/nvram2" opt="vf=0,posix=1"/> <vfs name="boot" fstype="nvram2" src="/dev/nvram2" dst="/dev/nvram" opt="nr_mirror=4"/> </mounttable> </fs> ``` vfs nodes are being replaced as they are not unique by node name -> add tag __replace="true" or something similar to <config kind="inline" replace="true">
This pull request has changes conflicting with the target branch.
  • Module.mk
  • README.md
  • dist/ddr/gamestart-17.bat
  • dist/ddr/gamestart-18.bat
  • doc/ddrhook/ddrhook2.md
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin icex2/launcher-refactoring:icex2/launcher-refactoring
git checkout icex2/launcher-refactoring
Sign in to join this conversation.