diff --git a/.eslintrc b/.eslintrc
index 6862cb8c5..fcb1cad3f 100644
--- a/.eslintrc
+++ b/.eslintrc
@@ -44,7 +44,7 @@
"contexts": [
"ClassDeclaration",
// TODO(cemmer): require private methods as well
- "MethodDefinition[accessibility!=private][key.name!=/^(get|set)[A-Z][a-zA-Z]+/]"
+ "MethodDefinition[accessibility!=private][key.name!=/^(get|set|with)[A-Z][a-zA-Z]+/]"
]
}],
"jsdoc/require-param": "off",
@@ -111,7 +111,7 @@
// TypeScript doesn't do a good job of reporting indexed values as potentially undefined, such as `[1,2,3][999]`
"unicorn/prefer-at": "error",
// Try to enforce early terminations of loops, rather than statements such as `.find(x=>x)[0]`
- "unicorn/prefer-array-find": "error",
+ "unicorn/prefer-array-find": ["error", {"checkFromLast": false}],
"unicorn/prefer-array-flat": "error",
"unicorn/prefer-array-flat-map": "error",
"unicorn/prefer-includes": "error",
diff --git a/.gitattributes b/.gitattributes
index 09c000ee3..500122eff 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,4 +1,4 @@
# Stop `core.autocrlf true`
-*.lnx binary
-*.nes binary
-*.rom binary
+test/fixtures/roms/** binary
+*.cue text eol=lf
+*.gdi text eol=crlf
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index c84e9b688..8961b68ff 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -2,7 +2,7 @@
Welcome! If you're viewing this, it means that you are likely interested in contributing to the project. That's marvelous!
-The following is a set of guidelines for contributing to `igir`. These guidelines are published in order to clarify expectations and reduce potential conflict of opinions.
+The following is a set of guidelines for contributing to Igir. These guidelines are published in order to clarify expectations and reduce potential conflict of opinions.
## Feature requests & bug reports
@@ -16,7 +16,7 @@ If you are experiencing an issues, please submit a detailed [bug report](https:/
[GitHub discussions](https://github.com/emmercm/igir/discussions) are a great tool for a number of topics:
-- Getting help with `igir` CLI syntax or usage
+- Getting help with Igir CLI syntax or usage
- Clarifying support for specific features or scenarios
- Brainstorming new feature requests
- ...and more!
@@ -27,13 +27,13 @@ Discussions are intended to be low-pressure spaces for questions and collaborati
### Environment setup
-First, you will want to check out `igir`'s source code from GitHub:
+First, you will want to check out Igir's source code from GitHub:
```shell
git clone https://github.com/emmercm/igir.git
```
-`igir` is written in TypeScript for the Node.js runtime. The current version of Node.js that `igir` uses is defined under the `"volta"` object in the `package.json` file. After [installing](https://docs.volta.sh/guide/getting-started), Volta will make sure you're always using the correct Node.js version.
+Igir is written in TypeScript for the Node.js runtime. The current version of Node.js that Igir uses is defined under the `"volta"` object in the `package.json` file. After [installing](https://docs.volta.sh/guide/getting-started), Volta will make sure you're always using the correct Node.js version.
Third-party dependencies are managed and easily installed with npm:
@@ -51,7 +51,7 @@ npm pack
### Running code
-A script has been defined for the `npm start` command to easily run `igir`:
+A script has been defined for the `npm start` command to easily run Igir:
```shell
npm start -- [commands..] [options]
@@ -65,25 +65,25 @@ npm start -- report --dat *.dat --input ROMs/
### Code style
-`igir` uses [ESLint](https://eslint.org/) as its linter and style enforcer. Rules have been specifically chosen to increase code consistency, safety, readability, and maintainability.
+Igir uses [ESLint](https://eslint.org/) as its linter and style enforcer. Rules have been specifically chosen to increase code consistency, safety, readability, and maintainability.
All code changes must pass the existing ESLint rules. Discussions on adding, removing, and changing ESLint rules should happen outside of pull requests that contain code changes, in their own dedicated pull request or discussion thread (above).
### Automated tests
-`igir` uses [Jest](https://jestjs.io/) as its testing framework, and it uses [Codecov](https://about.codecov.io/) to ensure a minimum amount of test coverage.
+Igir uses [Jest](https://jestjs.io/) as its testing framework, and it uses [Codecov](https://about.codecov.io/) to ensure a minimum amount of test coverage.
All code changes must come with appropriate automated tests in order to prove correctness and to protect against future regressions.
### Docs
-`igir` uses [MkDocs](https://www.mkdocs.org/) to turn Markdown files into a documentation website.
+Igir uses [MkDocs](https://www.mkdocs.org/) to turn Markdown files into a documentation website.
Appropriate updates must be made to all relevant documentation pages if functionality is added, removed, or changed.
### Git commit messages
-`igir` is configured to squash-merge all pull requests, such that only the pull request title ends up in the commit history of the main branch. This means that individual commit messages are less important, and it puts more emphasis on quality pull request titles & descriptions.
+Igir is configured to squash-merge all pull requests, such that only the pull request title ends up in the commit history of the main branch. This means that individual commit messages are less important, and it puts more emphasis on quality pull request titles & descriptions.
That said, quality commit messages help future maintainers understand past intentions. Please use your best judgement on descriptive, clear, and concise commit messages.
@@ -91,7 +91,7 @@ That said, quality commit messages help future maintainers understand past inten
Here are steps that should be completed prior to submitting a pull request:
-- [ ] Validate your change works as expected locally by running `igir` (not just the unit tests)
+- [ ] Validate your change works as expected locally by running Igir (not just the unit tests)
- [ ] Unit tests have been added to cover your change
- [ ] `npm test` has been run locally for your change, to validate:
- Your added & changed tests are passing
@@ -113,8 +113,8 @@ To contribute code changes, you will need to:
## License
-`igir` is licensed under [GNU General Public License v3.0](https://github.com/emmercm/igir/blob/main/LICENSE).
+Igir is licensed under [GNU General Public License v3.0](https://github.com/emmercm/igir/blob/main/LICENSE).
-โ That means that `igir` can be used for free commercially, can be modified, can be distributed, and can be used for private use.
+โ That means that Igir can be used for free commercially, can be modified, can be distributed, and can be used for private use.
โ ๏ธ But it also means that distribution of closed-source versions is _not_ allowed.
diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml
index ea69ce21d..f0632eed2 100644
--- a/.github/ISSUE_TEMPLATE/bug-report.yml
+++ b/.github/ISSUE_TEMPLATE/bug-report.yml
@@ -15,7 +15,7 @@ body:
attributes:
label: Paste the command
description: |
- The exact `igir` command you ran when you experienced a bug.
+ The exact Igir command you ran when you experienced a bug.
Include the full `node`, `npm`, or `npx` command when not running the standalone `igir` executable.
validations:
@@ -53,12 +53,12 @@ body:
attributes:
label: DAT(s) used
description: |
- Links to the DATs that were used for this `igir` command, if any.
+ Links to the DATs that were used for this Igir command, if any.
- type: input
attributes:
label: igir version
- description: What version of `igir` are you running? This is visible in the output header.
+ description: What version of Igir are you running? This is visible in the output header.
validations:
required: true
@@ -68,7 +68,7 @@ body:
description: |
What version of Node.js are you running? This can be seen with the `node --version` command.
- You can specify "N/A" when using a standalone version of `igir` (one downloaded from GitHub).
+ You can specify "N/A" when using a standalone version of Igir (one downloaded from GitHub).
validations:
required: true
diff --git a/.github/workflows/codecov.yml b/.github/workflows/codecov.yml
index 0be3cf904..96fee7b97 100644
--- a/.github/workflows/codecov.yml
+++ b/.github/workflows/codecov.yml
@@ -35,6 +35,9 @@ jobs:
- uses: actions/checkout@v4
- uses: volta-cli/action@v4
- run: npm ci
+ - run: |
+ sudo apt-get update
+ sudo apt-get install -y libsdl2-2.0-0 libsdl2-ttf-2.0-0
# Run test coverage
- run: npm run test:coverage
diff --git a/.github/workflows/gh-automerge-rebase.yml b/.github/workflows/gh-automerge-rebase.yml
index 7a64bc6f8..5cb9d1de2 100644
--- a/.github/workflows/gh-automerge-rebase.yml
+++ b/.github/workflows/gh-automerge-rebase.yml
@@ -12,6 +12,8 @@ on:
push:
branches:
- 'main'
+ - '*feature*'
+ - '**/*feature*'
schedule:
# Every hour
- cron: '0 * * * *'
@@ -26,7 +28,7 @@ jobs:
with:
# GitHub won't run workflows off of code commits+pushes from the `github-actions` user
token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
- base: 'main'
+ base: ${{ github.head_ref || github.ref_name || 'main' }}
required_approval_count: 0
require_passed_checks: false
# Oldest pull request
diff --git a/.github/workflows/gh-first-interaction.yml b/.github/workflows/gh-first-interaction.yml
index 9cb03b78f..dda8cf39a 100644
--- a/.github/workflows/gh-first-interaction.yml
+++ b/.github/workflows/gh-first-interaction.yml
@@ -16,7 +16,7 @@ jobs:
pr-message: |
## :wave: Welcome
- Thank you for your first pull request, @${{ github.event.pull_request.head.user.login }}! If you haven't yet, please familiarize yourself with `igir`'s [contribution guidelines](https://github.com/emmercm/igir/blob/main/.github/CONTRIBUTING.md).
+ Thank you for your first pull request, @${{ github.event.pull_request.head.user.login }}! If you haven't yet, please familiarize yourself with Igir's [contribution guidelines](https://github.com/emmercm/igir/blob/main/.github/CONTRIBUTING.md).
Some GitHub Actions CI workflows may not automatically run for you due to GitHub's [security best practices](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#controlling-changes-from-forks-to-workflows-in-public-repositories), so a maintainer may need to manually approve the workflows to run. As a result, it is important to make sure tests pass locally before submitting a pull request to help ensure a fast review. Thank you!
diff --git a/.github/workflows/gh-pages.yml b/.github/workflows/gh-pages.yml
index 6fc68100e..47f4de3ef 100644
--- a/.github/workflows/gh-pages.yml
+++ b/.github/workflows/gh-pages.yml
@@ -42,7 +42,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- - run: docker run --rm --volume "${PWD}:/workdir" ghcr.io/igorshubovych/markdownlint-cli:latest --disable MD013 MD033 MD046 -- "**/*.md"
+ - run: docker run --rm --volume "${PWD}:/workdir" ghcr.io/igorshubovych/markdownlint-cli:latest --disable MD013 MD033 MD041 MD046 -- "**/*.md"
build:
needs:
diff --git a/.github/workflows/node-test.yml b/.github/workflows/node-test.yml
index 41fe28ce4..16573d494 100644
--- a/.github/workflows/node-test.yml
+++ b/.github/workflows/node-test.yml
@@ -69,6 +69,7 @@ jobs:
- path-filter
if: ${{ needs.path-filter.outputs.changes == 'true' }}
runs-on: ${{ matrix.os }}-latest
+ timeout-minutes: 10
strategy:
matrix:
os: [ ubuntu, macos, windows ]
@@ -82,6 +83,13 @@ jobs:
with:
node-version: ${{ matrix.node-version }}
- run: npm ci
+ - if: ${{ matrix.os == 'macos' }}
+ run: brew install --overwrite sdl2
+ - if: ${{ matrix.os == 'ubuntu' }}
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y libsdl2-2.0-0 libsdl2-ttf-2.0-0
+
# Test the source files
- run: npm run test:unit
@@ -90,6 +98,7 @@ jobs:
- path-filter
if: ${{ needs.path-filter.outputs.changes == 'true' }}
runs-on: ubuntu-latest
+ timeout-minutes: 10
strategy:
matrix:
node-version: [ lts, 18, 16.7.0 ]
@@ -102,6 +111,9 @@ jobs:
with:
node-version: ${{ matrix.node-version }}
- run: npm ci
+ - run: |
+ sudo apt-get update
+ sudo apt-get install -y libsdl2-2.0-0 libsdl2-ttf-2.0-0
# Test the built files
- run: npm run build
- run: ./test/endToEndTest.sh
diff --git a/.gitignore b/.gitignore
index 89b5986ef..3d910f86d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -108,7 +108,7 @@ dist
dist/
site/
*.bat
-*.cache
+*.cache*
*.sh
!test/*.sh
@@ -122,16 +122,21 @@ site/
*.bin
*.cd1
*.cd2
+*.chd
*.col
+*.cso
*.cue
+*.dax
*.dvd
*.gb
*.gba
*.gbc
+*.gcz
*.gdi
*.gg
*.ic1
*.img
+*.iso
*.jar
*.lo
*.lyx
@@ -142,17 +147,21 @@ site/
*.pce
*.pk3
*.pup
+*.raw
*.rom
+*.rvz
*.sfc
*.smc
*.sms
*.szx
*.wad
+*.wia
*.x1
*.x1t
*.zim
*.z64
*.zip
+*.zso
# ROM pack excess
*.bmp
diff --git a/README.md b/README.md
index 06487bd79..c7f2c12c9 100644
--- a/README.md
+++ b/README.md
@@ -1,44 +1,47 @@
-
๐น๏ธ igir
+
+
+
+
-
Pronounced "eager," igir is a video game ROM collection manager to help filter, sort, patch, archive, and report on collections on any OS.
+
Pronounced "eager," Igir is a zero-setup ROM collection manager that sorts, filters, extracts or archives, patches, and reports on collections of any size on any OS.
-
+
-
+
See the project website for complete documentation, installation & usage instructions, and examples!
-## What does `igir` do?
+## What does Igir do?
A video of an example use case:
-[](https://asciinema.org/a/Sum1WBdZRsSTvbZvVuP5Ho1N9)
+
-With `igir` you can manage a ROM collection of any size:
+With Igir you can manage a ROM collection of any size:
-- ๐ Scan for DATs, ROMs, and ROM patches - including those in archives (see [scanning](https://igir.io/input/file-scanning) & [archive docs](https://igir.io/input/reading-archives))
+- ๐ Scan for DATs, ROMs, and ROM patchesโincluding those in archives (see [scanning](https://igir.io/input/file-scanning) & [archive docs](https://igir.io/input/reading-archives))
- ๐ Organize ROM files by console (see [DAT docs](https://igir.io/dats/overview))
- ๐ช Name ROM files consistently, including the right extension (see [DAT docs](https://igir.io/dats/overview))
- โ๏ธ Filter out duplicate ROMs, or ROMs in languages you don't understand (see [filtering docs](https://igir.io/roms/filtering-preferences))
- ๐๏ธ Extract or archive ROMs in mass (see [archive docs](https://igir.io/output/writing-archives))
- ๐ฉน Patch ROMs automatically in mass (see [scanning](https://igir.io/input/file-scanning) & [patching docs](https://igir.io/roms/patching))
-- ๐ฉ Parse ROMs with headers, and optionally remove them (see [header docs](https://igir.io/roms/headers))
+- ๐ฉ Parse ROMs with headers and optionally remove them (see [header docs](https://igir.io/roms/headers))
- โ๏ธ Build & re-build (un-merge, split, or merge) MAME ROM sets (see [arcade docs](https://igir.io/usage/arcade))
-- ๐ฎ Report on what ROMs are present or missing for each console, and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/dats/overview))
+- ๐ฎ Report on what ROMs are present or missing for each console and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/dats/overview))
-## How do I run `igir`?
+## How do I run Igir?
Either download the latest version for your OS from the [releases page](https://github.com/emmercm/igir/releases/latest), or if you have Node.js installed you can use [`npx`](https://docs.npmjs.com/cli/v9/commands/npx) to always run the latest version from the command line:
diff --git a/codecov.yml b/codecov.yml
index 3b7f31352..9f77df147 100644
--- a/codecov.yml
+++ b/codecov.yml
@@ -9,5 +9,5 @@ coverage:
patch: off
project:
default:
- target: 94%
+ target: 93%
threshold: 1%
diff --git a/docs/advanced/internals.md b/docs/advanced/internals.md
index ce366f5ad..e7875180f 100644
--- a/docs/advanced/internals.md
+++ b/docs/advanced/internals.md
@@ -1,10 +1,10 @@
# Internal Operations
-Information about the inner workings of `igir`.
+Information about the inner workings of Igir.
## Order of operations
-`igir` runs these steps in the following order:
+Igir runs these steps in the following order:
1. Scan each DAT input path for every file and parse them, if provided (`--dat `)
2. Scan each ROM input path for every file (`--input `)
@@ -24,6 +24,6 @@ Information about the inner workings of `igir`.
- Written ROMs are tested for accuracy, if specified (`test`)
- A "dir2dat" DAT is created, if specified (`dir2dat`) (see [dir2dat docs](../dats/dir2dat.md))
- A "fixdat" is created, if specified (`fixdat`) (see [fixdats docs](../dats/fixdats.md))
-5. "Moved" input ROMs are deleted (`move`)
+5. Leftover "moved" input ROMs are deleted (`move`)
6. Unknown files are recycled from the output directory, if specified (`clean`, see [cleaning docs](../output/cleaning.md))
7. An output report is written to the output directory, if specified (`report`, see [reporting docs](../output/reporting.md))
diff --git a/docs/advanced/logging.md b/docs/advanced/logging.md
index e52800dd5..9eae7272e 100644
--- a/docs/advanced/logging.md
+++ b/docs/advanced/logging.md
@@ -1,6 +1,6 @@
# Logging
-By default, `igir` will print the following log levels:
+By default, Igir will print the following log levels:
- `ERROR`: an unexpected error has prevented an intended [command](../commands.md)
- `WARN`: a preventable error has prevented an intended [command](../commands.md)
@@ -14,7 +14,7 @@ There are additional levels of verbosity that can be enabled with the `-v` flag:
- Files being copied, zipped, and linked
- [dir2dat](../dats/dir2dat.md) files being created
- [Fixdat](../dats/fixdats.md) files being created
- - Input files deleted after being moved
+ - Leftover input files deleted after being moved
- Output files being [cleaned](../output/cleaning.md) (including files skipped due to `--clean-dry-run`)
- [Report](../output/reporting.md) files being created
@@ -40,7 +40,7 @@ There are additional levels of verbosity that can be enabled with the `-v` flag:
igir [commands..] [options] -vv
```
- This level is helpful to turn on if you want debug why an action didn't take place.
+ This level is helpful to turn on if you want to debug why an action didn't take place.
- **`TRACE` (`-vvv`): print information about actions taken, skipped, and additional information that can be helpful to debug issues.**
diff --git a/docs/advanced/temp-dir.md b/docs/advanced/temp-dir.md
index 0d8d89b08..511c99934 100644
--- a/docs/advanced/temp-dir.md
+++ b/docs/advanced/temp-dir.md
@@ -1,13 +1,13 @@
# Temp Directory
-`igir` needs to write some temporary files to disk for a few reasons:
+Igir needs to write some temporary files to disk for a few reasons:
- Downloading [DAT URLs](../dats/processing.md#scanning-for-dats) to disk before parsing
- Extracting [some archives](../input/reading-archives.md) to disk during scanning, and when reading when extracting or [zipping](../output/writing-archives.md)
-Temporary files are ones that are deleted as soon as `igir` no longer needs them for processing. `igir` will also delete any leftover temporary files on exit.
+Temporary files are ones that are deleted as soon as Igir no longer needs them for processing. Igir will also delete any leftover temporary files on exit.
-`igir` will use your operating system's temporary directory for these files by default. The option `--temp-dir ` is provided to let you change the directory, and you may want to do this for a few reasons:
+Igir will use your operating system's temporary directory for these files by default. The option `--temp-dir ` is provided to let you change the directory, and you may want to do this for a few reasons:
- Your operating system drive has minimal space available
- You want to protect your operating system drive from excess wear and tear
diff --git a/docs/advanced/troubleshooting.md b/docs/advanced/troubleshooting.md
index 9e0ae0fc5..b4a89667c 100644
--- a/docs/advanced/troubleshooting.md
+++ b/docs/advanced/troubleshooting.md
@@ -34,9 +34,9 @@ FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memor
11: 0x7fe14fed9ef6
```
-The issue is that `igir` ran out of memory likely due to low system limits, large DAT packs, or large ROM collections.
+The issue is that Igir ran out of memory likely due to low system limits, large DAT packs, or large ROM collections.
-You likely need to process your ROM collection in batches, just be careful when using the [`igir clean` command](../commands.md). If you don't need every DAT from a pack, you can try reducing the number of DATs being processed with the [`--dat-*-regex ` and `--dat-*-regex-exclude ` options](../dats/processing.md#dat-filtering) like this:
+You likely need to process your ROM collection in batches, just be careful when using the [`igir clean` command](../commands.md). If you don't need every DAT from a pack, you can try reducing the number of DATs being processed with the [`--dat-*-regex ` and `--dat-*-regex-exclude ` options](../dats/processing.md#dat-filtering) like this:
```shell
igir [commands..] --dat "*.dat" --dat-name-regex "/nintendo/i"
diff --git a/docs/alternatives.md b/docs/alternatives.md
index 55b6e1044..747696fcd 100644
--- a/docs/alternatives.md
+++ b/docs/alternatives.md
@@ -2,44 +2,52 @@
There are a few different popular ROM managers that have similar features:
-| Feature | [igir](index.md) | [clrmamepro](https://mamedev.emulab.it/clrmamepro/) | [RomVault](https://www.romvault.com/) | [RomCenter](http://www.romcenter.com/) |
-|------------------------------------------|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------|-------------------------------------------------------------|--------------------------------------------|
-| Code: still in development | โ | โ | โ | โ |
-| Code: open source | โ GPL | โ | โ | โ |
-| App: OS compatibility | โ anything [Node.js supports](https://nodejs.org/en/download) | โ ๏ธ Windows, macOS & Linux via [Wine](https://www.winehq.org/) | โ ๏ธ Windows, Linux via [Mono](https://www.mono-project.com/) | โ Windows only |
-| App: UI or CLI | CLI only by design | UI only | Separate UI & CLI versions | UI only |
-| App: required setup steps | โ no setup required | โ requires "profile" setup per DAT | โ ๏ธ if specifying DAT & ROM dirs | โ requires per-DAT DB setup |
-| DATs: supported formats | Logiqx XML, MAME ListXML, MAME Software List, CMPro, HTGD SMDB ([DATs docs](dats/processing.md)) | Logiqx XML, MAME ListXML, MAME Software List, CMPro | Logiqx XML, MAME ListXML, CMPro, RomCenter, HTGD SMDB | Logiqx XML, CMPro, RomCenter |
-| DATs: process multiple at once | โ | โ ๏ธ via the batcher | โ | โ |
-| DATs: infer parent/clone info | โ | โ | โ | โ |
-| DATs: built-in download manager | โ | โ | โ ๏ธ via [DatVault](https://www.datvault.com/) | โ |
-| DATs: supports DAT URLs | โ | โ | โ | โ |
-| DATs: create from files (dir2dat) | โ [dir2dat docs](dats/dir2dat.md) | โ | โ | โ |
-| DATs: fixdat creation | โ [Fixdat docs](dats/fixdats.md) | โ | โ | โ |
-| DATs: combine multiple | โ | โ | โ | โ |
-| Archives: extraction formats | โ many formats ([reading archives docs](input/reading-archives.md)) | โ `.zip`, `.7z`, `.rar` | โ ๏ธ `.zip`, `.7z` | โ ๏ธ `.zip`, `.7z` |
-| Archives: creation formats | โ `.zip` only by design ([writing archives docs](output/writing-archives.md)) | โ `.zip`, `.7z`, `.rar` | โ ๏ธ `.zip` (TorrentZip), `.7z` | โ ๏ธ `.zip`, `.7z` |
-| Archives: automatic extension correction | โ | โ | โ | โ |
-| ROMs: checksum matching strategies | โ CRC32+size, MD5, SHA1, SHA256 | โ ๏ธ CRC32+size, MD5, SHA1 | โ ๏ธ CRC32+size, MD5, SHA1 | โ |
-| ROMs: CHD scanning | โ | โ ๏ธ via chdman | โ v1-5 natively | โ ๏ธ v1-4 natively |
-| ROMs: scan/checksum caching | โ | โ | โ | โ |
-| ROMs: header parsing | โ | โ | โ | โ ๏ธ via plugins |
-| ROMs: header removal | โ [automatic and forced](roms/headers.md) | โ | โ | โ |
-| ROMs: automatic extension correction | โ [output writing docs](output/options.md#fixing-rom-extensions) | โ | โ | โ |
-| ROMs: supported merge types | โ full non-merged, non-merged, split, merged | โ full non-merged, non-merged, split, merged | โ ๏ธ full non-merged, split, merged | โ ๏ธ full non-merged, split, merged |
-| ROMs: patching support | โ [patching docs](roms/patching.md) | โ | โ ๏ธ SNES SuperDAT | โ |
-| Filtering: region, language, type, etc. | โ [many options](roms/filtering-preferences.md#filters) | โ only 1G1R options | โ | โ ๏ธ only at DB setup |
-| Filtering: 1G1R support | โ [many options](roms/filtering-preferences.md#preferences-for-1g1r) | โ ๏ธ region & language only | โ | โ ๏ธ only at DB setup |
-| Reports: report-only mode | โ | โ | โ | โ |
-| Reports: easily parseable | โ CSV | โ ๏ธ newline-separated "have" & "miss" lists | โ ๏ธ newline-separated "full" & "fix" reports | โ ๏ธ newline-separated "have" & "miss" lists |
-| Output: file link support | โ hard & symbolic links | โ | โ | โ |
-| Output: separate input & output dirs | โ | โ | โ ๏ธ yes but files are always moved | โ |
-| Output: subdirectory customization | โ [many options](output/path-options.md) | โ | โ ๏ธ depends on DAT organization | โ |
-| Output: create single archive for DAT | โ | โ | โ | โ |
+| Feature | [igir](index.md) | [RomVault](https://www.romvault.com/) | [clrmamepro](https://mamedev.emulab.it/clrmamepro/) | [RomCenter](http://www.romcenter.com/) |
+|------------------------------------------|--------------------------------------------------------------------------------------------------|-------------------------------------------------------------|---------------------------------------------------------------|--------------------------------------------|
+| App: still in development | โ | โ | โ | โ |
+| App: OS compatibility | โ anything [Node.js supports](https://nodejs.org/en/download) | โ ๏ธ Windows, Linux via [Mono](https://www.mono-project.com/) | โ ๏ธ Windows, macOS & Linux via [Wine](https://www.winehq.org/) | โ Windows only |
+| App: UI or CLI | CLI only by design | Separate UI & CLI versions | UI only | UI only |
+| App: required setup steps | โ no setup required | โ ๏ธ if specifying DAT & ROM dirs | โ requires "profile" setup per DAT | โ requires per-DAT DB setup |
+| App: open source | โ GPL | โ | โ | โ |
+| DATs: supported formats | Logiqx XML, MAME ListXML, MAME Software List, CMPro, HTGD SMDB ([DATs docs](dats/processing.md)) | Logiqx XML, MAME ListXML, CMPro, RomCenter, HTGD SMDB | Logiqx XML, MAME ListXML, MAME Software List, CMPro | Logiqx XML, CMPro, RomCenter |
+| DATs: process multiple at once | โ | โ | โ ๏ธ via the batcher | โ |
+| DATs: infer parent/clone info | โ | โ | โ | โ |
+| DATs: built-in download manager | โ | โ ๏ธ via [DatVault](https://www.datvault.com/) | โ | โ |
+| DATs: supports DAT URLs | โ | โ | โ | โ |
+| DATs: create from files (dir2dat) | โ [dir2dat docs](dats/dir2dat.md) | โ | โ | โ |
+| DATs: fixdat creation | โ [Fixdat docs](dats/fixdats.md) | โ | โ | โ |
+| DATs: combine multiple | โ | โ | โ | โ |
+| ROM Scanning: parallel scanning | โ | โ | โ | โ |
+| ROM Scanning: scanning exclusions | โ | โ | โ | โ |
+| ROM Scanning: quick scanning | โ [matching docs](roms/matching.md) | โ | โ ๏ธ by default | โ |
+| ROM Scanning: scan/checksum caching | โ | โ | โ | โ |
+| ROMs: checksum matching strategies | โ CRC32+size, MD5, SHA1, SHA256 | โ ๏ธ CRC32+size, MD5, SHA1 | โ ๏ธ CRC32+size, MD5, SHA1 | โ |
+| ROMs: header detection | โ | โ | โ ๏ธ via supplemental XMLs | โ ๏ธ via plugins |
+| ROMs: header removal | โ [automatic and forced](roms/headers.md) | โ | โ | โ |
+| ROMs: automatic extension correction | โ [output writing docs](output/options.md#fixing-rom-extensions) | โ | โ | โ |
+| ROMs: patching support | โ [patching docs](roms/patching.md) | โ ๏ธ SNES SuperDAT | โ | โ |
+| Arcade: supported merge types | โ full non-merged, non-merged, split, merged ([arcade docs](usage/arcade.md)) | โ ๏ธ full non-merged, split, merged | โ full non-merged, non-merged, split, merged | โ ๏ธ full non-merged, split, merged |
+| Arcade: CHD disk inclusion | โ by default, can be turned off ([arcade docs](usage/arcade.md)) | โ by default, can be turned off | โ | โ |
+| Arcade: sample inclusion | โ | โ | โ | โ |
+| Archives: extraction formats | โ many formats ([reading archives docs](input/reading-archives.md)) | โ ๏ธ `.zip`, `.7z` (natively) | โ `.zip`, `.7z` (via `7z`), `.rar` (via `rar`) | โ ๏ธ `.zip`, `.7z` |
+| Archives: `.chd` support | โ ๏ธ via `chdman` (bundled) | โ v1-5 natively | โ ๏ธ via `chdman` | โ ๏ธ v1-4 natively |
+| Archives: `.cso` & `.zso` support | โ ๏ธ via `maxcso` (bundled) | โ | โ | โ |
+| Archives: `.nkit.iso` support | โ ๏ธ matching but no extraction [GameCube docs](usage/console/gamecube.md#nkit) | โ | โ | โ |
+| Archives: creation formats | โ `.zip` only by design ([writing archives docs](output/writing-archives.md)) | โ ๏ธ `.zip` (TorrentZip), `.7z` (RV7Z) | โ `.zip`, `.7z`, `.rar` | โ ๏ธ `.zip`, `.7z` |
+| Archives: contents checksums | โ when needed ([reading archives docs](input/reading-archives.md)) | โ ๏ธ requires "files only" mode | โ ๏ธ if DAT has forcepacking=unzip | โ |
+| Archives: automatic extension correction | โ | โ | โ | โ |
+| Filtering: region, language, type, etc. | โ [many options](roms/filtering-preferences.md#filters) | โ | โ only 1G1R options | โ ๏ธ only at DB setup |
+| Filtering: 1G1R support | โ [many options](roms/filtering-preferences.md#preferences-for-1g1r) | โ | โ ๏ธ region & language only | โ ๏ธ only at DB setup |
+| Reports: report-only mode | โ | โ | โ | โ |
+| Reports: easily parseable | โ CSV | โ ๏ธ newline-separated "full" & "fix" reports | โ ๏ธ newline-separated "have" & "miss" lists | โ ๏ธ newline-separated "have" & "miss" lists |
+| Output: file link support | โ hard & symbolic links | โ | โ | โ |
+| Output: separate input & output dirs | โ | โ ๏ธ yes but files are always moved | โ | โ |
+| Output: subdirectory customization | โ [many options](output/path-options.md) | โ ๏ธ depends on DAT organization | โ | โ |
+| Output: create single archive for DAT | โ | โ | โ | โ |
!!! note
- Just like `igir`, other ROM managers that are in active development are likely to release new features often. The above table is not guaranteed to be perfectly up-to-date, it is just a best effort.
+ Just like Igir, other ROM managers that are in active development are likely to release new features often. The above table is not guaranteed to be perfectly up-to-date, it is just a best effort.
Other alternative ROM managers can be found in a number of other wikis, such as:
diff --git a/docs/cli.md b/docs/cli.md
new file mode 100644
index 000000000..8ea595c26
--- /dev/null
+++ b/docs/cli.md
@@ -0,0 +1,40 @@
+# CLI Overview
+
+Igir uses a series of live-updating progress bars to indicate what it is currently working on and how much processing is left to do.
+
+
+
+See the [internal operations](advanced/internals.md#order-of-operations) page for more information on every processing that Igir might do.
+
+## Progress bar icons
+
+ASCII symbols are used to indicate what processing is happening. Here is a table of those symbols, in order:
+
+| Symbol (magenta) | Scanning operation |
+|------------------------------------------------------------|-------------------------------------------------------------------------------------------|
+| โป (circle arrow) | Files (DATs, ROMs, patches, etc.) are being found/enumerated |
+| โ (down arrow) | [DATs](dats/introduction.md) are being [downloaded](dats/processing.md#scanning-for-dats) |
+| ฮฃ (sigma) | [DATs](dats/introduction.md) are being parsed |
+| # (hash) | ROMs are having checksums calculated for [matching](roms/matching.md) |
+| ^ (hat) | ROMs are being checked for [headers](roms/headers.md) |
+
+| Symbol (cyan) | Per-DAT processing operation |
+|--------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
+| โฉ (intersection) | DATs are having parent/clone information [inferred](dats/processing.md#parentclone-inference) |
+| โ (split arrow) | DATs are having [merge/split rules](usage/arcade.md#rom-set-merge-types) applied |
+| ฮฃ (sigma) | ROMs are being [matched](roms/matching.md) to the DAT |
+| โ (delta) | DAT is being [filtered](roms/filtering-preferences.md#filters), ROM [1G1R rules](roms/filtering-preferences.md#preferences-for-1g1r) are being applied |
+| . (period) | ROM matches hare having their [extension corrected](output/options.md#fixing-rom-extensions) |
+| โ (question equal) | ROM matches are being checked for issues |
+| โช (union) | ROM matches are being combined into one zip |
+
+| Symbol (yellow) | Per-DAT writing operation |
+|--------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|
+| # (hash) | Archives are having checksums calculated to [test](commands.md#test) after [writing](commands.md#rom-writing) |
+| โ (question equal) | Output files are being checked before being [overwritten](output/options.md#overwriting-files), no writing has started yet |
+| โ (pencil) | Output files are or have been written |
+
+| Symbol | Deleting operation |
+|-------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
+| โป (recycle) | Output directory [cleaned files](output/cleaning.md) are being recycled |
+| โ (x) | Moved ROM matches are being deleted, output directory [cleaned files](output/cleaning.md) are being deleted |
diff --git a/docs/commands.md b/docs/commands.md
index bfb947ecf..64abf9be2 100644
--- a/docs/commands.md
+++ b/docs/commands.md
@@ -1,6 +1,6 @@
# Commands
-`igir` takes actions based on commands you specify. Each command has a clear input and output, and `igir` will never take surprise actions you did not specify. Multiple commands can (and will likely) be specified at once.
+Igir takes actions based on commands you specify. Each command has a clear input and output, and Igir will never take surprise actions you did not specify. Multiple commands can (and will likely) be specified at once.
!!! tip
@@ -8,31 +8,31 @@
## ROM writing
-`igir` has three writing commands. Only one writing command can be specified at a time, and all require the `--output ` option.
+Igir has three writing commands. Only one writing command can be specified at a time, and all require the `--output ` option.
### `copy`
-Copy ROMs from an input directory to the output directory.
+Copy files from an input directory to the output directory.
Files in the input directories will be left alone, they will _not_ be modified or deleted.
### `move`
-Move ROMs from an input directory to the output directory. The same directory can be specified for both input & output, resulting in ROMs being renamed as their names change in [DATs](dats/introduction.md).
+Move files from an input directory to the output directory. The same directory can be specified for both input & output, resulting in ROMs being renamed as their names change in [DATs](dats/introduction.md).
-ROMs will be deleted from their input directory after _all_ ROMs for _every_ [DAT](dats/introduction.md) have been written.
+Files that match to multiple ROMs in [DATs](dats/introduction.md) will be copied as needed.
### `link`
-Create a link in the output directory to a ROM in the input directory.
+Create a link in the output directory to a file in the input directory.
By default, hard links are created, similar to [ln(1)](https://linux.die.net/man/1/ln). Use the `--symlink` option to create symbolic links.
-## ROM archiving
+## ROM extracting & zipping
-`igir` has two ROM archive commands. Archive commands require either the `copy` or `move` write command. Only one archive command can be specified at a time.
+Igir has two ROM archive commands. Archive commands require either the `copy` or `move` write command. Only one archive command can be specified at a time.
-If no archive command is specified, files will be left as-is. If they are already extracted, then they will stay extracted. If they are already archived (including non-`.zip` archives), then they will stay archived.
+If no archive command is specified, files will be left as-is. If they are already extracted, then they will stay extracted. If they are already archived (including non-`.zip` archives), then they will stay archived in their original format.
!!! note
@@ -50,11 +50,15 @@ ROMs will be archived into a `.zip` file as they are being copied or moved. ROMs
ROMs that are already in an archive will be re-archived.
+!!! note
+
+ You can use the [`--dat-combine` option](dats/processing.md#dat-combining) to cause every ROM in a DAT to be zipped together.
+
## ROM verification
### `test`
-After performing one of the ROM writing commands, verify that the file was written correctly.
+After performing one of the ROM writing commands (above), verify that the file was written correctly.
- `extract test` tests that each ROM file written has the correct size & checksum
- `zip test` tests that the `.zip` file has all the correct archive entry sizes & checksums, and contains no excess entries
diff --git a/docs/dats/dir2dat.md b/docs/dats/dir2dat.md
index 1d5f2a132..de6e5cb0e 100644
--- a/docs/dats/dir2dat.md
+++ b/docs/dats/dir2dat.md
@@ -2,7 +2,7 @@
"dir2dat" refers to DATs that have been automatically created based on files in an input directory. [DATs](./introduction.md) generated this way are not typically useful as-is, they usually require some hand editing after creation.
-`igir` has the ability to create these DATs with the `igir dir2dat` command. Example:
+Igir can create these DATs with the `igir dir2dat` command. Example:
```shell
igir dir2dat --input [--input ..]
@@ -10,7 +10,7 @@ igir dir2dat --input [--input ..]
## dir2dat rules
-`igir` uses the following rules when creating dir2dat DAT files:
+Igir uses the following rules when creating dir2dat DAT files:
- **A DAT file will be created for every input path.**
@@ -87,6 +87,6 @@ Once DATs have been generated from input files, they are processed the same as a
## Alternative tools
-It is unlikely that any ROM tool, including `igir`, will ever meet every person's exact DAT creation needs.
+It is unlikely that any ROM tool, including Igir, will ever meet every person's exact DAT creation needs.
[SabreTools](https://github.com/SabreTools/SabreTools) is a great tool for DAT management that offers many complex options for DAT creation, filtering, merging, and splitting.
diff --git a/docs/dats/introduction.md b/docs/dats/introduction.md
index 6285a8d9e..bc1513c43 100644
--- a/docs/dats/introduction.md
+++ b/docs/dats/introduction.md
@@ -8,7 +8,7 @@ From the [RetroPie docs](https://retropie.org.uk/docs/Validating%2C-Rebuilding%2
DATs are catalog files of every known ROM that exists per game system, complete with enough information to identify each file.
-These DAT files ("DATs") help `igir` distinguish known ROM files in input directories from other files. Because DATs typically contain the complete catalog for a console, `igir` also uses them to generate reports for you on what ROMs were found and which are missing.
+These DAT files ("DATs") help Igir distinguish known ROM files in input directories from other files. Because DATs typically contain the complete catalog for a console, Igir also uses them to generate reports for you on what ROMs were found and which are missing.
The location to your DAT files are specified with the [`--dat ` option](./processing.md#scanning-for-dats):
@@ -22,7 +22,7 @@ you can even specify archives that can contain multiple DATs (such as No-Intro's
igir [commands..] --dat "No-Intro*.zip" --input
```
-See the [DAT processing page](./processing.md) for information on how `igir` scans for and processes DATs.
+See the [DAT processing page](./processing.md) for information on how Igir scans for and processes DATs.
## DAT release groups
@@ -46,9 +46,9 @@ And some less popular release groups are:
## Parent/clone (P/C) DATs
-DATs that include "parent" and "clone" information help `igir` understand what game releases are actually the same game (are "clones" of each other). Frequently a game will be released in many regions or with different revisions, usually with only language translations and minor bug fixes. For example, No-Intro has 6+ "clones" of Pokรฉmon Blue cataloged.
+DATs that include "parent" and "clone" information help Igir understand what game releases are actually the same game (are "clones" of each other). Frequently, a game will be released in many regions or with different revisions, usually with only language translations and minor bug fixes. For example, No-Intro has 6+ "clones" of Pokรฉmon Blue cataloged.
-Being able to know that many releases are actually the same game gives `igir` the ability to produce "one game, one ROM" (1G1R) sets with the [`--single` option](../roms/filtering-preferences.md#preferences-for-1g1r). 1G1R sets include only one of these "clone" releases, usually filtered to a language and region, because many people don't care about ROMs they can't understand.
+Being able to know that many releases are actually the same game gives Igir the ability to produce "one game, one ROM" (1G1R) sets with the [`--single` option](../roms/filtering-preferences.md#preferences-for-1g1r). 1G1R sets include only one of these "clone" releases, usually filtered to a language and region, because many people don't care about ROMs they can't understand.
!!! note
@@ -73,4 +73,4 @@ See the [arcade usage page](../usage/arcade.md) for more information on building
## Next steps
-See the [DAT processing page](./processing.md) for information on how `igir` scans for and processes DATs.
+See the [DAT processing page](./processing.md) for information on how Igir scans for and processes DATs.
diff --git a/docs/dats/processing.md b/docs/dats/processing.md
index 8333594d6..f1894427e 100644
--- a/docs/dats/processing.md
+++ b/docs/dats/processing.md
@@ -1,14 +1,14 @@
# DAT Processing
-`igir` has a number of ways it can process [DATs](./introduction.md), and it processes them in the following order.
+Igir has a number of ways it can process [DATs](./introduction.md), and it processes them in the following order.
## Just tell me what to do
-[DATs](./introduction.md) can get fairly complicated, and there are many release groups each with their own focus areas and naming patterns. If all you want to do is organize your ROMs with `igir` in some sane way, follow these instructions:
+[DATs](./introduction.md) can get fairly complicated, and there are many release groups, each with their own focus areas and naming patterns. If all you want to do is organize your ROMs with Igir in some consistent way, follow these instructions:
1. Go to the No-Intro DAT-o-MATIC [daily download page](https://datomatic.no-intro.org/index.php?page=download&s=64&op=daily)
2. Select the "P/C XML" radio option (as opposed to "standard DAT") and download the `.zip` to wherever you store your ROMs
-3. Every time you run `igir`, specify the `.zip` file you downloaded with the `--dat ` option:
+3. Every time you run Igir, specify the `.zip` file you downloaded with the `--dat ` option:
```shell
igir [commands..] --dat "No-Intro*.zip" --input
@@ -18,7 +18,7 @@
The `--dat ` option supports files, archives, directories, and globs like any of the other file options. See the [file scanning page](../input/file-scanning.md) for more information.
-`igir` also supports URLs to DAT files and archives. This is helpful to make sure you're always using the most up-to-date version of a DAT hosted on sites such as GitHub. For example:
+Igir also supports URLs to DAT files and archives. This is helpful to make sure you're always using the most up-to-date version of a DAT hosted on sites such as GitHub. For example:
```shell
igir [commands..] --dat "https://raw.githubusercontent.com/libretro/libretro-database/master/dat/DOOM.dat" --input
@@ -30,14 +30,14 @@ igir [commands..] --dat "https://raw.githubusercontent.com/libretro/libretro-dat
### Supported DAT formats
-There have been a few DAT-like formats developed over the years. `igir` supports the following:
+There have been a few DAT-like formats developed over the years. Igir supports the following:
- [Logiqx XML](https://github.com/SabreTools/SabreTools/wiki/DatFile-Formats#logiqx-xml-format) (most common) (No-Intro, Redump, TOSEC, and more)
- [MAME ListXML](https://easyemu.mameworld.info/mameguide/command_line/frontend_commands/listxml.html) (XML exported by the `mame -listxml` command)
!!! tip
- Instead of exporting the ListXML to a file yourself, you can also specify a MAME executable for the DAT path and then `igir` is smart enough to parse it:
+ Instead of exporting the ListXML to a file yourself, you can also specify a MAME executable for the DAT path and then Igir is smart enough to parse it:
=== ":simple-windowsxp: Windows"
@@ -69,11 +69,11 @@ There have been a few DAT-like formats developed over the years. `igir` supports
!!! tip
- In case you come across a DAT in a format that `igir` doesn't support, SabreTools supports reading [a number of obscure formats](https://github.com/SabreTools/SabreTools/wiki/DatFile-Formats) and converting them to more standard formats such as Logiqx XML.
+ In case you come across a DAT in a format that Igir doesn't support, SabreTools supports reading [a number of obscure formats](https://github.com/SabreTools/SabreTools/wiki/DatFile-Formats) and converting them to more standard formats such as Logiqx XML.
## DAT filtering
-To be able to process only the DATs you want in downloaded archives, `igir` has a few filtering options.
+To be able to process only the DATs you want in downloaded archives, Igir has a few filtering options.
### DAT name regex filtering
@@ -92,12 +92,12 @@ Headerless|Encrypted
!!! tip
- `--dat-name-regex-exclude` is particularly helpful for excluding some No-Intro DATs versions such as "encrypted" and "headerless".
+ `--dat-name-regex-exclude ` is particularly helpful for excluding some No-Intro DATs versions such as "encrypted" and "headerless".
### DAT description regex filtering
```text
---dat-description-regex, --dat-description-regex-exclude
+--dat-description-regex , --dat-description-regex-exclude
```
These options limit which DATs are processed. The regex is applied to the DAT's description found within its file contents.
@@ -108,11 +108,17 @@ The `--dat-combine` option lets you combine every game from every parsed DAT int
This may be desirable when creating a [dir2dat](./dir2dat.md), a [fixdat](fixdats.md), or other complicated situations.
+!!! note
+
+ Using this option with the [`igir zip` command](../output/writing-archives.md) will result in all ROMs in a DAT being archived into one file. This can work great for archiving older, cartridge-based consoles with smaller ROM sizes, but will likely not work well with larger ROMs.
+
+ To keep files organized in a human-readable way, it is _not_ recommended to use the [`--dir-game-subdir never`](../output/path-options.md#append-the-game-name) option along with `igir zip --dat-combine`.
+
## Parent/clone inference
-One feature that sets `igir` apart from other ROM managers is its ability to infer parent/clone information when DATs don't provide it. For example, Redump DATs don't provide parent/clone information, which makes it much more difficult to create 1G1R sets.
+One feature that sets Igir apart from other ROM managers is its ability to infer parent/clone information when DATs don't provide it. For example, Redump DATs don't provide parent/clone information, which makes it much more difficult to create 1G1R sets.
-For example, all of these Super Smash Bros. Melee releases should be considered the same game, even if a DAT doesn't provide proper information. If the releases are all considered the same game, then the `--single` option can be used in combination with [ROM preferences](../roms/filtering-preferences.md) to make a 1G1R set. `igir` is smart enough to understand that the only differences between these releases are the regions, languages, and revisions.
+For example, all of these Super Smash Bros. Melee releases should be considered the same game, even if a DAT doesn't provide proper information. If the releases are all considered the same game, then the `--single` option can be used in combination with [ROM preferences](../roms/filtering-preferences.md) to make a 1G1R set. Igir is smart enough to understand that the only differences between these releases are the regions, languages, and revisions.
```text
Super Smash Bros. Melee (Europe) (En,Fr,De,Es,It)
@@ -124,14 +130,14 @@ Super Smash Bros. Melee (USA) (En,Ja) (Rev 2)
!!! note
- If a DAT has any parent/clone information then `igir` will use that and skip inference. If you want to ignore this information, you can provide the `--dat-ignore-parent-clone` option.
+ If a DAT has any parent/clone information then Igir will use that and skip inference. If you want to ignore this information, you can provide the `--dat-ignore-parent-clone` option.
!!! note
- It is unlikely that `igir` will ever be perfect with inferring parent/clone information. If you find an instance where `igir` made the wrong choice, please create a [GitHub issue](https://github.com/emmercm/igir/issues).
+ It is unlikely that Igir will ever be perfect with inferring parent/clone information. If you find an instance where Igir made the wrong choice, please create a [GitHub issue](https://github.com/emmercm/igir/issues).
!!! tip
[Retool](https://github.com/unexpectedpanda/retool) (no longer maintained) is a DAT manipulation tool that has a set of hand-maintained [parent/clone lists](https://github.com/unexpectedpanda/retool-clonelists-metadata) to supplement common DAT groups such as No-Intro and Redump. This helps cover situations such as release titles in different languages that would be hard to group together automatically.
- 1G1R DATs made by Retool can be used seamlessly with `igir`. You won't need to supply the `--single` option or any [ROM preferences](../roms/filtering-preferences.md) for `igir`, as you would have already applied these preferences in Retool, but you can still supply [ROM filtering](../roms/filtering-preferences.md) options if desired.
+ 1G1R DATs made by Retool can be used seamlessly with Igir. You won't need to supply the `--single` option or any [ROM preferences](../roms/filtering-preferences.md) for Igir, as you would have already applied these preferences in Retool, but you can still supply [ROM filtering](../roms/filtering-preferences.md) options if desired.
diff --git a/docs/input/file-scanning.md b/docs/input/file-scanning.md
index 747be77d4..07fcbd3ef 100644
--- a/docs/input/file-scanning.md
+++ b/docs/input/file-scanning.md
@@ -1,6 +1,6 @@
# File Scanning
-`igir` has a few options to specify input files, as well as files to exclude:
+Igir has a few options to specify input files, as well as files to exclude:
- ROMs: `--input ` (required), `--input-exclude `
- [DATs](../dats/processing.md): `--dat `, `--dat-exclude `
@@ -8,7 +8,7 @@
## Archive files
-`igir` can scan archives for DATs, ROMs, and patches. See the [archives](reading-archives.md) page for more information on supported formats.
+Igir can scan archives for DATs, ROMs, and patches. See the [archives](reading-archives.md) page for more information on supported formats.
## Glob patterns
diff --git a/docs/input/reading-archives.md b/docs/input/reading-archives.md
index 2be505334..fb4379612 100644
--- a/docs/input/reading-archives.md
+++ b/docs/input/reading-archives.md
@@ -1,38 +1,41 @@
# Reading Archives
-`igir` supports scanning the contents of archives for ROMs, DATs, and ROM patches.
+Igir supports scanning the contents of archives for ROMs, DATs, and ROM patches.
## Supported types for reading
-`igir` supports most common archive formats:
-
-| Extension | Contains file CRC32s | `igir` can extract without a third-party binary | `igir` can checksum without temporary files |
-|--------------------------|----------------------|-------------------------------------------------|---------------------------------------------|
-| `.7z` | โ | โ | โ |
-| `.gz`, `.gzip` | โ CRC16 | โ | โ |
-| `.rar` | โ | โ | โ |
-| `.tar` | โ | โ | โ โค64MiB |
-| `.tar.gz`, `.tgz` | โ | โ | โ โค64MiB |
-| `.z01` | โ | โ | โ |
-| `.zip` (including zip64) | โ | โ | โ โค64MiB |
-| `.zip.001` | โ | โ | โ |
-| `.zipx` | โ | โ | โ |
+Igir supports most common archive formats:
+
+| Extension | Contains file CRC32s | Igir can extract without a third-party binary | Igir can checksum without temporary files |
+|------------------------------------------------------------------|----------------------|-----------------------------------------------|-------------------------------------------|
+| `.7z` | โ | โ `7za` | โ |
+| `.chd` | โ SHA1 | โ `chdman` | โ |
+| `.cso`, `.zso`, `.dax` | โ | โ `maxcso` | โ ๏ธ CRC32 only |
+| `.gz`, `.gzip` | โ CRC16 | โ `7za` | โ |
+| `.nkit.iso` ([GameCube docs](../usage/console/gamecube.md#nkit)) | โ | โ no extraction support | โ |
+| `.rar` | โ | โ | โ |
+| `.tar` | โ | โ | โ โค64MiB |
+| `.tar.gz`, `.tgz` | โ | โ | โ โค64MiB |
+| `.z01` | โ | โ `7za` | โ |
+| `.zip` (including zip64) | โ | โ | โ โค64MiB |
+| `.zip.001` | โ | โ `7za` | โ |
+| `.zipx` | โ | โ `7za` | โ |
**You should prefer archive formats that have CRC32 checksum information for each file.**
-By default, `igir` uses CRC32 information to [match ROMs](../roms/matching.md) to DAT entries. If an archive already contains CRC32 information for each file, then `igir` doesn't need to extract each file and compute its CRC32. This can save a lot of time on large archives.
+By default, Igir uses CRC32 information to [match ROMs](../roms/matching.md) to DAT entries. If an archive already contains CRC32 information for each file, then Igir doesn't need to extract each file and compute its CRC32. This can save a lot of time on large archives.
-This is why you should use the [`igir zip` command](../output/writing-archives.md) when organizing your primary ROM collection. It is much faster for `igir` to scan archives with CRC32 information, speeding up actions such as merging new ROMs into an existing collection.
+This is why you should use the [`igir zip` command](../output/writing-archives.md) when organizing your primary ROM collection. It is much faster for Igir to scan archives with CRC32 information, speeding up actions such as merging new ROMs into an existing collection.
-**You should prefer archive formats that `igir` can extract natively.**
+**You should prefer archive formats that Igir can extract natively.**
-Somewhat proprietary archive formats such as `.7z` and `.rar` require `igir` to use an external tool to enumerate and extract files. This can greatly slow down processing speed.
+Somewhat proprietary archive formats such as `.7z` and `.rar` require Igir to use an external tool to enumerate and extract files. This can greatly slow down processing speed.
-This is why `igir` uses `.zip` as its output archive of choice, `.zip` files are easy and fast to read, even if they can't offer as high of compression as other formats.
+This is why Igir uses `.zip` as its output archive of choice, `.zip` files are easy and fast to read, even if they can't offer as high of compression as other formats.
## Exact archive matching
-Some DAT files such as the [libretro BIOS System.dat](https://github.com/libretro/libretro-database/blob/master/dat/System.dat) catalog archives such as zip files, rather than the contents of those archives. By default, `igir` will try to detect DATs like these and calculate checksums for all archive files, in addition to the files they contain.
+Some DAT files such as the [libretro BIOS System.dat](https://github.com/libretro/libretro-database/blob/master/dat/System.dat) catalog archives such as zip files, rather than the contents of those archives. By default, Igir will try to detect DATs like these and calculate checksums for all archive files, in addition to the files they contain.
This adds a potentially non-trivial amount of processing time during ROM scanning, so this behavior can be turned off with the option:
@@ -40,7 +43,7 @@ This adds a potentially non-trivial amount of processing time during ROM scannin
--input-checksum-archives never
```
-If for some reason `igir` isn't identifying an input file correctly as an archive, this additional processing can be forced with the option:
+If for some reason Igir isn't identifying an input file correctly as an archive, this additional processing can be forced with the option:
```text
--input-checksum-archives always
@@ -48,6 +51,6 @@ If for some reason `igir` isn't identifying an input file correctly as an archiv
## Checksum cache
-It can be expensive to calculate checksums of files within archives, especially MD5, SHA1, and SHA256. If `igir` needs to calculate a checksum that is not easily read from the archive (see above), it will cache the result in a file named `igir.cache`. This cached result will then be used as long as the input file's size and modified timestamp remain the same.
+It can be expensive to calculate checksums of files within archives, especially MD5, SHA1, and SHA256. If Igir needs to calculate a checksum not easily read from the archive (see above), it will cache the result in a file named `igir.cache`. This cached result will then be used as long as the input file's size and modified timestamp remain the same.
-The location of this cache file can be controlled with the `--cache-path ` option, or caching can be disabled entirely with the `--disable-cache` option. You can safely delete `igir.cache` when `igir` isn't running if the file becomes too large for you.
+The location of this cache file can be controlled with the `--cache-path ` option, or caching can be disabled entirely with the `--disable-cache` option. You can safely delete `igir.cache` when Igir isn't running if the file becomes too large for you.
diff --git a/docs/installation.md b/docs/installation.md
index 979a3a217..cc6475a71 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -1,15 +1,15 @@
# Installation
-`igir` is supported on :simple-windowsxp: Windows, :simple-apple: macOS, :simple-linux: Linux, and every other operating system that [Node.js](https://nodejs.org) supports.
+Igir is supported on :simple-windowsxp: Windows, :simple-apple: macOS, :simple-linux: Linux, and every other operating system that [Node.js](https://nodejs.org) supports.
-There are a few different installation options offered for `igir` with varying levels of technical complexity. Every option will require some baseline understanding of command-line interfaces (CLIs).
+There are a few different installation options offered for Igir with varying levels of technical complexity. Every option will require some baseline understanding of command-line interfaces (CLIs).
## Via Node.js
[](https://www.npmjs.com/package/igir)
[](https://nodejs.org/en/download/)
-The best way to ensure that you are always running the most up-to-date version of `igir` is to run it via [`npx`](https://docs.npmjs.com/cli/v9/commands/npx) which comes installed with [Node.js](https://nodejs.org/en/download/):
+The best way to ensure that you are always running the most up-to-date version of Igir is to run it via [`npx`](https://docs.npmjs.com/cli/v9/commands/npx) which comes installed with [Node.js](https://nodejs.org/en/download/):
```shell
npx igir@latest [commands..] [options]
@@ -21,7 +21,7 @@ for example:
npx igir@latest copy extract --dat *.dat --input ROMs/ --output ROMs-Sorted/ --dir-dat-name
```
-[](https://asciinema.org/a/hjMOlN3DwSgo9NGHzPtncOoq9)
+
!!! tip
@@ -41,7 +41,7 @@ npx igir@latest copy extract --dat *.dat --input ROMs/ --output ROMs-Sorted/ --d
!!! note
- If you want to help beta test `igir`, you can run the most bleeding-edge version with the command:
+ If you want to help beta test Igir, you can run the most bleeding-edge version with the command:
```shell
npm exec --yes -- "github:emmercm/igir#main" [commands..] [options]
@@ -49,14 +49,14 @@ npx igir@latest copy extract --dat *.dat --input ROMs/ --output ROMs-Sorted/ --d
## Via Homebrew (macOS)
-[Homebrew](https://brew.sh/) is third-party package manager for macOS. You can install `igir` with these simple commands:
+[Homebrew](https://brew.sh/) is a third-party package manager for macOS. You can install Igir with these simple commands:
```shell
brew tap emmercm/igir
brew install igir
```
-You can then update `igir` with _either_ of these commands
+You can then update Igir with _either_ of these commands
```shell
# Update every Homebrew package
@@ -71,43 +71,3 @@ brew upgrade igir
[](https://github.com/emmercm/igir/releases/latest)
If you don't want to download Node.js, you can download executables for various OSes from the [GitHub releases](https://github.com/emmercm/igir/releases) page.
-
-## Via Docker
-
-If none of the above options work for you, [Docker](https://www.docker.com/) may be an option. You will need to mount your input and output directories as volumes, which will significantly reduce your file read and write speeds.
-
-=== ":simple-windowsxp: Windows"
-
- ```batch
- docker run --interactive --tty --rm ^
- --volume "%cd%:\pwd" ^
- --workdir "/pwd" ^
- node:lts ^
- npx igir@latest copy zip --dat "*.dat" --input ROMs\ --output ROMs-Sorted\ --dir-dat-name
- ```
-
-=== ":simple-apple: macOS"
-
- ```shell
- docker run --interactive --tty --rm \
- --volume "$PWD:/pwd" \
- --workdir "/pwd" \
- node:lts \
- npx igir@latest copy zip --dat "*.dat" --input ROMs/ --output ROMs-Sorted/ --dir-dat-name
- ```
-
-=== ":simple-linux: Linux"
-
- ```shell
- docker run --interactive --tty --rm \
- --volume "$PWD:/pwd" \
- --workdir "/pwd" \
- node:lts \
- npx igir@latest copy zip --dat "*.dat" --input ROMs/ --output ROMs-Sorted/ --dir-dat-name
- ```
-
-!!! warning
-
- Make sure to quote all of your [file globs](input/file-scanning.md)!
-
-[](https://asciinema.org/a/5OAVbSXXoosTr0WyBvjQGBqRp)
diff --git a/docs/overview.md b/docs/introduction.md
similarity index 68%
rename from docs/overview.md
rename to docs/introduction.md
index 227f6f1b5..0b0cd41aa 100644
--- a/docs/overview.md
+++ b/docs/introduction.md
@@ -1,4 +1,4 @@
-# Overview
+# Introduction
## What is a ROM?
@@ -8,7 +8,7 @@ From [Wikipedia](https://en.wikipedia.org/wiki/ROM_image):
ROMs are complete copies of game data stored in cartridges or on discs.
-A game may consist of multiple ROMs. For example, arcade cabinets that contain multiple chips, or disc-based games that have multiple tracks on the disc.
+A game may consist of multiple ROMs. For example, arcade cabinets, which contain multiple chips, or disc-based games that have multiple tracks on the disc.
## What is a ROM manager?
@@ -19,22 +19,22 @@ ROM managers are applications that serve two main purposes:
all additional features help serve these two purposes.
-Most ROM managers can automatically read & write many different ROM types including those in [archives](input/reading-archives.md) and those with [headers](roms/headers.md) so that you don't have to do much pre-work.
+Most ROM managers can automatically read & write many different ROM types, including those in [archives](input/reading-archives.md) and those with [headers](roms/headers.md) so that you don't have to do much pre-work.
-Most ROM managers rely on [DATs](dats/introduction.md), files that catalog every known ROM that exists per game system. DATs are published by release groups dedicated to keeping these catalogs accurate and up-to-date. DATs help ROM collectors name their ROMs in a consistent way as well as understand what ROMs may be missing from their collection.
+Most ROM managers rely on [DATs](dats/introduction.md), files that catalog every known ROM that exists per game system. DATs are published by release groups dedicated to keeping these catalogs accurate and up to date. DATs help ROM collectors name their ROMs consistently as well as understand what ROMs may be missing from their collection.
-## What is `igir`?
+## What is Igir?
-`igir` is a ROM manager for the modern age.
+Igir is a ROM manager for the modern age.
-Most ROM managers are only built for Windows, and some offer workarounds for running on macOS and Linux. Most of these managers have confusing GUIs that make batch-able, repeatable actions difficult. `igir` is a command line tool that works on any OS.
+Most ROM managers are only built for Windows, and some offer workarounds for running on macOS and Linux. Most of these managers have confusing GUIs that make batch-able, repeatable actions difficult. Igir is a command line tool that works on any OS.
-In addition, `igir` has features that aren't found in any other ROM managers, such as [ROM patching](roms/patching.md).
+In addition, Igir has features that aren't found in any other ROM managers, such as [ROM patching](roms/patching.md).
!!! info
- See the [alternative managers](alternatives.md) page for a feature comparison between `igir` and other ROM managers.
+ See the [alternative managers](alternatives.md) page for a feature comparison between Igir and other ROM managers.
## Next steps
-See the [installation](installation.md) page for instructions on getting `igir` installed.
+See the [installation](installation.md) page for instructions on getting Igir installed.
diff --git a/docs/output/cleaning.md b/docs/output/cleaning.md
index 5f35aef94..7b8dafc3b 100644
--- a/docs/output/cleaning.md
+++ b/docs/output/cleaning.md
@@ -15,9 +15,9 @@ In practical terms, this means:
**2. If [tokens](tokens.md) are used with the `--output ` option, only subdirectories that are written to will be considered for cleaning.**
-For example, if the output directory is specified as `--output "games/{mister}"`, and only Game Boy Color games are found in `--input `, then only the `games/Gameboy/` directory would be considered for cleaning. Other directories that may already exist such as `games/GBA/` and `games/NES/` would _not_ be considered for cleaning, as `igir` did not write there.
+For example, if the output directory is specified as `--output "games/{mister}"`, and only Game Boy Color games are found in `--input `, then only the `games/Gameboy/` directory would be considered for cleaning. Other directories that may already exist such as `games/GBA/` and `games/NES/` would _not_ be considered for cleaning, as Igir did not write there.
-In other words, `games/{mister}` is _not_ equivalent to `games/*`. `igir` will _not_ indiscriminately delete files in `games/`.
+In other words, `games/{mister}` is _not_ equivalent to `games/*`. Igir will _not_ indiscriminately delete files in `games/`.
If you want to clean _every_ directory in `games/`, you could specify it as both the `--input ` and `--output `:
@@ -47,7 +47,7 @@ See the [Analogue Pocket](../usage/hardware/analogue-pocket.md) page for a pract
## Backing up cleaned files
-By default, `igir` will recycle cleaned files, and if recycle fails then it will delete them. This is potentially destructive, so a `--clean-backup ` option is provided to instead move files to a backup directory.
+By default, Igir will recycle cleaned files, and if recycle fails, then it will delete them. This is potentially destructive, so a `--clean-backup ` option is provided to instead move files to a backup directory.
The input directory structure is not maintained, no subdirectories will be created in the backup directory. Files of conflicting names will have a number appended to their name, e.g. `File (1).rom`.
diff --git a/docs/output/options.md b/docs/output/options.md
index 01ff9c926..ef24a1a3a 100644
--- a/docs/output/options.md
+++ b/docs/output/options.md
@@ -2,7 +2,7 @@
## Overwriting files
-By default, `igir` will _not_ overwrite or delete any files already in the output directory.
+By default, Igir will _not_ overwrite or delete any files already in the output directory.
To change this behavior, the `--overwrite` option will force overwriting files in the output directory as necessary. Be careful with this option as it can cause unnecessary wear and tear on your hard drives.
@@ -10,7 +10,7 @@ The `--overwrite-invalid` option can also overwrite files in the output director
## Fixing ROM extensions
-ROM dumpers don't always do a good job of using the generally accepted filename extension when writing files. In situations where DATs aren't provided, or information in DATs is incomplete, `igir` has some ability to find the correct extension that filenames should have. This is done using [file signatures](https://en.wikipedia.org/wiki/List_of_file_signatures), pieces of data that are common to every file of a certain format.
+ROM dumpers don't always do a good job of using the generally accepted filename extension when writing files. In situations where DATs aren't provided, or information in DATs is incomplete, Igir has some ability to find the correct extension that filenames should have. This is done using [file signatures](https://en.wikipedia.org/wiki/List_of_file_signatures), pieces of data that are common to every file of a certain format.
Here are some examples of common mistakes:
diff --git a/docs/output/path-options.md b/docs/output/path-options.md
index 562243258..d4947d84b 100644
--- a/docs/output/path-options.md
+++ b/docs/output/path-options.md
@@ -1,6 +1,6 @@
# Output Path Options
-`igir` offer many options to control how ROMs are sorted in the specified output directory.
+Igir offer many options to control how ROMs are sorted in the specified output directory.
All `--dir-*` options append subdirectories to whatever is specified in the `--output ` option. Many `--dir-*` options have an [output path token](./tokens.md) equivalent, which also controls how ROMs are sorted.
@@ -485,3 +485,67 @@ You can also combine this option with `--dir-letter-count ` for ranges wi
```text
--dir-game-subdir
```
+
+By default, games with multiple ROMs are grouped together into their own output subdirectory. This is because emulators typically expect these files to be next to each other, but also because different games may have duplicate filenames (e.g. Sega Dreamcast GDIs all have a `track01.bin`).
+
+```text
+ROMS-Output/
+โโโ TOSEC
+ โโโ Sega Dreamcast - Games - US
+ โ โโโ Sonic Adventure 2 v1.008 (2001)(Sega)(US)(M5)[!][3S]
+ โ โ โโโ Sonic Adventure 2 v1.008 (2001)(Sega)(US)(M5)[!][3S].gdi
+ โ โ โโโ track01.bin
+ โ โ โโโ track02.raw
+ โ โ โโโ track03.bin
+ โ โโโ Sonic Adventure v1.005 (1999)(Sega)(US)(M5)[!][26S]
+ โ โโโ Sonic Adventure v1.005 (1999)(Sega)(US)(M5)[!][26S].gdi
+ โ โโโ track01.bin
+ โ โโโ track02.raw
+ โ โโโ track03.bin
+ โโโ Sega Mega-CD & Sega CD - CD - Games - [ISO]
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)[!][SEGA4407RE152 R7D]
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 01 of 35)[!][SEGA4407RE152 R7D].iso
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 02 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 03 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 04 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 05 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 06 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 07 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 08 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 09 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 10 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 11 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 12 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 13 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 14 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 15 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 16 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 17 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 18 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 19 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 20 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 21 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 22 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 23 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 24 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 25 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 26 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 27 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 28 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 29 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 30 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 31 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 32 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 33 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 34 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)(Track 35 of 35)[!][SEGA4407RE152 R7D].wav
+ โโโ Sonic CD (1993)(Sega)(NTSC)(US)[!][SEGA4407RE152 R7D].cue
+```
+
+You can change this behavior with the `--dir-game-subdir ` option:
+
+| Mode | Outcome |
+|------------------------------------|------------------------------------------------------------------------------------------------------------------|
+| `--dir-game-subdir never` | Games with multiple ROMs are never grouped into their own subdirectory, which may cause conflicting output files |
+| `--dir-game-subdir auto` (default) | Games with multiple ROMs are grouped into their own subdirectory, games with a single ROM are not |
+| `--dir-game-subdir always` | Every game is grouped into its on subdirectory, no matter the number of ROMs it has |
diff --git a/docs/output/reporting.md b/docs/output/reporting.md
index b2059d434..dbc5d68f5 100644
--- a/docs/output/reporting.md
+++ b/docs/output/reporting.md
@@ -11,9 +11,9 @@ When using DATs (the [`--dat ` option](../dats/processing.md#scanning-for-
- `UNUSED`: what input files didn't match to any ROM
- `DELETED`: what output files were [cleaned](cleaning.md) (`igir clean` command)
-At least one DAT is required for the `igir report` command to work, otherwise `igir` has no way to understand what input files are known ROMs and which aren't. See the [DAT docs](../dats/introduction.md) for more information about DATs.
+At least one DAT is required for the `igir report` command to work, otherwise Igir has no way to understand what input files are known ROMs and which aren't. See the [DAT docs](../dats/introduction.md) for more information about DATs.
-The `igir report` command can be specified on its own without any [writing command](../commands.md) (i.e. `igir copy`, `igir move`, etc.) in order to report on an existing collection. This causes `igir` to operate in a _read-only_ mode, no files will be copied, moved, or deleted. For example:
+The `igir report` command can be specified on its own without any [writing command](../commands.md) (i.e. `igir copy`, `igir move`, etc.) to report on an existing collection. This causes Igir to operate in a _read-only_ mode, no files will be copied, moved, or deleted. For example:
=== ":simple-windowsxp: Windows"
@@ -48,7 +48,7 @@ See the `igir --help` message for the report's default location.
The output report format is a standard CSV that can be opened in Microsoft Excel, Apple Numbers, Google Sheets, LibreOffice Calc, and other similar spreadsheet applications.
-Unlike the report formats of [other ROM managers](../alternatives.md), CSVs allow you to filter rows by column values. For example, you can filter the "Status" column to only "MISSING" to understand what ROMs are missing from your collection, or to "UNUSED" to understand what input files weren't used as the source of any output file. The ability to filter CSVs in spreadsheet applications means that `igir` should not need use-case-specific report options to achieve your goal.
+Unlike the report formats of [other ROM managers](../alternatives.md), CSVs allow you to filter rows by column values. For example, you can filter the "Status" column to only "MISSING" to understand what ROMs are missing from your collection, or to "UNUSED" to understand what input files weren't used as the source of any output file. The ability to filter CSVs in spreadsheet applications means that Igir shouldnโt need use-case-specific report options to achieve your goal.
To perform this filtering, most spreadsheet applications have a button or menu item to "create a filter" or "auto filter."
diff --git a/docs/output/tokens.md b/docs/output/tokens.md
index 1aed1dfb5..b178d1786 100644
--- a/docs/output/tokens.md
+++ b/docs/output/tokens.md
@@ -1,6 +1,6 @@
# Output Path Tokens
-When specifying a ROM [writing command](../commands.md) you have to specify an `--output ` directory. `igir` has a few replaceable "tokens" that can be referenced in the `--output ` directory value. This can aid in sorting ROMs into a more complicated directory structure.
+When specifying a ROM [writing command](../commands.md) you have to specify an `--output ` directory. Igir has a few replaceable "tokens" that can be referenced in the `--output ` directory value. This can aid in sorting ROMs into a more complicated directory structure.
See [output path tokens](./path-options.md) for other options that will further sort your ROMs into subdirectories.
@@ -60,14 +60,10 @@ When using [DATs](../dats/introduction.md), you can make use of console & game i
- `{datName}` the matching DAT's name, similar to how the [`--dir-dat-name` option](./path-options.md) works
- `{datDescription}` the matching DAT's description, similar to how the [`--dir-dat-description` option](./path-options.md) works
-- `{region}` each of the ROM's region(s) (e.g. `USA`, `EUR`, `JPN`, `WORLD`)
-- `{language}` each of the ROM's language(s) (e.g. `EN`, `ES`, `JA`)
-
-## Game information
-
-You can use some information about each game:
-
-- `{gameType}` the game's "type," one of: `Aftermarket`, `Alpha`, `Bad`, `Beta`, `BIOS`, `Demo`, `Device`, `Fixed`, `Hacked`, `Homebrew`, `Overdump`, `Pending Dump`, `Pirated`, `Prototype`, `Retail` (most games will be this), `Sample`, `Test`, `Trained`, `Translated`, `Unlicensed`
+- `{region}` each of the game's region(s) (e.g. `USA`, `EUR`, `JPN`, `WORLD`)
+- `{language}` each of the game's language(s) (e.g. `EN`, `ES`, `JA`)
+- `{type}` the game's "type," one of: `Aftermarket`, `Alpha`, `Bad`, `Beta`, `BIOS`, `Demo`, `Device`, `Fixed`, `Hacked`, `Homebrew`, `Overdump`, `Pending Dump`, `Pirated`, `Prototype`, `Retail` (most games will be this), `Sample`, `Test`, `Trained`, `Translated`, `Unlicensed`
+- `{genre}` the game's "genre" (most DATs don't provide this)
## File information
@@ -80,7 +76,7 @@ You can use some information about the input and output file's name & location:
## Specific hardware
-To help sort ROMs into unique file structures for popular frontends & hardware, `igir` offers a few specific tokens:
+To help sort ROMs into unique file structures for popular frontends & hardware, Igir offers a few specific tokens:
- `{adam}` the ['Adam' image](../usage/handheld/adam.md) emulator's directory for the ROM
- `{batocera}` the [Batocera](../usage/desktop/batocera.md) emulator's directory for the ROM
diff --git a/docs/output/writing-archives.md b/docs/output/writing-archives.md
index 742a5e7f8..570288750 100644
--- a/docs/output/writing-archives.md
+++ b/docs/output/writing-archives.md
@@ -1,12 +1,12 @@
# Writing Zip Archives
-`igir` supports creating `.zip` archives with the `igir zip` [command](../commands.md).
+Igir supports creating `.zip` archives with the `igir zip` [command](../commands.md).
!!! note
- It is intentional that `igir` only supports `.zip` archives right now.
+ It is intentional that Igir only supports `.zip` archives right now.
- `.zip` archives store CRC32 information in their "central directory" which helps drastically speed up `igir`'s file scanning, and they are easy to create without proprietary tools (e.g. 7-Zip, Rar).
+ `.zip` archives store CRC32 information in their "central directory" which helps drastically speed up Igir's file scanning, and they are easy to create without proprietary tools (e.g. 7-Zip, Rar).
See the [reading archives](../input/reading-archives.md) page for more information on archive formats and their capabilities.
diff --git a/docs/rom-dumping.md b/docs/rom-dumping.md
index e73bd4b1c..14ed87166 100644
--- a/docs/rom-dumping.md
+++ b/docs/rom-dumping.md
@@ -10,7 +10,7 @@
[Dumping.Guide](https://dumping.guide/start) and [Emulation General Wiki](https://emulation.gametechwiki.com/index.php/Ripping_games) are some of the best resources for legally creating ROM files from games you own.
-Here is a condensed version that isn't guaranteed to be up-to-date.
+Here is a condensed version that isn't guaranteed to be up to date.
## Generation 1-5 cartridge-based consoles
diff --git a/docs/roms/filtering-preferences.md b/docs/roms/filtering-preferences.md
index 63dd40c33..2d8c0a62f 100644
--- a/docs/roms/filtering-preferences.md
+++ b/docs/roms/filtering-preferences.md
@@ -1,6 +1,6 @@
# ROM Filtering & Preferences
-`igir` offers many options for filtering as well as 1G1R preferences/priorities (when combined with the `--single` option).
+Igir offers many options for filtering as well as 1G1R preferences/priorities (when combined with the `--single` option).
ROM filters cut down the list of games desired for a set, and any games filtered out will not appear in [reports](../output/reporting.md). ROM preferences decide what duplicates to eliminate (1G1R).
@@ -14,7 +14,7 @@ Multiple filter options can be specified at once.
--filter-regex , --filter-regex-exclude
```
-Only include, or exclude games based if their DAT name (or filename if not using DATs) matches a regular expression.
+Only include or exclude games based if their DAT name (or filename if not using DATs) matches a regular expression.
Regex flags can be optionally provided in the form `//`, for example:
@@ -50,9 +50,9 @@ Wario Land II (USA, Europe) (SGB Enhanced)
Languages are two-letter codes, and multiple languages can be specified with commas between them. See the `--help` message for the full list of understood languages.
-If a game does not have language information specified, it will be inferred from the region.
+If a game doesnโt have language information specified, it will be inferred from the region.
-Here are some example game names that `igir` can parse languages from, including ones with multiple languages:
+Here are some example game names that Igir can parse languages from, including ones with multiple languages:
```text
English:
@@ -82,7 +82,7 @@ A game can have many languages, and all of them are considered during filtering.
Regions are two or three-letter codes, and you can specify multiple regions with commas between them. See the `--help` message for the full list of understood regions.
-Here are some example game names that `igir` can parse regions from:
+Here are some example game names that Igir can parse regions from:
```text
USA:
@@ -129,7 +129,7 @@ Filter out, or only include games that are marked `bios="yes"` in the DAT, or co
--no-device, --only-device
```
-Filter out, or only include [MAME devices](https://wiki.mamedev.org/index.php/MAME_Device_Basics). MAME devices typically represent physical devices, such as microcontrollers, video display controllers, sounds boards, and more. Many MAME devices don't have any associated ROM files.
+Filter out or only include [MAME devices](https://wiki.mamedev.org/index.php/MAME_Device_Basics). MAME devices typically represent physical devices, such as microcontrollers, video display controllers, sounds boards, and more. Many MAME devices don't have any associated ROM files.
### Unlicensed
@@ -245,7 +245,7 @@ Perfect Dark (USA) (2000-03-22) (Debug)
--no-demo, --only-demo
```
-Filter out, or only include games that contain one of the following in their name:
+Filter out or only include games that contain one of the following in their name:
- `(Demo[a-z0-9. -]*)` (regex)
- `@barai`
@@ -310,7 +310,7 @@ Sword of Hope, The (Europe) (Proto)
--no-program, --only-program
```
-Filter out, or only include games that contain one of the following in their name
+Filter out or only include games that contain one of the following in their name
- `([a-z0-9. ]*Program)` (regex)
- `Check Program`
@@ -450,7 +450,7 @@ See the [bad dumps](#bad-dumps) section for more information about "good" and "b
Prefer games of certain languages over those in other languages. Multiple languages can be specified, in priority order, with commas between them. See the `--help` message for the full list of understood languages.
-If a game does not have language information specified, it will be inferred from the region.
+If a game doesnโt have language information specified, it will be inferred from the region.
For example, to prefer games in English and _then_ Japanese, the command would be:
@@ -475,10 +475,10 @@ For example, to prefer games from: USA (highest priority), "world," and then Eur
### Prefer revision
```text
---prefer-revision-newer, --prefer-revision-older
+--prefer-revision
```
-Prefer newer or older revisions of a game.
+Prefer newer or older revisions, versions, or ring codes of a game.
Revisions can be numeric:
@@ -496,27 +496,30 @@ MSR - Metropolis Street Racer (Europe) (En,Fr,De,Es) (Rev A)
MSR - Metropolis Street Racer (Europe) (En,Fr,De,Es) (Rev B)
```
-### Prefer retail
+Versions can be semantic:
```text
---prefer-retail
+F1 World Grand Prix for Dreamcast v1.011 (1999)(Video System)(JP)(en)[!]
+F1 World Grand Prix for Dreamcast v1.000 (1999)(Video System)(PAL)(M4)[!]
+F1 World Grand Prix v1.006 (2000)(Video System)(US)(M4)[!]
```
-Prefer games that are considered "retail" releases over those that aren't.
+Ring codes can be numeric:
-See the [only retail](#only-retail) section for more information on what games are considered "retail."
+```text
+Sonic CD (USA) (RE125)
+Sonic CD (USA) (RE125) (Alt)
+```
-### Prefer NTSC, PAL
+### Prefer retail
```text
---prefer-ntsc, --prefer-pal
+--prefer-retail
```
-Prefer games that are explicitly labeled as NTSC or PAL, over those that aren't.
-
-!!! note
+Prefer games that are considered "retail" releases over those that aren't.
- Most DAT groups do not label games with this information, generally games are labeled by region instead.
+See the [only retail](#only-retail) section for more information on what games are considered "retail."
### Prefer parent
diff --git a/docs/roms/headers.md b/docs/roms/headers.md
index 1ac75da65..b5dc7b5a9 100644
--- a/docs/roms/headers.md
+++ b/docs/roms/headers.md
@@ -6,7 +6,7 @@ Some of these headers are used to tell the emulator information about how to emu
## Header detection
-`igir` can detect headers for the following consoles and file extensions:
+Igir can detect headers for the following consoles and file extensions:
| Console | Header | Extension |
|--------------------------------|---------------|-----------|
@@ -16,13 +16,13 @@ Some of these headers are used to tell the emulator information about how to emu
| Nintendo - Famicom Disk System | fsNES/FDS | `.fds` |
| Nintendo - SNES | SMC | `.smc` |
-Those file extensions above are the commonly accepted "correct" extensions and `igir` will attempt to detect if a header is present in those ROM files automatically. If for some reason your files don't have the right extension (e.g. `.rom`) you can force header detection with the `--header` glob option:
+Those file extensions above are the commonly accepted "correct" extensions, and Igir will attempt to detect if a header is present in those ROM files automatically. If for some reason your files don't have the right extension (e.g. `.rom`) you can force header detection with the `--header` glob option:
```shell
igir [commands..] --dat --input --header "*.rom"
```
-`igir` will use this detected header information to compute both "headered" and "headerless" checksums of ROMs and use both of those to match against DAT files.
+Igir will use this detected header information to compute both "headered" and "headerless" checksums of ROMs and use both of those to match against DAT files.
!!! warning
@@ -30,7 +30,7 @@ igir [commands..] --dat --input --header "*.rom"
## Manual header removal
-Some emulators cannot parse ROMs with headers and instead need a "headerless" version. This seems to be most common with SNES. Sometimes "headerless" files will have a different file extension:
+Some emulators cannot parse ROMs with headers and instead need a "headerless" version. This seems most common with SNES. Sometimes "headerless" files will have a different file extension:
| Console | Header | Headered Extension | Headerless Extension |
|--------------------------------|---------------|------------------------|--------------------------|
@@ -40,7 +40,7 @@ Some emulators cannot parse ROMs with headers and instead need a "headerless" ve
| Nintendo - Famicom Disk System | fsNES/FDS | `.fds` | N/A |
| Nintendo - SNES | SMC | `.smc` | `.sfc` |
-For every console that `igir` can understand the headers for, it can also remove them with the `--remove-headers` option. This only makes sense for the consoles above with different "headerless" extensions, so you have to specify the extensions like this:
+For every console that Igir can understand the headers for, it can also remove them with the `--remove-headers` option. This only makes sense for the consoles above with different "headerless" extensions, so you have to specify the extensions like this:
```shell
igir [commands..] --dat --input --remove-headers .lnx,.smc
@@ -54,6 +54,6 @@ igir [commands..] --dat --input --remove-headers
## Automatic header removal
-Some DAT groups such as No-Intro publish "headered" and "headerless" DATs for the same console, such as NES. `igir` will treat these DATs differently, it will automatically remove headers (if present) for "headerless" DATs, and leave the header intact for "headered" DATs (ignoring the `--remove-headers` option completely).
+Some DAT groups such as No-Intro publish "headered" and "headerless" DATs for the same console, such as NES. Igir will treat these DATs differently, it will automatically remove headers (if present) for "headerless" DATs, and leave the header intact for "headered" DATs (ignoring the `--remove-headers` option completely).
As explained above, you almost always want the "headered" version. It's only in very specific circumstances that you might need the "headerless" version.
diff --git a/docs/roms/matching.md b/docs/roms/matching.md
index f2f5d69b0..6a8445d91 100644
--- a/docs/roms/matching.md
+++ b/docs/roms/matching.md
@@ -1,8 +1,8 @@
# ROM Matching
-When `igir` [scans ROM files](../input/file-scanning.md) in the input directory, it calculates a number of checksums to uniquely identify each file. These checksums are then matched to ones found in [DATs](../dats/introduction.md).
+When Igir [scans ROM files](../input/file-scanning.md) in the input directory, it calculates a number of checksums to uniquely identify each file. These checksums are then matched to ones found in [DATs](../dats/introduction.md).
-By default, `igir` will use CRC32 + filesize to match input files to ROMs found in DATs. CRC32 checksums are fast to calculate, and many [archive formats](../input/reading-archives.md) include them in their directory of files, which greatly speeds up scanning.
+By default, Igir will use CRC32 + filesize to match input files to ROMs found in DATs. CRC32 checksums are fast to calculate, and many [archive formats](../input/reading-archives.md) include them in their directory of files, which greatly speeds up scanning.
!!! note
@@ -10,13 +10,15 @@ By default, `igir` will use CRC32 + filesize to match input files to ROMs found
## Automatically using other checksum algorithms
-Some DAT release groups do not include every checksum for every file. For example, MAME CHDs only include SHA1 checksums and nothing else, not even filesize information.
+Some DAT release groups do not include every checksum for every file. For example, CHDs in MAME DATs only include SHA1 checksums and nothing else, not even filesize information.
And some DAT release groups do not include filesize information for every file, preventing a safe use of CRC32. For example, not every [Hardware Target Game Database SMDB](https://github.com/frederic-mahe/Hardware-Target-Game-Database/tree/master/EverDrive%20Pack%20SMDBs) includes file sizes, but they typically include all the normal checksums.
-!!! success
+!!! warning
- For situations like these, `igir` will automatically detect what combination of checksums it needs to calculate for input files to be able to match them to DATs. This has the chance of greatly slowing down file scanning, especially with archives.
+ For situations like these, Igir will automatically detect what combination of checksums it needs to calculate for input files to be able to match them to DATs. This _does_ have the chance of greatly slowing down file scanning, especially with archives.
+
+ To constrain what checksums are calculated, you can use the `--input-checksum-quick` option (below), or `--input-checksum-max ` which accepts the same algorithm options as `--input-checksum-min ` (also below).
For example, if you provide all of these DATs at once with the [`--dat ` option](../dats/processing.md):
@@ -24,11 +26,25 @@ For example, if you provide all of these DATs at once with the [`--dat ` o
- Hardware Target Game Database's Atari Lynx SMBD (which includes CRC32, MD5, SHA1, and SHA256 information but _not_ filesize)
- MAME ListXML (which only includes SHA1 information for CHD "disks")
-...then `igir` will determine that SHA1 is necessary to calculate because not every ROM in every DAT includes CRC32 _and_ filesize information.
+...then Igir will determine that SHA1 is the minimum necessary checksum to calculate because not every ROM in every DAT includes CRC32 _and_ filesize information.
!!! note
- When generating a [dir2dat](../dats/dir2dat.md) with the `igir dir2dat` command, `igir` will calculate CRC32, MD5, and SHA1 information for every file. This helps ensure that the generated DAT has the most complete information it can. You can additionally add SHA256 information with the option `igir [commands..] [options] --input-min-checksum SHA256` (below).
+ When generating a [dir2dat](../dats/dir2dat.md) with the `igir dir2dat` command, Igir will calculate CRC32, MD5, and SHA1 information for every file. This helps ensure that the generated DAT has the most complete information it can. You can additionally add SHA256 information with the option `igir [commands..] [options] --input-checksum-min SHA256` (below).
+
+## Quick scanning files
+
+A number of archives formats require the extraction of files to calculate their checksums, and this extraction can greatly increase scanning time and add hard drive wear & tear. Igir's default settings will give you the best chance of matching input files to DATs, but there may be situations where you want to make scanning faster.
+
+The `--input-checksum-quick` option will prevent the extraction of archives (either in memory _or_ using temporary files) to calculate checksums of files contained inside. This means that Igir will rely solely on the information available in the archive's file directory. Non-archive files will still have their checksum calculated as normal. See the [archive formats](../input/reading-archives.md) page for more information about what file types contain what checksum information.
+
+!!! warning
+
+ If an archive format doesn't contain any checksum information (e.g. `.cso`, `.tar.gz`), then there will be no way to match those input files to DATs when quick scanning! Only use quick scanning when all input archives store checksums of their files!
+
+!!! warning
+
+ Different DAT groups catalog CHDs of CD-ROMs (`.bin` & `.cue`) and GD-ROMs (`.gdi` & `.bin`/`.raw`) that use a track sheet plus one or more track files differnetly. Take the Sega Dreamcast for example, Redump catalogs `.bin` & `.cue` files (which is [problematic with CHDs](https://github.com/mamedev/mame/issues/11903)), [MAME Redump](https://github.com/MetalSlug/MAMERedump) catalogs `.chd` CD files, and TOSEC catalogs `.gdi` & `.bin`/`.raw` files. Quick scanning of CHDs means only the SHA1 stored in its header will be used for matching, which may or may not work depending on the DATs you use.
## Manually using other checksum algorithms
@@ -36,17 +52,17 @@ For example, if you provide all of these DATs at once with the [`--dat ` o
Most people do not need to calculate checksums above CRC32. CRC32 + filesize is sufficient to match ROMs and test written files in the gross majority of cases. The below information is for people that _truly_ know they need higher checksums.
-You can specify higher checksum algorithms with the `--input-min-checksum ` option like this:
+You can specify higher checksum algorithms with the `--input-checksum-min ` option like this:
```shell
-igir [commands..] [options] --input-min-checksum MD5
-igir [commands..] [options] --input-min-checksum SHA1
-igir [commands..] [options] --input-min-checksum SHA256
+igir [commands..] [options] --input-checksum-min MD5
+igir [commands..] [options] --input-checksum-min SHA1
+igir [commands..] [options] --input-checksum-min SHA256
```
-This option defines the _minimum_ checksum that will be used based on digest size (below). If not every ROM in every DAT provides the checksum you specify, `igir` may automatically calculate and match files based on a higher checksum (see above).
+This option defines the _minimum_ checksum that will be used based on digest size (below). If not every ROM in every DAT provides the checksum you specify, Igir may automatically calculate and match files based on a higher checksum (see above), but never lower.
-The reason you might want to do this is to have a higher confidence that found files _exactly_ match ROMs in DATs. Just keep in mind that explicitly enabling non-CRC32 checksums will _greatly_ slow down scanning of files within archives.
+The reason you might want to do this is to have a higher confidence that found files _exactly_ match ROMs in DATs. Keep in mind that explicitly enabling non-CRC32 checksums will _greatly_ slow down scanning of files within archives (see "quick scanning" above).
Here is a table that shows the keyspace for each checksum algorithm, where the higher number of bits reduces the chances of collisions:
@@ -57,4 +73,4 @@ Here is a table that shows the keyspace for each checksum algorithm, where the h
| SHA1 | 160 bits | 2^160 = 1.46 quindecillion | `666d29a15d92f62750dd665a06ce01fbd09eb98a` |
| SHA256 | 256 bits | 2^256 = 115.79 quattuorvigintillion | `1934e26cf69aa49978baac893ad5a890af35bdfb2c7a9393745f14dc89459137` |
-When files are [tested](../commands.md#test) after being written, `igir` will use the highest checksum available from the scanned file to check the written file. This lets you have equal confidence that a file was written correctly as well as matched correctly.
+When files are [tested](../commands.md#test) after being written, Igir will use the highest checksum available from the scanned file to check the written file. This lets you have equal confidence that a file was written correctly as well as matched correctly.
diff --git a/docs/roms/patching.md b/docs/roms/patching.md
index c0cee1a6d..7f9f6376a 100644
--- a/docs/roms/patching.md
+++ b/docs/roms/patching.md
@@ -2,7 +2,7 @@
Patches contain a set of changes that can be applied to a file, turning that file into something different. Common examples for patching ROMs are: translating text to a different language but keeping game logic the same, and fan-made creations such as new levels for an existing game.
-Games and their ROMs are protected under copyrights, so patches are used in order to not share copyrighted code online. A person needs the original ROM file plus a patch file in order to get the resulting patched ROM that will be played with an emulator.
+Games and their ROMs are protected under copyrights, so patches are used to not share copyrighted code online. A person needs the original ROM file plus a patch file to get the resulting patched ROM that will be played with an emulator.
## Specifying patch files
@@ -12,7 +12,7 @@ Patch files can be specified with the `--patch ` option. See the [file sca
There are many, _many_ patch types that ROM hackers use to distribute their changes on the internet ([xkcd "Standards"](https://xkcd.com/927/)). Typically, a patch will only be distributed in one format, so gamers are entirely at the mercy of the ROM hacker's choice.
-Not all patch types are created equal. Here are some tables of some existing formats, whether `igir` supports them, and what the patch supports.
+Not all patch types are created equal. Here are some tables of some existing formats, whether Igir supports them, and what the patch supports.
**Common patch types:**
@@ -46,7 +46,7 @@ If you have a choice in patch format, choose one that contains CRC32 checksums i
## ROM checksums
-`igir` needs to be able to know what source ROM each patch file applies to, and it does this using CRC32 checksums.
+Igir needs to be able to know what source ROM each patch file applies to, and it does this using CRC32 checksums.
A few patch formats include the source ROM's CRC32 checksum in the patch's file contents. This is the most accurate and therefore the best way to get source ROM information. `.bps` is a great example of an efficient and simple patch format that includes this information.
diff --git a/docs/static b/docs/static
new file mode 120000
index 000000000..4dab1644d
--- /dev/null
+++ b/docs/static
@@ -0,0 +1 @@
+../static
\ No newline at end of file
diff --git a/docs/usage/arcade.md b/docs/usage/arcade.md
index db1a56181..d5f4f76b9 100644
--- a/docs/usage/arcade.md
+++ b/docs/usage/arcade.md
@@ -31,7 +31,7 @@ Here is a chart of instructions for various setups:
| [FinalBurn Neo](https://github.com/finalburnneo/FBNeo) | FinalBurn Neo doesn't provide an obvious way to find the correct DAT for each version. But it is likely that you are using FinalBurn Neo through a frontend, so use the above instructions. | N/A |
| [FinalBurn Alpha](https://www.fbalpha.com/) | FinalBurn Alpha was forked into FinalBurn Neo, so you should use that if possible. Otherwise, hopefully your frontend's documentation has links to download the correct DAT. | N/A |
-## ROM set types
+## ROM set merge types
There are three broadly accepted types of ROM sets, with one extra variation, resulting in four types.
@@ -82,6 +82,12 @@ The ROM merge type can be specified with the `--merge-roms ` option:
--merge-roms split
```
+## CHD disks
+
+As arcade machines got more complicated, their storage requirements grew beyond what ROM chips can handle cost effectively. Cabinets started embedding hard drives, optical drives, laser disc drives, and more. Because backup images of these media types can get large, the MAME developers created a new compression format called "compressed hunks of data" (CHD).
+
+MAME DATs catalog these "disks" separately from "ROMs," which lets users choose whether to care about them or not. Typically, games that require disks will not run without them, so Igir requires them for a game to be considered present/complete. You can use the `--exclude-disks` option to exclude disks and only process ROMs to save some space.
+
## Example: building a new ROM set
Let's say we want to build an arcade ROM set that's compatible with the most recent version of [RetroArch](desktop/retroarch.md). The steps would look like this:
@@ -106,7 +112,7 @@ Let's say we want to build an arcade ROM set that's compatible with the most rec
Let's say we care first and foremost that the arcade games "just work," and then we would like to conserve disk space. A "split" ROM set makes a good choice because RetroArch should be able to automatically index every game, including both parents and clones.
-6. **Run `igir`.**
+6. **Run Igir.**
!!! note
@@ -160,11 +166,11 @@ Most other ROM managers use the terms "re-build" & "fix" when talking about taki
!!! note
- A game's required ROM files may change between emulator versions. This usually occurs when bad ROM dumps are replaced with better dumps. `igir` cannot magically deal with these ROM differences, and `igir` will only write complete ROM sets, so you may see games disappear when re-building. You will need to source the differing ROM files in order to keep your full game set.
+ A game's required ROM files may change between emulator versions. This usually occurs when bad ROM dumps are replaced with better dumps. Igir cannot magically deal with these ROM differences, and Igir will only write complete ROM sets, so you may see games disappear when re-building. You will need to source the differing ROM files in order to keep your full game set.
-A major reason `igir` was created was to help disambiguate what it means to build & re-build ROM sets. `igir` explicitly requires users to choose whether ROM files are copied or moved, so that users know what decision they are making. To "re-build" a ROM set, a user just needs to `igir move` ROMs from an input directory to the same directory specified again as the output.
+A major reason Igir was created was to help disambiguate what it means to build & re-build ROM sets. Igir explicitly requires users to choose whether ROM files are copied or moved, so that users know what decision theyโre making. To "re-build" a ROM set, a user just needs to `igir move` ROMs from an input directory to the same directory specified again as the output.
-Taking the MAME v0.258 set we created above, let's say we want to "downgrade" it to MAME 2003 (v0.78) because an under-powered device requires it. The steps would look like this:
+Taking the MAME v0.258 set we created above, let's say we want to "downgrade" it to MAME 2003 (v0.78) because an underpowered device requires it. The steps would look like this:
1. **Locate or download the emulator version's DAT.**
@@ -176,7 +182,7 @@ Taking the MAME v0.258 set we created above, let's say we want to "downgrade" it
This is left as an exercise for the reader.
-3. **Run `igir`.**
+3. **Run Igir.**
=== ":simple-windowsxp: Windows (64-bit)"
diff --git a/docs/usage/collection-sorting.md b/docs/usage/basic.md
similarity index 53%
rename from docs/usage/collection-sorting.md
rename to docs/usage/basic.md
index 39207cd95..e0593648d 100644
--- a/docs/usage/collection-sorting.md
+++ b/docs/usage/basic.md
@@ -1,4 +1,4 @@
-# Example Collection Sorting
+# Basic Usage Examples
A walkthrough of an example way to sort your ROM collection.
@@ -6,11 +6,15 @@ A walkthrough of an example way to sort your ROM collection.
See the `igir --help` message for a few common examples.
-## First time collection sort
+## With DATs
+
+Even though Igir can work without [DATs](../dats/introduction.md), using DATs to sort your collection is the [best practice](best-practices.md) to end up with the most accurate and organized set of ROMs.
+
+### First time collection sort
First, you need to download a set of [DATs](../dats/introduction.md). For these examples I'll assume you downloaded a No-Intro daily P/C XML `.zip`.
-Let's say that you have a directory named `ROMs/` that contains ROMs for many different systems, and it needs some organization. To make sure we're alright with the output, we'll have `igir` copy these files rather than move them. We'll also zip them to reduce disk space & speed up future scans.
+Let's say that you have a directory named `ROMs/` that contains ROMs for many different systems, and it needs some organization. To make sure we're alright with the output, we'll have Igir copy these files to a different directory rather than move them. We'll also [zip](../output/writing-archives.md) them to reduce disk space & speed up future scans.
=== ":simple-windowsxp: Windows"
@@ -42,12 +46,12 @@ Let's say that you have a directory named `ROMs/` that contains ROMs for many di
--dir-dat-name
```
-This will organize your ROMs into system-specific subdirectories within `ROMs-Sorted/` and name all of your ROMs accurately. Because we copied and didn't move, no files were deleted from the `ROMs/` input directory.
+This will organize your ROMs into system-specific subdirectories within `ROMs-Sorted/` and name all of your ROMs according to the No-Intro DATs. Because we copied and didn't move the files, no files were deleted from the `ROMs/` input directory.
`ROMs-Sorted/` then might look something like this:
```text
-ROMs-Sorted
+ROMs-Sorted/
โโโ Nintendo - Game Boy
โ โโโ Pokemon - Blue Version (USA, Europe) (SGB Enhanced).zip
โ โโโ Pokemon - Yellow Version - Special Pikachu Edition (USA, Europe) (CGB+SGB Enhanced).zip
@@ -59,13 +63,17 @@ ROMs-Sorted
โโโ Pokemon Pinball (USA, Australia) (Rumble Version) (SGB Enhanced) (GB Compatible).zip
```
-[](https://asciinema.org/a/rOWJwgbbODaXuQeQY4B6uWc4i)
+
+
+!!! info
+
+ See the [output path options](../output/path-options.md) and [output path tokens](../output/tokens.md) pages for other ways that you can organize your collection.
-## Subsequent collection sorts
+### Subsequent collection sorts
-Let's say that we've done the above first time sort and were happy with the results. We can now consider the `ROMs-Sorted/` directory to be our primary collection, every file in there has been matched to a DAT.
+Let's say that we've done the above first time sort and were happy with the results. We can now consider the `ROMs-Sorted/` directory to be our "golden" or "primary" collection, as every file in there has been matched to a DAT.
-Now we have new ROMs that we want to merge into our collection, and we want to generate a [report](../output/reporting.md) of what ROMs are still missing. We also want to delete any unknown files that may have made their way into our collection.
+We now have new ROMs that we want to newly merge into our collection, and we want to generate a [report](../output/reporting.md) of what ROMs are still missing. We also want to "[clean](../output/cleaning.md)" or delete any unknown files that may have made their way into our collection.
=== ":simple-windowsxp: Windows"
@@ -102,10 +110,16 @@ Now we have new ROMs that we want to merge into our collection, and we want to g
Any new ROMs in `ROMs-New/` that we didn't already have in `ROMs-Sorted/` will be moved, and a report will be generated for us.
+!!! note
+
+ Note that we're using `ROMs-Sorted/` as both an input directory _and_ as the output directory. This is required to ensure the [`clean` command](../output/cleaning.md) doesn't delete "good" files already in the output directory!
+
+ You can always use the [`--clean-dry-run` option](../output/cleaning.md#dry-run) to see what files would be deleted without actually deleting them.
+
`ROMs-Sorted/` then might look something like this, with new ROMs added:
```text
-ROMs-Sorted
+ROMs-Sorted/
โโโ Nintendo - Game Boy
โ โโโ Pokemon - Blue Version (USA, Europe) (SGB Enhanced).zip
โ โโโ Pokemon - Red Version (USA, Europe) (SGB Enhanced).zip
@@ -119,13 +133,13 @@ ROMs-Sorted
โโโ Pokemon Pinball (USA, Australia) (Rumble Version) (SGB Enhanced) (GB Compatible).zip
```
-[](https://asciinema.org/a/PWAfBcvCikzJ7wObLcdFGtZbI)
+
-## Flash cart 1G1R
+### Flash cart 1G1R
Let's say we've done the above sorting we want to copy some ROMs from `ROMs-Sorted/` to a flash cart.
-We would prefer having only one copy of every game (1G1R), so there is less to scroll through to find what we want, and because we have a preferred language. Our flash cart can't read `.zip` files, so we'll need to extract our ROMs during copying.
+We would prefer having only one copy of every game ([1G1R](../roms/filtering-preferences.md#preferences-for-1g1r)), because we have a preferred language, and so there is less to scroll through to find what game we want. Our flash cart can't read `.zip` files, so we'll need to extract our ROMs during copying.
=== ":simple-windowsxp: Windows"
@@ -185,8 +199,85 @@ Your flash cart might then look something like this:
โโโ Pokemon - Yellow Version - Special Pikachu Edition (USA, Europe) (CGB+SGB Enhanced).gb
```
-[](https://asciinema.org/a/K8ROFbX8c4NJfUue3lwbe7d8V)
+
!!! info
See the [ROM filtering & preference](../roms/filtering-preferences.md) page for other ways that you can filter your collection.
+
+## Without DATs
+
+ROM organization is very opinion-based, and your opinion may not match that of DAT groups. To preserve your custom ROM sorting, you can skip providing any DATs by omitting the `--dat ` option.
+
+!!! note
+
+ If your custom ROM sorting includes directories, you will want to provide the [`--dir-mirror` option](../output/path-options.md#mirror-the-input-subdirectory) to preserve the structure.
+
+### Extracting or zipping all ROMs
+
+It is possible to extract or zip your ROM files en masse without complicated Bash or Batch scripts, and you can do this without DATs because the root of the filename won't change.
+
+=== ":simple-windowsxp: Windows"
+
+ ```batch
+ igir move extract test ^
+ --input "ROMs\" ^
+ --output "ROMs\" ^
+ --dir-mirror
+ ```
+
+=== ":simple-apple: macOS"
+
+ ```shell
+ igir move extract test \
+ --input "ROMs/" \
+ --output "ROMs/" \
+ --dir-mirror
+ ```
+
+=== ":simple-linux: Linux"
+
+ ```shell
+ igir move extract test \
+ --input "ROMs/" \
+ --output "ROMs/" \
+ --dir-mirror
+ ```
+
+
+
+### Fixing file extensions
+
+Igir is able to detect more than 50 ROM and archive file types and automatically correct file extensions when needed during writing. See the [writing options](../output/options.md#fixing-rom-extensions) page for more information.
+
+=== ":simple-windowsxp: Windows"
+
+ ```batch
+ igir move extract test ^
+ --input "ROMs\" ^
+ --output "ROMs\" ^
+ --dir-mirror ^
+ --fix-extension always
+ ```
+
+=== ":simple-apple: macOS"
+
+ ```shell
+ igir move extract test \
+ --input "ROMs/" \
+ --output "ROMs/" \
+ --dir-mirror \
+ --fix-extension always
+ ```
+
+=== ":simple-linux: Linux"
+
+ ```shell
+ igir move extract test \
+ --input "ROMs/" \
+ --output "ROMs/" \
+ --dir-mirror \
+ --fix-extension always
+ ```
+
+
diff --git a/docs/usage/best-practices.md b/docs/usage/best-practices.md
new file mode 100644
index 000000000..01591d4e9
--- /dev/null
+++ b/docs/usage/best-practices.md
@@ -0,0 +1,89 @@
+# Best Practices
+
+**Use an installation method that auto-updates.**
+
+Downloading bundled binaries from GitHub is the most difficult way to receive updates to Igir. See the [installation page](../installation.md) for options available to you.
+
+## DATs
+
+**Use DATs.**
+
+While [DATs](../dats/introduction.md) are optional, they allow you to organize your ROMs in a human-understandable manner while trimming out unknown files. Additional metadata provided by some DAT groups allows you [filter your ROM set](../roms/filtering-preferences.md) to only what you care about.
+
+**Choose DAT groups with parent/clone information.**
+
+[Parent/clone information](../dats/introduction.md#parentclone-pc-dats) lets you apply [1G1R preference rules](../roms/filtering-preferences.md). For example, prefer No-Intro's Game Boy DAT over TOSEC's, as TOSEC doesn't provide parent/clone information.
+
+**Use consistent versions across all devices.**
+
+DATs work best if you store them alongside your primary ROM collection, and when you use the same DAT versions across all devices (i.e. your primary collection, handhelds, flash carts, etc.). Some DAT groups release new versions as often as daily, so keeping your collection in sync is easier with consistent DATs.
+
+**Process DATs from different groups separately.**
+
+DAT groups have some overlap between them, so using DATs from multiple groups at the same time may cause duplicate files or filename collisions. Different groups also have different conventions that may require different settings, such as [filters](../roms/filtering-preferences.md#filters) and [1G1R preferences](../roms/filtering-preferences.md#preferences-for-1g1r).
+
+Also, keep ROM sets organized by DATs from different groups in separate directories. For example, create different directories for No-Intro, Redump, and TOSEC-organized ROM sets.
+
+## File Inputs
+
+**Keep one primary collection and then copy to other sub-collections.**
+
+Provide your output directory as one of the input directories, and then any other input directories you wish to copy or move into your primary collection. Doing so will let you [clean the output directory](../output/cleaning.md) safely.
+
+Then, create sub-collections by copying files from your main collection to other devices, optionally applying [filtering and preference rules](../roms/filtering-preferences.md).
+
+**Prefer ROMs with headers.**
+
+Igir can [remove headers automatically](../roms/headers.md#automatic-header-removal) when needed, but it cannot add them back. Keep ROMs with headers in your primary collection and then modify them when copying to other devices as needed.
+
+**Don't use quick scanning unless you absolutely need it.**
+
+The default settings for Igir will have the best chance for you to match input files to DATs. Using the [`--input-checksum-quick` option](../roms/matching.md#quick-scanning-files) will reduce those chances.
+
+**Don't increase the minimum checksum level unless you absolutely need it.**
+
+The default settings for Igir will cause accurate file matching for the gross majority of cases with the least amount of processing. Additionally, most [archive formats](../input/reading-archives.md) only store CRC32 checksums, so forcing any others will greatly increase scanning time. Use the `--input-checksum-min ` option with caution.
+
+## File Outputs
+
+**Zip ROMs wherever possible.**
+
+Zip files generally save file space and are faster to scan, at the expense of more time to create them. For collections that will be read from more often than written to, such as a primary collection, prefer to eat the cost of [archiving files](../output/writing-archives.md) once with the `igir zip` command.
+
+**Organize ROM sets by DAT name or description.**
+
+Ignoring [arcade ROM sets](../usage/arcade.md), one purpose of sorting your ROM collection using DATs is to organize them in some human-understandable manner. A common way to help with this is to group ROMs from the same console together using [`--dir-dat-name`](../output/path-options.md#append-dat-name) or [`--dir-dat-description`](../output/path-options.md#append-dat-description)`
+
+Alternatively, you can [filter to only the DATs](../dats/processing.md#dat-filtering) you want, and then [combine them together](../dats/processing.md#dat-combining) and write the resulting ROMs to one directory.
+
+**Organize ROMs by letter for non-keyboard & mouse devices.**
+
+Devices that only have a D-pad to browse through files can make ROM selection tedious. Use the [`--dir-letter` option](../output/path-options.md#append-game-letters) and its `--dir-letter-*` modifier options to make this easier with large collections.
+
+**Use the default game name appending option.**
+
+Igir will automatically group games with multiple ROMs together into their own subfolder. Leave this [`--dir-game-subdir ` option](../output/path-options.md#append-the-game-name) as the default unless you know what you're doing.
+
+**Overwrite invalid files.**
+
+If you value keeping a clean and accurate ROM collection, use the [`--overwrite-invalid` option](../output/options.md) to overwrite files in the output directory that don't match what's expected with a "valid" file.
+
+## Arcade
+
+**Use the right DAT version for your emulator version.**
+
+You must choose the right DAT for your emulator (e.g. MAME) and emulator version (e.g. MAME 0.258) or your ROMs may not work correctly. See the [arcade ROM sets page](../usage/arcade.md#emulator-versions--dats) for more information.
+
+**For MAME, use the official DATs or ones from progetto-SNAPS.**
+
+These DATs provide the most flexibility (i.e. can use any merge type) and the most amount of metadata (i.e. [parent/clone information](../dats/introduction.md#parentclone-pc-dats), ROMs and CHDs together in one DAT) for Igir to use for processing. Other DAT groups such as pleasuredome modify the official DATs quite heavily by pre-applying filters.
+
+**Pick a ROM merge type intentionally.**
+
+Igir will produce full non-merged sets by default for the highest level of compatability. However, you should understand the difference between the supported [merge types](../usage/arcade.md#rom-set-merge-types) and choose one that best suits your needs.
+
+## Advanced
+
+**Use an SSD or a RAM drive for the temp directory.**
+
+Igir sometimes needs to write files to a [temporary directory](../advanced/temp-dir.md), such as when extracting archives that it [can't read natively](../input/reading-archives.md). Using a fast hard drive for this directory can speed up processing.
diff --git a/docs/usage/console/gamecube.md b/docs/usage/console/gamecube.md
index 7f5ae90c2..c2f5d3fbe 100644
--- a/docs/usage/console/gamecube.md
+++ b/docs/usage/console/gamecube.md
@@ -8,17 +8,17 @@
Swiss is sensitive to files being fragmented on SD cards ([swiss-gc#763](https://github.com/emukidid/swiss-gc/issues/763), [swiss-gc#122](https://github.com/emukidid/swiss-gc/issues/122), etc.). This means that you should only write one ISO at a time!
-`igir` has a `--writer-threads` option to limit the number of files being written at once. You can use the option like this:
+Igir has a `--writer-threads ` option to limit the number of files being written at once. You can use the option like this:
=== ":simple-windowsxp: Windows"
Replace the `E:\` drive letter with wherever your SD card is:
```batch
- igir copy extract test clean ^
+ igir copy test clean ^
--dat "Redump*.zip" ^
--dat-name-regex '/gamecube/i' ^
- --input "ISOs" ^
+ --input "Games" ^
--output "E:\Games" ^
--dir-letter ^
--writer-threads 1
@@ -29,10 +29,10 @@
Replace the `/Volumes/SD2SP2` drive name with whatever your SD card is named:
```shell
- igir copy extract test clean \
+ igir copy test clean \
--dat "Redump*.zip" \
--dat-name-regex '/gamecube/i' \
- --input "ISOs/" \
+ --input "Games/" \
--output "/Volumes/SD2SP2/Games/" \
--dir-letter \
--writer-threads 1
@@ -43,11 +43,17 @@
Replace the `/media/SD2SP2` path with wherever your SD card is mounted:
```shell
- igir copy extract test clean \
+ igir copy test clean \
--dat "Redump*.zip" \
--dat-name-regex '/gamecube/i' \
- --input "ISOs/" \
+ --input "Games/" \
--output "/media/SD2SP2/Games/" \
--dir-letter \
--writer-threads 1
```
+
+## NKit
+
+Swiss supports ISOs in the trimmed [NKit format](https://wiki.gbatemp.net/wiki/NKit), which can save significant space on your SD card. Some games such as Animal Crossing can be compressed as small as 28MB, while other games such as Wave Race: Blue Storm don't compress much at all.
+
+Igir can read the original ISO's CRC32 information stored in `.nkit.iso` files, which means it can match files to DATs (as long as you don't raise the minimum checksum level!). However, Igir can't extract NKit ISOs, you'll need to use Nanook's [NKit tool](https://wiki.gbatemp.net/wiki/NKit#Download) instead.
diff --git a/docs/usage/console/ps2.md b/docs/usage/console/ps2.md
index 465807c72..66df8d3b5 100644
--- a/docs/usage/console/ps2.md
+++ b/docs/usage/console/ps2.md
@@ -8,7 +8,7 @@
OPL is sensitive to files being fragmented on USB drives and SD cards (MX4SIO/SIO2SD). This means that you should only write one ISO at a time!
-`igir` has a `--writer-threads` option to limit the number of files being written at once. You can use the option like this:
+Igir has a `--writer-threads ` option to limit the number of files being written at once. You can use the option like this:
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/desktop/batocera.md b/docs/usage/desktop/batocera.md
index 837d9accc..eb8f5f8ee 100644
--- a/docs/usage/desktop/batocera.md
+++ b/docs/usage/desktop/batocera.md
@@ -19,7 +19,7 @@ Because Batocera uses RetroArch under the hood, the instructions are generally t
## ROMs
-Batocera uses its own proprietary [ROM folder structure](https://wiki.batocera.org/systems), so `igir` has a replaceable `{batocera}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
+Batocera uses its own proprietary [ROM folder structure](https://wiki.batocera.org/systems), so Igir has a replaceable `{batocera}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-linux: Batocera (Linux)"
diff --git a/docs/usage/desktop/emuelec.md b/docs/usage/desktop/emuelec.md
index 73b335344..77efe28c8 100644
--- a/docs/usage/desktop/emuelec.md
+++ b/docs/usage/desktop/emuelec.md
@@ -21,4 +21,4 @@ Because EmuELEC is mostly Libretro under the hood, the instructions are generall
!!! failure
- EmuELEC uses its own proprietary [ROM folder structure](https://github.com/EmuELEC/EmuELEC/wiki/Supported-Platforms-And--Correct-Rom-Path). `igir` does not support this folder structure, yet.
+ EmuELEC uses its own proprietary [ROM folder structure](https://github.com/EmuELEC/EmuELEC/wiki/Supported-Platforms-And--Correct-Rom-Path). Igir does not support this folder structure, yet.
diff --git a/docs/usage/desktop/emulationstation.md b/docs/usage/desktop/emulationstation.md
index 74ef1f08c..4e664332f 100644
--- a/docs/usage/desktop/emulationstation.md
+++ b/docs/usage/desktop/emulationstation.md
@@ -23,7 +23,7 @@ Other emulators may use other names for their BIOS images but all reside in the
## ROMs
-EmulationStation uses its own proprietary ROM folder structure, so `igir` has a replaceable `{es}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
+EmulationStation uses its own proprietary ROM folder structure, so Igir has a replaceable `{es}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-linux: EmulationStation (Linux)"
diff --git a/docs/usage/desktop/launchbox.md b/docs/usage/desktop/launchbox.md
index 6d3559c35..5e4c82dd4 100644
--- a/docs/usage/desktop/launchbox.md
+++ b/docs/usage/desktop/launchbox.md
@@ -6,4 +6,4 @@ LaunchBox uses [RetroArch](retroarch.md) for its game emulation by default, as o
!!! failure
- LaunchBox has its own ROM importing mechanism that copies files to `\Games\*\*` in your install directory (so `%USERPROFILE%\LaunchBox\Games\*\*` by default). There _is_ a mechanism to scan for ROMs added to these folders manually, but they must be sorted into the correct "platform" folder. LaunchBox doesn't have documentation cataloging these "platform" folders, so `igir` does not currently support them.
+ LaunchBox has its own ROM importing mechanism that copies files to `\Games\*\*` in your install directory (so `%USERPROFILE%\LaunchBox\Games\*\*` by default). There _is_ a mechanism to scan for ROMs added to these folders manually, but they must be sorted into the correct "platform" folder. LaunchBox doesn't have documentation cataloging these "platform" folders, so Igir does not currently support them.
diff --git a/docs/usage/desktop/openemu.md b/docs/usage/desktop/openemu.md
index 53772984e..329fa6790 100644
--- a/docs/usage/desktop/openemu.md
+++ b/docs/usage/desktop/openemu.md
@@ -4,4 +4,4 @@
!!! failure
- OpenEmu has its own ROM importing mechanism that copies files to `~/Library/Application Support/OpenEmu/Game Library/roms` and adds them to a database. OpenEmu will _not_ automatically scan files you place into this folder, so `igir` is unable to help sort them.
+ OpenEmu has its own ROM importing mechanism that copies files to `~/Library/Application Support/OpenEmu/Game Library/roms` and adds them to a database. OpenEmu will _not_ automatically scan files you place into this folder, so Igir is unable to help sort them.
diff --git a/docs/usage/desktop/retroarch.md b/docs/usage/desktop/retroarch.md
index fbca65be1..cdbfeb1c1 100644
--- a/docs/usage/desktop/retroarch.md
+++ b/docs/usage/desktop/retroarch.md
@@ -10,7 +10,7 @@
First, RetroArch needs a number of [BIOS files](https://docs.libretro.com/library/bios/). Thankfully, the libretro team maintains a DAT of these "system" files, so we don't have to guess at the correct filenames.
-With `igir`'s support for [DAT URLs](../../dats/processing.md#scanning-for-dats) we don't even have to download the DAT! Locate your "System/BIOS" directory as configured in the RetroArch UI and use it as your output directory:
+With Igir's support for [DAT URLs](../../dats/processing.md#scanning-for-dats) we don't even have to download the DAT! Locate your "System/BIOS" directory as configured in the RetroArch UI and use it as your output directory:
=== ":simple-windowsxp: Windows (64-bit)"
diff --git a/docs/usage/desktop/retrodeck.md b/docs/usage/desktop/retrodeck.md
index 7d25d5b94..64ed47ff4 100644
--- a/docs/usage/desktop/retrodeck.md
+++ b/docs/usage/desktop/retrodeck.md
@@ -21,7 +21,7 @@ Other emulators may use other names for their BIOS images but all reside in the
## ROMs
-RetroDECK uses its own proprietary ROM folder structure, so `igir` has a replaceable `{retrodeck}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
+RetroDECK uses its own proprietary ROM folder structure, so Igir has a replaceable `{retrodeck}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-linux: RetroDECK (Linux)"
diff --git a/docs/usage/desktop/romm.md b/docs/usage/desktop/romm.md
index af4fc47a0..80d22fdd3 100644
--- a/docs/usage/desktop/romm.md
+++ b/docs/usage/desktop/romm.md
@@ -4,7 +4,7 @@
## ROMs
-RomM uses its own [proprietary ROM folder structure](https://github.com/rommapp/romm/wiki/Supported-Platforms), so `igir` has a replaceable `{romm}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
+RomM uses its own [proprietary ROM folder structure](https://github.com/rommapp/romm/wiki/Supported-Platforms), so Igir has a replaceable `{romm}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
You can run RomM using [Docker Compose](https://docs.docker.com/compose/). Create a file named `docker-compose.yml` with the following contents, but change all of the environment variables with the value of `CHANGEME!`:
diff --git a/docs/usage/handheld/adam.md b/docs/usage/handheld/adam.md
index 7cc1d442d..6ccd0c788 100644
--- a/docs/usage/handheld/adam.md
+++ b/docs/usage/handheld/adam.md
@@ -53,11 +53,11 @@ The Adam image does not come with BIOS files. Where you have to put which of you
## ROMs
-Adam supports many different ROM formats in subfolders of `ROMS` on the second SD card (TF2). An exhaustive list can be found in [their wiki](https://github.com/eduardofilo/RG350_adam_image/tree/master/data/local/home/.simplemenu/section_groups), where you can also find information about which ROMS are supported in compressed form. Most supported systems and their ROMS can be automatically sorted by `igir` using the `{adam}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
+Adam supports many different ROM formats in subfolders of `ROMS` on the second SD card (TF2). An exhaustive list can be found in [their wiki](https://github.com/eduardofilo/RG350_adam_image/tree/master/data/local/home/.simplemenu/section_groups), where you can also find information about which ROMS are supported in compressed form. Most supported systems and their ROMS can be automatically sorted by Igir using the `{adam}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
!!! tip
- Please note that sorting the supported Arcade machine releases (MAME, CPS, FBA) in a single pass is not supported by `igir` at this time. Try the [Arcade docs](../arcade.md) docs for help with this.
+ Please note that sorting the supported Arcade machine releases (MAME, CPS, FBA) in a single pass is not supported by Igir at this time. Try the [Arcade docs](../arcade.md) docs for help with this.
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/handheld/funkeyos.md b/docs/usage/handheld/funkeyos.md
index 02cef1539..12f04f321 100644
--- a/docs/usage/handheld/funkeyos.md
+++ b/docs/usage/handheld/funkeyos.md
@@ -17,7 +17,7 @@ To sum up the documentation, two files need to be copied:
## ROMs
-Funkey OS uses its own proprietary [ROM folder structure](https://github.com/FunKey-Project/FunKey-OS/tree/master/FunKey/board/funkey/rootfs-overlay/usr/games/collections) based in the root of the SD card, so `igir` has a replaceable `{funkeyos}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
+Funkey OS uses its own proprietary [ROM folder structure](https://github.com/FunKey-Project/FunKey-OS/tree/master/FunKey/board/funkey/rootfs-overlay/usr/games/collections) based in the root of the SD card, so Igir has a replaceable `{funkeyos}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/handheld/jelos.md b/docs/usage/handheld/jelos.md
index 26db832f9..36fde18e2 100644
--- a/docs/usage/handheld/jelos.md
+++ b/docs/usage/handheld/jelos.md
@@ -69,7 +69,7 @@ JELOS has its BIOS folder at `roms/bios/`, and it uses the RetroArch filenames.
## ROMs
-JELOS supports many many systems and ROM formats. Check sections under the `Systems` menu in the [JELOS Wiki](https://jelos.org/) for more precise instructions about the indivudual systems. Most supported systems and their ROMS can be automatically sorted by `igir` using the `{jelos}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
+JELOS supports many many systems and ROM formats. Check sections under the `Systems` menu in the [JELOS Wiki](https://jelos.org/) for more precise instructions about the indivudual systems. Most supported systems and their ROMS can be automatically sorted by Igir using the `{jelos}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/handheld/minui.md b/docs/usage/handheld/minui.md
index 5f2f9c1f6..4cbc21db3 100644
--- a/docs/usage/handheld/minui.md
+++ b/docs/usage/handheld/minui.md
@@ -25,9 +25,9 @@ Place these files under `/Bios//`:
## ROMs
-MinUI supports many many systems and ROM formats. Check the folders [here (base)](https://github.com/shauninman/MinUI/tree/main/skeleton/BASE/Roms) and [here (extras)](https://github.com/shauninman/MinUI/tree/main/skeleton/EXTRAS/Roms) for a comprehensive list about the indivudual systems. Most supported systems and their ROMS can be automatically sorted by `igir` using the `{minui}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
+MinUI supports many many systems and ROM formats. Check the folders [here (base)](https://github.com/shauninman/MinUI/tree/main/skeleton/BASE/Roms) and [here (extras)](https://github.com/shauninman/MinUI/tree/main/skeleton/EXTRAS/Roms) for a comprehensive list about the indivudual systems. Most supported systems and their ROMS can be automatically sorted by Igir using the `{minui}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
-MinUI uses the names unter /Roms on the SD card in a more creative way than most other frontends. The folder names consist of a *UI name* and a *PAK name*. The *UI name* is used as the name shown in the User interface as a list item name, while the *PAK name* controls which software pack is used to open the files within. Files with the same *UI name* but different *PAK name* are listed in the same list in the UI but are opened with different PAKs. `igir` uses the vendor recommendations for the folder names with some exceptions.
+MinUI uses the names unter /Roms on the SD card in a more creative way than most other frontends. The folder names consist of a *UI name* and a *PAK name*. The *UI name* is used as the name shown in the User interface as a list item name, while the *PAK name* controls which software pack is used to open the files within. Files with the same *UI name* but different *PAK name* are listed in the same list in the UI but are opened with different PAKs. Igir uses the vendor recommendations for the folder names with some exceptions.
MinUI requires multi-file releases to be grouped into subdirectories (bin/cue releases of the PS1 for example). It is recommended to use the [`--dir-game-subdir multiple` option](../../output/path-options.md), which is the default at this time.
diff --git a/docs/usage/handheld/miyoocfw.md b/docs/usage/handheld/miyoocfw.md
index 4749c36bf..28fc69854 100644
--- a/docs/usage/handheld/miyoocfw.md
+++ b/docs/usage/handheld/miyoocfw.md
@@ -25,7 +25,7 @@ MiyooCFW doesn't seem to have a centralized folder for putting BIOS files so it'
## ROMs
-MiyooCFW supports many many systems and ROM formats. Check the table on the [MiyooCFW Wiki](https://github.com/TriForceX/MiyooCFW/wiki/Emulator-Info) for more precise instructions about the indivudual systems. Most supported systems and their ROMS can be automatically sorted by `igir` using the `{miyoocfw}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
+MiyooCFW supports many many systems and ROM formats. Check the table on the [MiyooCFW Wiki](https://github.com/TriForceX/MiyooCFW/wiki/Emulator-Info) for more precise instructions about the indivudual systems. Most supported systems and their ROMS can be automatically sorted by Igir using the `{miyoocfw}` output token. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/handheld/onionos.md b/docs/usage/handheld/onionos.md
index 29e289497..971b63e14 100644
--- a/docs/usage/handheld/onionos.md
+++ b/docs/usage/handheld/onionos.md
@@ -45,7 +45,7 @@ OnionOS has its BIOS folder at the root of the SD card at `/BIOS/`, and it uses
## ROMs
-OnionOS uses its own proprietary [ROM folder structure](https://github.com/OnionUI/Onion/wiki/Emulators#rom-folders---quick-reference), so `igir` has a replaceable `{onion}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
+OnionOS uses its own proprietary [ROM folder structure](https://github.com/OnionUI/Onion/wiki/Emulators#rom-folders---quick-reference), so Igir has a replaceable `{onion}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/handheld/twmenu.md b/docs/usage/handheld/twmenu.md
index a99fe148e..5c7cdaa81 100644
--- a/docs/usage/handheld/twmenu.md
+++ b/docs/usage/handheld/twmenu.md
@@ -12,7 +12,7 @@ TWiLightMenu++ ships with most emulators not needing BIOS files. No exceptions a
## ROMs
-TWiLightMenu uses its own proprietary [ROM folder structure](https://github.com/DS-Homebrew/TWiLightMenu/tree/master/7zfile/roms) based in the root of the SD card, so `igir` has a replaceable `{twmenu}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
+TWiLightMenu uses its own proprietary [ROM folder structure](https://github.com/DS-Homebrew/TWiLightMenu/tree/master/7zfile/roms) based in the root of the SD card, so Igir has a replaceable `{twmenu}` token to sort ROMs into the right place. See the [replaceable tokens page](../../output/tokens.md) for more information.
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/hardware/analogue-pocket.md b/docs/usage/hardware/analogue-pocket.md
index a2a652de7..78ab43e2e 100644
--- a/docs/usage/hardware/analogue-pocket.md
+++ b/docs/usage/hardware/analogue-pocket.md
@@ -14,7 +14,7 @@ Most Pocket updater utilities will download BIOS files required for each core fo
## ROMs
-`igir` has support for replaceable "tokens" in the `--output ` option. This makes it easier to sort ROMs on devices that have an expected directory structure. The `{pocket}` token exists to help sort ROMs on the Analogue pocket. See the [replaceable tokens page](../../output/tokens.md) for more information.
+Igir has support for replaceable "tokens" in the `--output ` option. This makes it easier to sort ROMs on devices that have an expected directory structure. The `{pocket}` token exists to help sort ROMs on the Analogue pocket. See the [replaceable tokens page](../../output/tokens.md) for more information.
This token can be used to reference each core's specific directory in the SD card's `Assets` directory. ROMs go in the `Assets/{pocket}/common` directory.
diff --git a/docs/usage/hardware/everdrive.md b/docs/usage/hardware/everdrive.md
index 20b05f603..ef40babbb 100644
--- a/docs/usage/hardware/everdrive.md
+++ b/docs/usage/hardware/everdrive.md
@@ -4,7 +4,7 @@ The [EverDrive](https://krikzz.com/) flash carts by Krikzz are some of the highe
## ROMs
-Because flash carts are specific to a specific console, you can provide specific input directories and [DATs](../../dats/introduction.md) when you run `igir`. For example:
+Because flash carts are specific to a specific console, you can provide specific input directories and [DATs](../../dats/introduction.md) when you run Igir. For example:
=== ":simple-windowsxp: Windows"
@@ -44,7 +44,7 @@ Because flash carts are specific to a specific console, you can provide specific
you can then add some other output options such as the [`--dir-letter` option](../../output/path-options.md), if desired.
-Alternatively, `igir` supports [Hardware Target Game Database SMDB files](https://github.com/frederic-mahe/Hardware-Target-Game-Database/tree/master/EverDrive%20Pack%20SMDBs) as [DATs](../../dats/introduction.md). Unlike typical DATs, Hardware Target Game Database SMDBs typically have an opinionated directory structure to help sort ROMs by language, category, genre, and more. Example usage:
+Alternatively, Igir supports [Hardware Target Game Database SMDB files](https://github.com/frederic-mahe/Hardware-Target-Game-Database/tree/master/EverDrive%20Pack%20SMDBs) as [DATs](../../dats/introduction.md). Unlike typical DATs, Hardware Target Game Database SMDBs typically have an opinionated directory structure to help sort ROMs by language, category, genre, and more. Example usage:
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/hardware/ezflash.md b/docs/usage/hardware/ezflash.md
index 7c85bba41..cc1dea387 100644
--- a/docs/usage/hardware/ezflash.md
+++ b/docs/usage/hardware/ezflash.md
@@ -4,7 +4,7 @@ The [EZ-FLASH](https://www.ezflash.cn/) flash carts for Nintendo handhelds are a
## ROMs
-Because flash carts are specific to a specific console, you can provide specific input directories & [DATs](../../dats/introduction.md) when you run `igir`. For example:
+Because flash carts are specific to a specific console, you can provide specific input directories & [DATs](../../dats/introduction.md) when you run Igir. For example:
=== ":simple-windowsxp: Windows"
diff --git a/docs/usage/hardware/mister.md b/docs/usage/hardware/mister.md
index 5efa4a550..ecf18514c 100644
--- a/docs/usage/hardware/mister.md
+++ b/docs/usage/hardware/mister.md
@@ -8,7 +8,7 @@ The MiSTer [`update_all.sh`](https://github.com/theypsilon/Update_All_MiSTer) sc
## ROMs
-`igir` has support for replaceable "tokens" in the `--output ` option. This makes it easier to sort ROMs on devices that have an expected directory structure. The `{mister}` token exists to help sort ROMs on the MiSTer. See the [replaceable tokens page](../../output/tokens.md) for more information.
+Igir has support for replaceable "tokens" in the `--output ` option. This makes it easier to sort ROMs on devices that have an expected directory structure. The `{mister}` token exists to help sort ROMs on the MiSTer. See the [replaceable tokens page](../../output/tokens.md) for more information.
This token can be used to reference each core's specific directory in the MiSTer's `games` directory.
diff --git a/docs/usage/personal.md b/docs/usage/personal.md
index aaa490db5..2ba9eda4b 100644
--- a/docs/usage/personal.md
+++ b/docs/usage/personal.md
@@ -1,6 +1,6 @@
-# Creator's Usage
+# Maintainer's Usage Example
-`igir` has many options available to fit almost any use case, but the number of options can be overwhelming. So that begs a question: _how do I, the creator of `igir`, use `igir` in the real world?_
+Igir has many options available to fit almost any use case, but the number of options can be overwhelming. So that begs a question: _how do I, the maintainer of Igir, use Igir in the real world?_
## Primary ROM library
@@ -58,22 +58,60 @@ for INPUT in "$@"; do
INPUTS+=(--input "${INPUT}")
done
+# Cartridge-based consoles, 1st-5th generations
npx --yes igir@latest move zip test clean report \
--dat "./No-Intro*.zip" \
- --dat-name-regex-exclude "/encrypted/i" \
+ --dat-name-regex-exclude "/encrypted|source code/i" \
--input "./No-Intro/" \
- "${INPUTS[@]}" \
+ "${INPUTS[@]:-}" \
+ `# Trust checksums in archive headers, don't checksum archives (we only care about the contents)` \
+ --input-checksum-max CRC32 \
+ --input-checksum-archives never \
--patch "./Patches/" \
--output "./No-Intro/" \
--dir-dat-name \
- --overwrite-invalid
+ --overwrite-invalid \
+ --zip-exclude "*.{chd,iso}" \
+ --reader-threads 4 \
+ -v
-npx --yes igir@latest move zip test \
+# Disc-based consoles, 4th+ generations
+npx --yes igir@latest move test clean report \
--dat "./Redump*.zip" \
+ --dat-name-regex-exclude "/Dreamcast/i" \
--input "./Redump/" \
"${INPUTS[@]}" \
+ `# Let maxcso calculate CSO CRC32s, don't checksum compressed discs (we only care about the contents)` \
+ --input-checksum-max CRC32 \
+ --input-checksum-archives never \
+ --patch "./Patches/" \
--output "./Redump/" \
- --dir-dat-name
+ --dir-dat-name \
+ --overwrite-invalid \
+ --only-retail \
+ --single \
+ --prefer-language EN \
+ --prefer-region USA,WORLD,EUR,JPN \
+ --prefer-revision newer \
+ -v
+
+# Dreamcast (because TOSEC catalogs chdman-compatible .gdi files and Redump catalogs .bin/.cue)
+npx --yes igir@latest move test clean report \
+ --dat "./TOSEC*.zip" \
+ --dat-name-regex "/Dreamcast/i" \
+ --dat-combine \
+ --input "./TOSEC/" \
+ "${INPUTS[@]}" \
+ --input-checksum-archives never \
+ --patch "./Patches/" \
+ --output "./TOSEC/Sega Dreamcast" \
+ --overwrite-invalid \
+ --only-retail \
+ --single \
+ --prefer-language EN \
+ --prefer-region USA,WORLD,EUR,JPN \
+ --prefer-revision newer \
+ -v
npx --yes igir@latest move zip test clean \
`# Official MAME XML extracted from the progetto-SNAPS archive` \
@@ -82,10 +120,13 @@ npx --yes igir@latest move zip test clean \
--dat "./MAME*Rollback*.zip" \
--input "./MAME/" \
"${INPUTS[@]}" \
+ --input-checksum-quick \
+ --input-checksum-archives never \
--output "./MAME/" \
--dir-dat-name \
--overwrite-invalid \
- --merge-roms split
+ --merge-roms merged \
+ -v
```
I then copy ROMs to other devices from this source of truth.
@@ -111,23 +152,29 @@ SOURCE=/Volumes/WDPassport4
npx igir@latest copy extract test clean \
--dat "${SOURCE}/No-Intro*.zip" \
- --dat-name-regex-exclude "/headerless/i" \
+ --dat-name-regex-exclude "/headerless|OSTs/i" \
--input "${SOURCE}/No-Intro/" \
+ --input-exclude "${SOURCE}/No-Intro/Atari - 7800 (BIN)/" \
+ --input-exclude "${SOURCE}/No-Intro/Commodore - Amiga*/**" \
+ --input-exclude "${SOURCE}/No-Intro/Nintendo - Nintendo - Family Computer Disk System (QD)/" \
--input-exclude "${SOURCE}/No-Intro/Nintendo - Game Boy Advance (e-Reader)/" \
+ --input-checksum-quick \
--patch "${SOURCE}/Patches/" \
--output "./Assets/{pocket}/common/" \
--dir-letter \
--dir-letter-limit 1000 \
`# Leave BIOS files alone` \
--clean-exclude "./Assets/*/common/*.*" \
+ --clean-exclude "./Assets/*/common/Palettes/**" \
--overwrite-invalid \
--no-bios \
--no-bad \
--single \
--prefer-language EN \
--prefer-region USA,WORLD,EUR,JPN \
- --prefer-revision-newer \
- --prefer-retail
+ --prefer-revision newer \
+ --prefer-retail \
+ -v
```
That lets me create an EN+USA preferred 1G1R set for my Pocket on the fly, making sure I don't delete BIOS files needed for each core.
@@ -144,16 +191,32 @@ I have this script `sd2sp2_pocket_sync.sh` at the root of my GameCube [SD2SP2](h
#!/usr/bin/env bash
set -euo pipefail
+# shellcheck disable=SC2064
+trap "cd \"${PWD}\"" EXIT
+cd "$(dirname "$0")"
+
+
SOURCE=/Volumes/WDPassport4
-npx --yes igir@latest copy extract test clean \
+npx --yes igir@latest copy test clean report \
+ --dat "${SOURCE}/Redump*.zip" \
+ --dat-name-regex "/GameCube/i" \
--input "${SOURCE}/Redump/Nintendo - GameCube" \
- --output "./ISOs/" \
+ --input-checksum-quick \
+ --input-checksum-archives never \
+ --patch "${SOURCE}/Patches" \
+ --output "./Games/" \
--dir-letter \
+ --overwrite-invalid \
+ --filter-regex-exclude "/(Angler|Baseball|Basketball|Bass|Bonus Disc|Cabela|Disney|ESPN|F1|FIFA|Football|Golf|Madden|MLB|MLS|NASCAR|NBA|NCAA|NFL|NHL|Nickelodeon|Nick Jr|Nicktoons|PGA|Poker|Soccer|Tennis|Tonka|UFC|WWE)/i" \
--no-bios \
--only-retail \
- --filter-regex-exclude "/(Baseball|Cabela|F1|FIFA|Football|Golf|Madden|MLB|NASCAR|NBA|NCAA|NFL|NHL|PGA|Soccer|Tennis|UFC|WWE)/i" \
- --writer-threads 1
+ --single \
+ --prefer-language EN \
+ --prefer-region USA,WORLD,EUR,JPN \
+ --prefer-revision newer \
+ --writer-threads 1 \
+ -v
```
-It doesn't use DATs because I have the ISOs in a trimmed NKit format (see [Swiss](https://github.com/emukidid/swiss-gc)), so they won't match the checksums in DATs. I also exclude some games due to limited SD card size.
+I use the trimmed [NKit format](https://wiki.gbatemp.net/wiki/NKit) for ISOs, which don't make sense to extract, so they're copied as-is. I also exclude some games due to limited SD card size.
diff --git a/index.ts b/index.ts
index b5284fce2..a12a06611 100644
--- a/index.ts
+++ b/index.ts
@@ -27,12 +27,11 @@ gracefulFs.gracefulify(realFs);
process.exit(1);
}
- process.once('SIGINT', async () => {
+ process.once('SIGINT', () => {
+ ProgressBarCLI.stop();
logger.newLine();
logger.notice(`Exiting ${Package.NAME} early`);
- await ProgressBarCLI.stop();
process.exit(0);
- // TODO(cemmer): does exit here cause cleanup not to happen?
});
// Parse CLI arguments
@@ -67,9 +66,9 @@ gracefulFs.gracefulify(realFs);
new UpdateChecker(logger).check();
await new Igir(options, logger).main();
- await ProgressBarCLI.stop();
+ ProgressBarCLI.stop();
} catch (error) {
- await ProgressBarCLI.stop();
+ ProgressBarCLI.stop();
if (error instanceof ExpectedError) {
logger.error(error);
} else if (error instanceof Error && error.stack) {
diff --git a/jest.config.ts b/jest.config.ts
index c4411386c..cfb729aa1 100644
--- a/jest.config.ts
+++ b/jest.config.ts
@@ -1,6 +1,26 @@
+import fs from 'node:fs';
+import path from 'node:path';
+
import { JestConfigWithTsJest } from 'ts-jest';
+// Fix some bad package.json files that don't play well with ts-jest
+[
+ // https://github.com/g-plane/cue/issues/1
+ '@gplane/cue',
+].forEach((moduleName) => {
+ const modulePath = path.join('node_modules', moduleName);
+ const packagePath = path.join(modulePath, 'package.json');
+ const packageJson = JSON.parse(fs.readFileSync(packagePath).toString());
+
+ packageJson.main = packageJson.main
+ ?? packageJson.exports['.'].import;
+ delete packageJson.exports;
+
+ fs.writeFileSync(packagePath, JSON.stringify(packageJson, undefined, 2));
+});
+
const jestConfig: JestConfigWithTsJest = {
+ preset: 'ts-jest',
testEnvironment: 'node',
setupFilesAfterEnv: ['jest-extended/all'],
diff --git a/mkdocs.yml b/mkdocs.yml
index 614c3c516..f67d59b3e 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -2,7 +2,7 @@ site_name: igir
# https://github.com/mkdocs/mkdocs/issues/1783: site_url required for sitemap.xml
site_url: https://igir.io
site_author: Christian Emmer
-# site_description: TODO
+site_description: Igir is a zero-setup ROM collection manager that sorts, filters, extracts or archives, patches, and reports on collections of any size on any OS.
repo_name: emmercm/igir
repo_url: https://github.com/emmercm/igir
@@ -11,8 +11,8 @@ edit_uri: edit/main/docs/
remote_branch: gh-pages
theme:
- logo: logo-light.svg
- favicon: logo-dark.svg
+ logo: static/logo-light.svg
+ favicon: static/favicon.svg
name: material
palette:
@@ -41,79 +41,82 @@ theme:
nav:
- Documentation:
- - index.md
- - Getting Started:
- - overview.md
- - installation.md
- - commands.md
- - alternatives.md
- - Example Usage:
- - usage/collection-sorting.md
- - Emulator Frontends:
- - usage/handheld/adam.md
- - usage/desktop/batocera.md
- - usage/desktop/emuelec.md
- - usage/desktop/emulationstation.md
- - usage/handheld/funkeyos.md
- - usage/handheld/jelos.md
- - usage/desktop/lakka.md
- - usage/desktop/launchbox.md
- - usage/handheld/minui.md
- - usage/handheld/miyoocfw.md
- - usage/handheld/onionos.md
- - usage/desktop/openemu.md
- - usage/desktop/recalbox.md
- - usage/desktop/retroarch.md
- - usage/desktop/retrodeck.md
- - usage/desktop/retropie.md
- - usage/desktop/romm.md
- - usage/handheld/twmenu.md
- - FPGA:
- - usage/hardware/mister.md
- - usage/hardware/analogue-pocket.md
- - Flash Carts:
- - usage/hardware/everdrive.md
- - usage/hardware/ezflash.md
- - Game Consoles:
- - usage/console/gamecube.md
- - usage/console/ps2.md
- - usage/arcade.md
- - usage/personal.md
- - DATs:
- - dats/introduction.md
- - dats/processing.md
- - dats/dir2dat.md
- - dats/fixdats.md
- - File Inputs:
- - input/file-scanning.md
- - input/reading-archives.md
- - ROM Processing:
- - roms/matching.md
- - roms/filtering-preferences.md
- - roms/headers.md
- - roms/patching.md
- - File Outputs:
- - output/path-options.md
- - output/tokens.md
- - output/options.md
- - output/writing-archives.md
- - output/reporting.md
- - output/cleaning.md
- - Advanced:
- - advanced/logging.md
- - advanced/temp-dir.md
- - advanced/troubleshooting.md
- - advanced/internals.md
- - Misc:
- - rom-dumping.md
- - Terms and Conditions:
- - contributing.md
- - license.md
+ - index.md
+ - Getting Started:
+ - introduction.md
+ - installation.md
+ - commands.md
+ - cli.md
+ - alternatives.md
+ - General Usage:
+ - usage/basic.md
+ - usage/personal.md
+ - usage/best-practices.md
+ - Hardware-Specific Usage:
+ - Emulator Frontends:
+ - usage/handheld/adam.md
+ - usage/desktop/batocera.md
+ - usage/desktop/emuelec.md
+ - usage/desktop/emulationstation.md
+ - usage/handheld/funkeyos.md
+ - usage/handheld/jelos.md
+ - usage/desktop/lakka.md
+ - usage/desktop/launchbox.md
+ - usage/handheld/minui.md
+ - usage/handheld/miyoocfw.md
+ - usage/handheld/onionos.md
+ - usage/desktop/openemu.md
+ - usage/desktop/recalbox.md
+ - usage/desktop/retroarch.md
+ - usage/desktop/retrodeck.md
+ - usage/desktop/retropie.md
+ - usage/desktop/romm.md
+ - usage/handheld/twmenu.md
+ - FPGA:
+ - usage/hardware/mister.md
+ - usage/hardware/analogue-pocket.md
+ - Flash Carts:
+ - usage/hardware/everdrive.md
+ - usage/hardware/ezflash.md
+ - Game Consoles:
+ - usage/console/gamecube.md
+ - usage/console/ps2.md
+ - usage/arcade.md
+ - DATs:
+ - dats/introduction.md
+ - dats/processing.md
+ - dats/dir2dat.md
+ - dats/fixdats.md
+ - File Inputs:
+ - input/file-scanning.md
+ - input/reading-archives.md
+ - ROM Processing:
+ - roms/matching.md
+ - roms/filtering-preferences.md
+ - roms/headers.md
+ - roms/patching.md
+ - File Outputs:
+ - output/path-options.md
+ - output/tokens.md
+ - output/options.md
+ - output/writing-archives.md
+ - output/reporting.md
+ - output/cleaning.md
+ - Advanced:
+ - advanced/logging.md
+ - advanced/temp-dir.md
+ - advanced/troubleshooting.md
+ - advanced/internals.md
+ - Misc:
+ - rom-dumping.md
+ - Terms and Conditions:
+ - contributing.md
+ - license.md
# https://github.com/squidfunk/mkdocs-material/issues/889#issuecomment-582297142: how-to open nav links in new tabs
- Download โ: https://github.com/emmercm/igir/releases/latest" target="_blank
- - Donate โ: https://github.com/sponsors/emmercm" target="_blank
- - Issues โ: https://github.com/emmercm/igir/issues?q=is%3Aopen+is%3Aissue+label%3Abug" target="_blank
- Discuss โ: https://github.com/emmercm/igir/discussions" target="_blank
+ - Issues โ: https://github.com/emmercm/igir/issues?q=is%3Aopen+is%3Aissue+label%3Abug" target="_blank
+ - Donate โ: https://github.com/sponsors/emmercm" target="_blank
plugins:
- unused_files:
@@ -134,15 +137,17 @@ plugins:
'archives.md': 'input/reading-archives.md'
'dats.md': 'dats/introduction.md'
'dats/overview.md': 'dats/introduction.md'
- 'examples.md': 'usage/collection-sorting.md'
+ 'examples.md': 'usage/basic.md'
'input/archives.md': 'input/reading-archives.md'
'input/dats.md': 'dats/introduction.md'
'internals.md': 'advanced/internals.md'
'output/arcade.md': 'usage/arcade.md'
+ 'overview.md': 'introduction.md'
'reporting.md': 'output/reporting.md'
'rom-filtering.md': 'roms/filtering-preferences.md'
'rom-headers.md': 'roms/headers.md'
'rom-patching.md': 'roms/patching.md'
+ 'usage/collection-sorting.md': 'usage/basic.md'
#- htmlproofer:
# raise_error_excludes:
# '-1': [ 'http://www.logiqx.com' ]
diff --git a/package-lock.json b/package-lock.json
index 0dce3dd32..c4f9d6ec4 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -12,12 +12,14 @@
"dependencies": {
"@fast-csv/format": "5.0.0",
"@fast-csv/parse": "5.0.0",
+ "@gplane/cue": "0.2.0",
"@node-rs/crc32": "1.10.3",
"7zip-min": "1.4.5",
"archiver": "7.0.1",
"async": "3.2.6",
"async-mutex": "0.5.0",
"chalk": "5.3.0",
+ "chdman": "0.267.3",
"class-transformer": "0.5.1",
"cli-progress": "3.12.0",
"fast-glob": "3.3.2",
@@ -26,6 +28,7 @@
"graceful-fs": "4.2.11",
"is-admin": "4.0.0",
"junk": "4.0.1",
+ "maxcso": "0.1130.6",
"micromatch": "4.0.8",
"moment": "2.30.1",
"node-disk-info": "1.3.0",
@@ -903,6 +906,11 @@
"lodash.uniq": "^4.5.0"
}
},
+ "node_modules/@gplane/cue": {
+ "version": "0.2.0",
+ "resolved": "https://registry.npmjs.org/@gplane/cue/-/cue-0.2.0.tgz",
+ "integrity": "sha512-X5OzPA/Y2NG6IJUbIXLiGSCev0L4AwbOLUoO+dhrLzw70Qcd9S5QeC0SDAjtZZ3jVxvrtO5mRgCSAwpQMHPmhg=="
+ },
"node_modules/@humanwhocodes/config-array": {
"version": "0.11.14",
"resolved": "https://registry.npmjs.org/@humanwhocodes/config-array/-/config-array-0.11.14.tgz",
@@ -4015,6 +4023,23 @@
"node": ">=10"
}
},
+ "node_modules/chdman": {
+ "version": "0.267.3",
+ "resolved": "https://registry.npmjs.org/chdman/-/chdman-0.267.3.tgz",
+ "integrity": "sha512-pLg59Xcc7ux4XCuXHO8gp7bNQIAL0dCHc0HebrddBdFzBRIzxbaDEyyIItMkoknKN372v95RWHe0plGnC8D+VQ==",
+ "dependencies": {
+ "which": "^4.0.0"
+ },
+ "bin": {
+ "chdman": "dist/src/bin.js"
+ },
+ "engines": {
+ "node": ">=16.6.0"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/emmercm"
+ }
+ },
"node_modules/chownr": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/chownr/-/chownr-2.0.0.tgz",
@@ -5676,7 +5701,6 @@
"url": "https://paypal.me/naturalintelligence"
}
],
- "license": "MIT",
"dependencies": {
"strnum": "^1.0.5"
},
@@ -6875,7 +6899,6 @@
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/isexe/-/isexe-3.1.1.tgz",
"integrity": "sha512-LpB/54B+/2J5hqQ7imZHfdU31OlgQqx7ZicVlkm9kzg9/w8GKLEcFfJl/t7DCEDueOyBAD6zCCwTO6Fzs0NoEQ==",
- "dev": true,
"license": "ISC",
"engines": {
"node": ">=16"
@@ -9001,6 +9024,23 @@
"tmpl": "1.0.5"
}
},
+ "node_modules/maxcso": {
+ "version": "0.1130.6",
+ "resolved": "https://registry.npmjs.org/maxcso/-/maxcso-0.1130.6.tgz",
+ "integrity": "sha512-Sv3dFgiJztpf8aUW29CqEkyHXYDfdvWhePp4+U7cimt3HI/7az4RxTkyR9A8BhIQdFeXu73kbZS9Qsw9Bf61Fw==",
+ "dependencies": {
+ "which": "^4.0.0"
+ },
+ "bin": {
+ "maxcso": "dist/src/bin.js"
+ },
+ "engines": {
+ "node": ">=16.6.0"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/emmercm"
+ }
+ },
"node_modules/merge-stream": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz",
@@ -11763,7 +11803,6 @@
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/which/-/which-4.0.0.tgz",
"integrity": "sha512-GlaYyEb07DPxYCKhKzplCWBJtvxZcZMrL+4UkrTSJHHPyZU4mYYTv3qaOe77H7EODLSSopAUFAc6W8U4yqvscg==",
- "dev": true,
"license": "ISC",
"dependencies": {
"isexe": "^3.1.1"
diff --git a/package.json b/package.json
index 77025d36c..c134105bc 100644
--- a/package.json
+++ b/package.json
@@ -1,7 +1,7 @@
{
"name": "igir",
"version": "2.11.0",
- "description": "๐น A video game ROM collection manager to help filter, sort, patch, archive, and report on collections on any OS.",
+ "description": "๐น A zero-setup ROM collection manager that sorts, filters, extracts or archives, patches, and reports on collections of any size on any OS.",
"keywords": [
"1g1r",
"analogue-pocket",
@@ -70,12 +70,14 @@
"dependencies": {
"@fast-csv/format": "5.0.0",
"@fast-csv/parse": "5.0.0",
+ "@gplane/cue": "0.2.0",
"@node-rs/crc32": "1.10.3",
"7zip-min": "1.4.5",
"archiver": "7.0.1",
"async": "3.2.6",
"async-mutex": "0.5.0",
"chalk": "5.3.0",
+ "chdman": "0.267.3",
"class-transformer": "0.5.1",
"cli-progress": "3.12.0",
"fast-glob": "3.3.2",
@@ -84,6 +86,7 @@
"graceful-fs": "4.2.11",
"is-admin": "4.0.0",
"junk": "4.0.1",
+ "maxcso": "0.1130.6",
"micromatch": "4.0.8",
"moment": "2.30.1",
"node-disk-info": "1.3.0",
diff --git a/package.ts b/package.ts
index fb0d7699a..040078f64 100644
--- a/package.ts
+++ b/package.ts
@@ -86,6 +86,12 @@ const fileFilter = (filters: FileFilter[]): string[] => {
// Only include the exact 7zip-bin we need
{ exclude: 'node_modules/{**/,}7zip-bin/**/7z*' },
{ include: path7za },
+ // Only include the exact chdman bin we need
+ { exclude: 'node_modules/{**/,}chdman/bin/*/*/chdman*' },
+ { include: `node_modules/{**/,}chdman/bin/${process.platform}/${process.arch}/chdman*` },
+ // Only include the exact maxcso bin we need
+ { exclude: 'node_modules/{**/,}maxcso/bin/*/*/maxcso*' },
+ { include: `node_modules/{**/,}maxcso/bin/${process.platform}/${process.arch}/maxcso*` },
]));
const includeSize = (await Promise.all([...include].map(async (file) => {
if (await FsPoly.isDirectory(file)) {
@@ -131,7 +137,7 @@ const fileFilter = (filters: FileFilter[]): string[] => {
proc.stdout.on('data', (chunk) => { procOutput += chunk.toString(); });
proc.stderr.on('data', (chunk) => { procOutput += chunk.toString(); });
await new Promise((resolve, reject) => {
- proc.on('exit', resolve);
+ proc.on('close', resolve);
proc.on('error', reject);
});
logger.trace(procOutput);
diff --git a/scripts/asciinema-rec.sh b/scripts/asciinema-rec.sh
index b2b406bb7..de46443c4 100755
--- a/scripts/asciinema-rec.sh
+++ b/scripts/asciinema-rec.sh
@@ -22,22 +22,20 @@ if [[ "${1:-}" == "play" ]]; then
# shellcheck disable=SC2317
npx() {
shift # discard "igir@latest"
- node ../dist/index.js "$@" --dat-name-regex-exclude "/encrypted|headerless/i"
+ node ../dist/index.js "$@" --dat-name-regex-exclude "/encrypted|headerless|3ds/i" --disable-cache
}
# shellcheck disable=SC2317
tree() {
- command tree -N -I -- *.rsl* "$@"
+ command tree -N "$@"
}
# BEGIN PLAYBACK
# ts-node ./index.ts copy zip clean -d demo/No-Intro*.zip -i GB/ -i NES/ -o demo/roms/ -D
- pei "ls -gn"
+ pei 'tree -L 1 .'
echo "" && sleep 2
- pei "unzip -l No-Intro*.zip | head -10" || true
+ pei 'npx igir@latest copy zip report --dat "No-Intro*.zip" --input ROMs/ --output ROMs-Sorted/ --dir-dat-name --only-retail'
echo "" && sleep 2
- pei "npx igir@latest copy zip report --dat No-Intro*.zip --input roms/ --output roms-sorted/ --dir-dat-name --only-retail"
- echo ""
- pei "ls -gn roms-sorted/"
+ pei 'tree -L 1 ROMs-Sorted/'
# END PLAYBACK
exit 0
@@ -82,9 +80,8 @@ npm --version &> /dev/null || exit 1
npm run build
# Clean any previous output
-if [[ -d "${DEMO_DIR}/roms-sorted" ]]; then
- rm -rf "${DEMO_DIR}/roms-sorted"
-fi
+rm -rf "${DEMO_DIR}/roms-sorted"
+rm -rf "${DEMO_DIR}/*.csv"
clear
if [[ "${1:-}" == "rec" ]]; then
diff --git a/src/console/logger.ts b/src/console/logger.ts
index e03d6f4bb..521398449 100644
--- a/src/console/logger.ts
+++ b/src/console/logger.ts
@@ -164,11 +164,11 @@ export default class Logger {
/**
* Create a {@link ProgressBar} with a reference to this {@link Logger}.
*/
- async addProgressBar(
+ addProgressBar(
name: string,
symbol = ProgressBarSymbol.WAITING,
initialTotal = 0,
- ): Promise {
+ ): ProgressBar {
return ProgressBarCLI.new(this, name, symbol, initialTotal);
}
diff --git a/src/console/progressBar.ts b/src/console/progressBar.ts
index 4c0c374e6..69fbe079e 100644
--- a/src/console/progressBar.ts
+++ b/src/console/progressBar.ts
@@ -12,21 +12,23 @@ export const ProgressBarSymbol = {
WAITING: chalk.grey(process.platform === 'win32' ? 'โฆ' : 'โฏ'),
DONE: chalk.green(process.platform === 'win32' ? 'โ' : 'โ'),
// Files
- SEARCHING: chalk.magenta(process.platform === 'win32' ? 'โ' : 'โป'),
- DOWNLOADING: chalk.magenta('โ'),
- PARSING_CONTENTS: chalk.magenta('ฮฃ'),
- DETECTING_HEADERS: chalk.magenta('^'),
- INDEXING: chalk.magenta('#'),
+ FILE_SCANNING: chalk.magenta(process.platform === 'win32' ? 'โ' : 'โป'),
+ DAT_DOWNLOADING: chalk.magenta('โ'),
+ DAT_PARSING: chalk.magenta('ฮฃ'),
+ ROM_HASHING: chalk.magenta('#'),
+ ROM_HEADER_DETECTION: chalk.magenta('^'),
+ ROM_INDEXING: chalk.magenta('โฆ'),
// Processing a single DAT
- GROUPING_SIMILAR: chalk.cyan('โฉ'),
- MERGE_SPLIT: chalk.cyan('โ'),
+ DAT_GROUPING_SIMILAR: chalk.cyan('โฉ'),
+ DAT_MERGE_SPLIT: chalk.cyan('โ'),
// Candidates
- GENERATING: chalk.cyan('ฮฃ'),
- FILTERING: chalk.cyan('โ'),
- EXTENSION_CORRECTION: chalk.cyan('.'),
- HASHING: chalk.cyan('#'),
- VALIDATING: chalk.cyan(process.platform === 'win32' ? '?' : 'โ'),
- COMBINING_ALL: chalk.cyan(process.platform === 'win32' ? 'U' : 'โช'),
+ CANDIDATE_GENERATING: chalk.cyan('ฮฃ'),
+ CANDIDATE_FILTERING: chalk.cyan('โ'),
+ CANDIDATE_EXTENSION_CORRECTION: chalk.cyan('.'),
+ CANDIDATE_HASHING: chalk.yellow('#'),
+ CANDIDATE_VALIDATING: chalk.cyan(process.platform === 'win32' ? '?' : 'โ'),
+ CANDIDATE_COMBINING: chalk.cyan(process.platform === 'win32' ? 'U' : 'โช'),
+ TESTING: chalk.yellow(process.platform === 'win32' ? '?' : 'โ'),
WRITING: chalk.yellow(process.platform === 'win32' ? 'ยป' : 'โ'),
RECYCLING: chalk.blue(process.platform === 'win32' ? 'ยป' : 'โป'),
DELETING: chalk.red(process.platform === 'win32' ? 'X' : 'โ'),
@@ -37,31 +39,31 @@ export const ProgressBarSymbol = {
* information about an operation.
*/
export default abstract class ProgressBar {
- abstract reset(total: number): Promise;
+ abstract reset(total: number): void;
- abstract setName(name: string): Promise;
+ abstract setName(name: string): void;
- abstract setSymbol(symbol: string): Promise;
+ abstract setSymbol(symbol: string): void;
abstract addWaitingMessage(waitingMessage: string): void;
abstract removeWaitingMessage(waitingMessage: string): void;
- abstract incrementTotal(increment: number): Promise;
+ abstract incrementTotal(increment: number): void;
- abstract incrementProgress(): Promise;
+ abstract incrementProgress(): void;
- abstract incrementDone(message?: string): Promise;
+ abstract incrementDone(message?: string): void;
- abstract update(current: number, message?: string): Promise;
+ abstract update(current: number, message?: string): void;
- abstract done(finishedMessage?: string): Promise;
+ abstract done(finishedMessage?: string): void;
/**
* Call the `done()` method with a completion message that indicates how many items were
* processed.
*/
- async doneItems(count: number, noun: string, verb: string): Promise {
+ doneItems(count: number, noun: string, verb: string): void {
let pluralSuffix = 's';
if (noun.toLowerCase().endsWith('ch')
|| noun.toLowerCase().endsWith('s')
@@ -69,10 +71,10 @@ export default abstract class ProgressBar {
pluralSuffix = 'es';
}
- return this.done(`${count.toLocaleString()} ${noun.trim()}${count !== 1 ? pluralSuffix : ''} ${verb}`);
+ this.done(`${count.toLocaleString()} ${noun.trim()}${count !== 1 ? pluralSuffix : ''} ${verb}`);
}
- abstract withLoggerPrefix(prefix: string): ProgressBar;
+ abstract setLoggerPrefix(prefix: string): void;
abstract log(logLevel: LogLevel, message: string): void;
@@ -120,7 +122,7 @@ export default abstract class ProgressBar {
return this.log(LogLevel.ERROR, message);
}
- abstract freeze(): Promise;
+ abstract freeze(): void;
abstract delete(): void;
}
diff --git a/src/console/progressBarCli.ts b/src/console/progressBarCli.ts
index 02b68229f..441ec4c7e 100644
--- a/src/console/progressBarCli.ts
+++ b/src/console/progressBarCli.ts
@@ -5,6 +5,7 @@ import cliProgress, { MultiBar } from 'cli-progress';
import wrapAnsi from 'wrap-ansi';
import ConsolePoly from '../polyfill/consolePoly.js';
+import TimePoly from '../polyfill/timePoly.js';
import Timer from '../timer.js';
import Logger from './logger.js';
import LogLevel from './logLevel.js';
@@ -24,11 +25,11 @@ export default class ProgressBarCLI extends ProgressBar {
private static progressBars: ProgressBarCLI[] = [];
- private static lastRedraw: [number, number] = [0, 0];
+ private static lastRedraw: number = 0;
private static logQueue: string[] = [];
- private readonly logger: Logger;
+ private logger: Logger;
private readonly payload: ProgressBarPayload;
@@ -36,7 +37,7 @@ export default class ProgressBarCLI extends ProgressBar {
private waitingMessageTimeout?: Timer;
- private readonly waitingMessages: Set = new Set();
+ private readonly waitingMessages: Map = new Map();
private constructor(
logger: Logger,
@@ -55,21 +56,24 @@ export default class ProgressBarCLI extends ProgressBar {
/**
* Create a new {@link ProgressBarCLI}, and initialize the {@link MultiBar} if it hasn't been yet.
*/
- static async new(
+ static new(
logger: Logger,
name: string,
symbol: string,
initialTotal = 0,
- ): Promise {
+ ): ProgressBarCLI {
if (!ProgressBarCLI.multiBar) {
ProgressBarCLI.multiBar = new cliProgress.MultiBar({
stream: logger.getLogLevel() < LogLevel.NEVER ? logger.getStream() : new PassThrough(),
- barsize: 25,
+ barsize: 20,
fps: 1 / 60, // limit the automatic redraws
forceRedraw: true,
emptyOnZero: true,
hideCursor: true,
}, cliProgress.Presets.shades_grey);
+ process.on('exit', () => {
+ this.multiBar?.stop();
+ });
}
const initialPayload: ProgressBarPayload = {
@@ -89,17 +93,17 @@ export default class ProgressBarCLI extends ProgressBar {
initialPayload,
);
const progressBarCLI = new ProgressBarCLI(logger, initialPayload, singleBarFormatted);
- await progressBarCLI.render(true);
+ progressBarCLI.render(true);
return progressBarCLI;
}
/**
* Stop the {@link MultiBar} (and therefore everyProgressBar).
*/
- static async stop(): Promise {
+ static stop(): void {
// Freeze (and delete) any lingering progress bars
const progressBarsCopy = ProgressBarCLI.progressBars.slice();
- await Promise.all(progressBarsCopy.map(async (progressBar) => progressBar.freeze()));
+ progressBarsCopy.forEach((progressBar) => progressBar.freeze());
// Clear the last deleted, non-frozen progress bar
ProgressBarCLI.multiBar?.log(' ');
@@ -117,58 +121,65 @@ export default class ProgressBarCLI extends ProgressBar {
* cli-progress clears previous output.
* @see https://github.com/npkgz/cli-progress/issues/79
*/
- async render(force = false): Promise {
+ render(force = false): void {
this.singleBarFormatted?.getSingleBar().update(this.payload);
- if (!force) {
- // Limit the frequency of redrawing
- const [elapsedSec, elapsedNano] = process.hrtime(ProgressBarCLI.lastRedraw);
- const elapsedMs = (elapsedSec * 1_000_000_000 + elapsedNano) / 1_000_000;
- if (elapsedMs < (1000 / ProgressBarCLI.FPS)) {
- return;
+ const callback = (): void => {
+ // Dequeue all log messages
+ if (ProgressBarCLI.multiBar && ProgressBarCLI.logQueue.length > 0) {
+ const consoleWidth = ConsolePoly.consoleWidth();
+ const logMessage = ProgressBarCLI.logQueue
+ // Wrapping is broken: https://github.com/npkgz/cli-progress/issues/142
+ .map((msg) => wrapAnsi(msg, consoleWidth, { trim: false })
+ // ...and if we manually wrap lines, we also need to deal with overwriting existing
+ // progress bar output.
+ .split('\n')
+ // TODO(cemmer): this appears to only overwrite the last line, not any others?
+ .join(`\n${this.logger.isTTY() ? '\x1b[K' : ''}`))
+ .join('\n');
+ ProgressBarCLI.multiBar.log(`${logMessage}\n`);
+ ProgressBarCLI.logQueue = [];
}
+
+ ProgressBarCLI.multiBar?.update();
+ ProgressBarCLI.lastRedraw = TimePoly.hrtimeMillis();
+ ProgressBarCLI.RENDER_MUTEX.cancel(); // cancel all waiting locks, we just redrew
+ };
+
+ if (force) {
+ callback();
+ return;
}
- try {
- await ProgressBarCLI.RENDER_MUTEX.runExclusive(() => {
- // Dequeue all log messages
- if (ProgressBarCLI.multiBar && ProgressBarCLI.logQueue.length > 0) {
- const consoleWidth = ConsolePoly.consoleWidth();
- const logMessage = ProgressBarCLI.logQueue
- // Wrapping is broken: https://github.com/npkgz/cli-progress/issues/142
- .map((msg) => wrapAnsi(msg, consoleWidth, { trim: false })
- // ...and if we manually wrap lines, we also need to deal with overwriting existing
- // progress bar output.
- .split('\n')
- .join(`\n${this.logger.isTTY() ? '\x1b[K' : ''}`))
- .join('\n');
- ProgressBarCLI.multiBar.log(`${logMessage}\n`);
- ProgressBarCLI.logQueue = [];
- }
+ // Limit the frequency of redrawing
+ const elapsedMs = TimePoly.hrtimeMillis(ProgressBarCLI.lastRedraw);
+ if (elapsedMs < (1000 / ProgressBarCLI.FPS)) {
+ return;
+ }
- ProgressBarCLI.multiBar?.update();
- ProgressBarCLI.lastRedraw = process.hrtime();
- ProgressBarCLI.RENDER_MUTEX.cancel(); // cancel all waiting locks, we just redrew
- });
- } catch (error) {
- if (error !== E_CANCELED) {
- throw error;
+ setImmediate(async () => {
+ try {
+ await ProgressBarCLI.RENDER_MUTEX.runExclusive(callback);
+ } catch (error) {
+ if (error !== E_CANCELED) {
+ throw error;
+ }
}
- }
+ });
}
/**
* Reset the {@link ProgressBar}'s progress to zero and change its total.
*/
- async reset(total: number): Promise {
+ reset(total: number): void {
this.singleBarFormatted?.getSingleBar().setTotal(total);
this.singleBarFormatted?.getSingleBar().update(0);
this.payload.inProgress = 0;
this.payload.waitingMessage = undefined;
- return this.render(true);
+ this.render(true);
}
- private async logPayload(): Promise {
+ private logPayload(): void {
const name = this.payload.name ?? '';
const finishedMessageWrapped = this.payload.finishedMessage
?.split('\n')
@@ -184,17 +195,23 @@ export default class ProgressBarCLI extends ProgressBar {
LogLevel.ALWAYS,
`${name}${finishedMessageWrapped ? ` ... ${finishedMessageWrapped}` : ''}`,
);
- await this.render(true);
+ this.render(true);
}
- async setName(name: string): Promise {
+ setName(name: string): void {
+ if (this.payload.name === name) {
+ return;
+ }
this.payload.name = name;
- return this.render(true);
+ this.render(true);
}
- async setSymbol(symbol: string): Promise {
+ setSymbol(symbol: string): void {
+ if (this.payload.symbol === symbol) {
+ return;
+ }
this.payload.symbol = symbol;
- return this.render(true);
+ this.render(true);
}
/**
@@ -205,9 +222,24 @@ export default class ProgressBarCLI extends ProgressBar {
if (!this.singleBarFormatted) {
return;
}
+ this.waitingMessages.set(waitingMessage, TimePoly.hrtimeMillis());
+
+ if (!this.waitingMessageTimeout) {
+ this.waitingMessageTimeout = Timer.setInterval(() => {
+ const currentMillis = TimePoly.hrtimeMillis();
+ const newWaitingMessagePair = [...this.waitingMessages]
+ .find(([, ms]) => currentMillis - ms >= 5000);
- this.waitingMessages.add(waitingMessage);
- this.setWaitingMessageTimeout();
+ const newWaitingMessage = newWaitingMessagePair !== undefined
+ ? newWaitingMessagePair[0]
+ : undefined;
+
+ if (newWaitingMessage !== this.payload.waitingMessage) {
+ this.payload.waitingMessage = newWaitingMessage;
+ this.render(true);
+ }
+ }, 1000 / ProgressBarCLI.FPS);
+ }
}
/**
@@ -217,32 +249,13 @@ export default class ProgressBarCLI extends ProgressBar {
if (!this.singleBarFormatted) {
return;
}
-
this.waitingMessages.delete(waitingMessage);
- if (this.payload.waitingMessage) {
- // Render immediately if the output could change
- this.setWaitingMessageTimeout(0);
- }
- }
-
- private setWaitingMessageTimeout(timeout = 10_000): void {
- this.waitingMessageTimeout?.cancel();
-
- this.waitingMessageTimeout = Timer.setTimeout(async () => {
- const total = this.singleBarFormatted?.getSingleBar().getTotal() ?? 0;
- if (total <= 1) {
- return;
- }
-
- [this.payload.waitingMessage] = this.waitingMessages;
- await this.render(true);
- }, timeout);
}
/**
* Increment the total by some amount.
*/
- async incrementTotal(increment: number): Promise {
+ incrementTotal(increment: number): void {
if (!this.singleBarFormatted) {
return;
}
@@ -250,39 +263,39 @@ export default class ProgressBarCLI extends ProgressBar {
this.singleBarFormatted.getSingleBar().setTotal(
this.singleBarFormatted.getSingleBar().getTotal() + increment,
);
- await this.render();
+ this.render();
}
/**
* Increment the in-progress count by one.
*/
- async incrementProgress(): Promise {
+ incrementProgress(): void {
this.payload.inProgress = Math.max(this.payload.inProgress ?? 0, 0) + 1;
- return this.render();
+ this.render();
}
/**
* Decrement the in-progress count by one, and increment the completed count by one.
*/
- async incrementDone(): Promise {
+ incrementDone(): void {
this.payload.inProgress = Math.max((this.payload.inProgress ?? 0) - 1, 0);
this.singleBarFormatted?.getSingleBar().increment();
- return this.render();
+ this.render();
}
/**
* Set the completed count.
*/
- async update(current: number): Promise {
+ update(current: number): void {
this.singleBarFormatted?.getSingleBar().update(current);
- return this.render();
+ this.render();
}
/**
* Set the completed count to the total, and render any completion message.
*/
- async done(finishedMessage?: string): Promise {
- await this.setSymbol(ProgressBarSymbol.DONE);
+ done(finishedMessage?: string): void {
+ this.setSymbol(ProgressBarSymbol.DONE);
const total = this.singleBarFormatted?.getSingleBar().getTotal() ?? 0;
if (total > 0) {
@@ -296,18 +309,14 @@ export default class ProgressBarCLI extends ProgressBar {
this.payload.finishedMessage = finishedMessage;
}
- await this.render(true);
+ this.render(true);
}
/**
* Return a copy of this {@link ProgressBar} with a new string prefix.
*/
- withLoggerPrefix(prefix: string): ProgressBar {
- return new ProgressBarCLI(
- this.logger.withLoggerPrefix(prefix),
- this.payload,
- this.singleBarFormatted,
- );
+ setLoggerPrefix(prefix: string): void {
+ this.logger = this.logger.withLoggerPrefix(prefix);
}
/**
@@ -335,13 +344,13 @@ export default class ProgressBarCLI extends ProgressBar {
* at once.
* @see https://github.com/npkgz/cli-progress/issues/59
*/
- async freeze(): Promise {
+ freeze(): void {
if (!this.singleBarFormatted) {
- await this.logPayload();
+ this.logPayload();
return;
}
- await this.render(true);
+ this.render(true);
ProgressBarCLI.multiBar?.log(`${this.singleBarFormatted?.getLastOutput()}\n`);
this.delete();
}
diff --git a/src/console/singleBarFormatted.ts b/src/console/singleBarFormatted.ts
index f2bfdeb6f..cf94b62c6 100644
--- a/src/console/singleBarFormatted.ts
+++ b/src/console/singleBarFormatted.ts
@@ -5,13 +5,15 @@ import {
import { linearRegression, linearRegressionLine } from 'simple-statistics';
import stripAnsi from 'strip-ansi';
+import ConsolePoly from '../polyfill/consolePoly.js';
+import TimePoly from '../polyfill/timePoly.js';
import ProgressBarPayload from './progressBarPayload.js';
/**
* A wrapper class for a cli-progress {@link SingleBar} that formats the output.
*/
export default class SingleBarFormatted {
- public static readonly MAX_NAME_LENGTH = 30;
+ public static readonly MAX_NAME_LENGTH = 35;
public static readonly BAR_COMPLETE_CHAR = '\u2588';
@@ -27,7 +29,7 @@ export default class SingleBarFormatted {
private valueTimeBuffer: number[][] = [];
- private lastEtaTime: [number, number] = [0, 0];
+ private lastEtaTime: number = 0;
private lastEtaValue = 'infinity';
@@ -35,18 +37,26 @@ export default class SingleBarFormatted {
this.multiBar = multiBar;
this.singleBar = this.multiBar.create(initialTotal, 0, initialPayload, {
format: (options, params, payload: ProgressBarPayload): string => {
- const symbolAndName = SingleBarFormatted.getSymbolAndName(payload);
+ const symbolAndName = `${SingleBarFormatted.getSymbolAndName(payload)} | `;
+
const progressWrapped = this.getProgress(options, params, payload)
.split('\n')
.map((line, idx) => {
+ // Wrapping is broken: https://github.com/npkgz/cli-progress/issues/142
+ let lineTrimmed = line;
+ const maxLineLength = ConsolePoly.consoleWidth() - stripAnsi(symbolAndName).length - 2;
+ if (line.length > maxLineLength) {
+ lineTrimmed = `${line.slice(0, maxLineLength - 3)}...`;
+ }
+
if (idx === 0) {
- return line;
+ return lineTrimmed;
}
- return ' '.repeat(stripAnsi(symbolAndName).length + 3) + line;
+ return ' '.repeat(stripAnsi(symbolAndName).length) + lineTrimmed;
})
.join('\n\x1b[K');
- this.lastOutput = `${symbolAndName} | ${progressWrapped}`.trim();
+ this.lastOutput = `${symbolAndName}${progressWrapped}`.trim();
return this.lastOutput
// cli-progress doesn't handle multi-line progress bars, collapse to one line. The multi-
// line message will get logged correctly when the progress bar is frozen & logged.
@@ -156,12 +166,11 @@ export default class SingleBarFormatted {
private getEtaFormatted(etaSeconds: number): string {
// Rate limit how often the ETA can change
// Update only every 5s if the ETA is >60s
- const [elapsedSec, elapsedNano] = process.hrtime(this.lastEtaTime);
- const elapsedMs = (elapsedSec * 1_000_000_000 + elapsedNano) / 1_000_000;
+ const elapsedMs = TimePoly.hrtimeMillis(this.lastEtaTime);
if (etaSeconds > 60 && elapsedMs < 5000) {
return this.lastEtaValue;
}
- this.lastEtaTime = process.hrtime();
+ this.lastEtaTime = TimePoly.hrtimeMillis();
if (etaSeconds < 0) {
this.lastEtaValue = 'infinity';
diff --git a/src/driveSemaphore.ts b/src/driveSemaphore.ts
index a9377d72b..05aba15ba 100644
--- a/src/driveSemaphore.ts
+++ b/src/driveSemaphore.ts
@@ -3,6 +3,7 @@ import path from 'node:path';
import async, { AsyncResultCallback } from 'async';
import { Mutex, Semaphore } from 'async-mutex';
+import ElasticSemaphore from './elasticSemaphore.js';
import Defaults from './globals/defaults.js';
import FsPoly from './polyfill/fsPoly.js';
import File from './types/files/file.js';
@@ -12,36 +13,109 @@ import File from './types/files/file.js';
* once per hard drive.
*/
export default class DriveSemaphore {
- private readonly keySemaphores = new Map();
+ private readonly driveSemaphores = new Map();
- private readonly keySemaphoresMutex = new Mutex();
-
- private readonly threads: number;
+ private readonly driveSemaphoresMutex = new Mutex();
private readonly threadsSemaphore: Semaphore;
- constructor(threads = 1) {
- this.threads = threads;
+ constructor(threads: number) {
this.threadsSemaphore = new Semaphore(threads);
}
+ getValue(): number {
+ return this.threadsSemaphore.getValue();
+ }
+
+ setValue(threads: number): void {
+ this.threadsSemaphore.setValue(threads);
+ }
+
+ /**
+ * Run a {@link runnable} exclusively for the given {@link file}.
+ */
+ async runExclusive(
+ file: File | string,
+ runnable: () => V | Promise,
+ ): Promise {
+ const filePathDisk = DriveSemaphore.getDiskForFile(file);
+ const driveSemaphore = await this.driveSemaphoresMutex.runExclusive(() => {
+ if (!this.driveSemaphores.has(filePathDisk)) {
+ // WARN(cemmer): there is an undocumented semaphore max value that can be used, the full
+ // 4,700,372,992 bytes of a DVD+R will cause runExclusive() to never run or return.
+ let maxKilobytes = Defaults.MAX_READ_WRITE_CONCURRENT_KILOBYTES;
+
+ if (FsPoly.isSamba(filePathDisk)) {
+ // Forcefully limit the number of files to be processed concurrently from a single
+ // Samba network share
+ maxKilobytes = 1;
+ }
+
+ this.driveSemaphores.set(filePathDisk, new ElasticSemaphore(maxKilobytes));
+ }
+
+ return this.driveSemaphores.get(filePathDisk) as ElasticSemaphore;
+ });
+
+ const fileSizeKilobytes = (file instanceof File && file.getSize() > 0
+ ? file.getSize()
+ : await FsPoly.size(file instanceof File ? file.getFilePath() : file)
+ ) / 1024;
+
+ // First, limit the number of threads per drive, which will better balance the processing of
+ // files on different drives vs. processing files sequentially
+ return driveSemaphore.runExclusive(
+ // Second, limit the overall number of threads
+ async () => this.threadsSemaphore.runExclusive(
+ async () => runnable(),
+ ),
+ fileSizeKilobytes,
+ );
+ }
+
/**
* Run some {@link runnable} for every value in {@link files}.
*/
async map(
files: K[],
- runnable: (file: K) => (V | Promise),
+ runnable: (file: K) => V | Promise,
): Promise {
- const disks = FsPoly.disksSync();
+ // Sort the files, then "stripe" them by their disk path for fair processing among disks
+ const disksToFiles = files
+ // Remember the original ordering of the files by its index
+ .map((file, idx) => ([file, idx] satisfies [K, number]))
+ .sort(([a], [b]) => {
+ const aPath = a instanceof File ? a.getFilePath() : a.toString();
+ const bPath = b instanceof File ? b.getFilePath() : b.toString();
+ return aPath.localeCompare(bPath);
+ })
+ .reduce((map, [file, idx]) => {
+ const key = DriveSemaphore.getDiskForFile(file);
+ if (!map.has(key)) {
+ map.set(key, [[file, idx]]);
+ } else {
+ map.get(key)?.push([file, idx]);
+ }
+ return map;
+ }, new Map());
+ const maxFilesOnAnyDisk = [...disksToFiles.values()]
+ .reduce((max, filesForDisk) => Math.max(max, filesForDisk.length), 0);
+ let filesStriped: [K, number][] = [];
+ const chunkSize = 5;
+ for (let i = 0; i < maxFilesOnAnyDisk; i += chunkSize) {
+ const batch = [...disksToFiles.values()]
+ .flatMap((filesForDisk) => filesForDisk.splice(0, chunkSize));
+ filesStriped = [...filesStriped, ...batch];
+ }
// Limit the number of ongoing threads to something reasonable
- return async.mapLimit(
- files,
+ const results = await async.mapLimit(
+ filesStriped,
Defaults.MAX_FS_THREADS,
- async (file, callback: AsyncResultCallback) => {
+ async ([file, idx], callback: AsyncResultCallback<[V, number], Error>) => {
try {
- const val = await this.processFile(file, runnable, disks);
- callback(undefined, val);
+ const val = await this.runExclusive(file, async () => runnable(file));
+ callback(undefined, [val, idx]);
} catch (error) {
if (error instanceof Error) {
callback(error);
@@ -53,48 +127,29 @@ export default class DriveSemaphore {
}
},
);
+
+ // Put the values back in order
+ return results
+ .sort(([, aIdx], [, bIdx]) => aIdx - bIdx)
+ .map(([result]) => result);
}
- private async processFile(
- file: K,
- runnable: (file: K) => (V | Promise),
- disks: string[],
- ): Promise {
+ private static getDiskForFile(file: File | string): string {
const filePath = file instanceof File ? file.getFilePath() : file as string;
const filePathNormalized = filePath.replace(/[\\/]/g, path.sep);
- const filePathResolved = path.resolve(filePathNormalized);
// Try to get the path of the drive this file is on
- let filePathDisk = disks.find((disk) => filePathResolved.startsWith(disk)) ?? '';
-
- if (!filePathDisk) {
- // If a drive couldn't be found, try to parse a samba server name
- const sambaMatches = filePathNormalized.match(/^([\\/]{2}[^\\/]+)/);
- if (sambaMatches !== null) {
- [, filePathDisk] = sambaMatches;
- }
+ const filePathDisk = FsPoly.diskResolved(filePathNormalized);
+ if (filePathDisk !== undefined) {
+ return filePathDisk;
}
- const keySemaphore = await this.keySemaphoresMutex.runExclusive(() => {
- if (!this.keySemaphores.has(filePathDisk)) {
- let { threads } = this;
- if (FsPoly.isSamba(filePathDisk)) {
- // Forcefully limit the number of files to be processed concurrently from a single
- // Samba network share
- threads = 1;
- }
- this.keySemaphores.set(filePathDisk, new Semaphore(threads));
- }
- return this.keySemaphores.get(filePathDisk) as Semaphore;
- });
+ // If a drive couldn't be found, try to parse a samba server name
+ const sambaMatches = filePathNormalized.match(/^([\\/]{2}[^\\/]+)/);
+ if (sambaMatches !== null) {
+ return sambaMatches[1];
+ }
- // First, limit the number of threads per drive, which will better balance the processing of
- // files on different drives vs. processing files sequentially
- return keySemaphore.runExclusive(
- // Second, limit the overall number of threads
- async () => this.threadsSemaphore.runExclusive(
- async () => runnable(file),
- ),
- );
+ return '';
}
}
diff --git a/src/elasticSemaphore.ts b/src/elasticSemaphore.ts
index 36250933e..95ebf55de 100644
--- a/src/elasticSemaphore.ts
+++ b/src/elasticSemaphore.ts
@@ -1,49 +1,35 @@
-import { Mutex, Semaphore } from 'async-mutex';
+import { Semaphore } from 'async-mutex';
/**
* Wrapper for an `async-mutex` {@link Semaphore} that can have its total increased if a weight
* exceeds the maximum.
*/
export default class ElasticSemaphore {
- private readonly valueMutex = new Mutex();
-
- private value: number;
+ private readonly semaphoreValue: number;
private readonly semaphore: Semaphore;
constructor(value: number) {
- this.value = Math.ceil(value);
- this.semaphore = new Semaphore(this.value);
+ this.semaphoreValue = Math.ceil(value);
+ this.semaphore = new Semaphore(this.semaphoreValue);
}
/**
* Run some {@link callback} with a required {@link weight}.
*/
- async runExclusive(callback: (value: number) => Promise | T, weight: number): Promise {
+ async runExclusive(
+ callback: (value: number) => Promise | T,
+ weight: number,
+ ): Promise {
const weightNormalized = Math.max(1, Math.ceil(weight));
- // If the weight of this call isn't even 1% of the max value then don't incur the overhead
- // of a semaphore
- if ((weightNormalized / this.value) * 100 < 1) {
- return callback(this.semaphore.getValue());
- }
-
- // If the weight of this call is larger than the max value then we need to increase the max
- if (weightNormalized > this.value) {
- await this.valueMutex.runExclusive(() => {
- const increase = weightNormalized - this.value;
- if (increase <= 0) {
- // A competing runnable already increased this semaphore's value
- return;
- }
- this.semaphore.setValue(this.semaphore.getValue() + increase);
- this.value += increase;
- });
- }
-
// NOTE(cemmer): this semaphore can take a measurable amount of time to actually call the
// callback. This is particularly noticeable when using single threads (e.g. via Async.js).
// Try to only use semaphores to traffic cop multiple concurrent threads.
- return this.semaphore.runExclusive(callback, weightNormalized);
+ return this.semaphore.runExclusive(
+ callback,
+ // If the weight of this call is larger than the max value then just use the max value
+ weightNormalized > this.semaphoreValue ? this.semaphoreValue : weightNormalized,
+ );
}
}
diff --git a/src/globals/temp.ts b/src/globals/temp.ts
index 59f543252..9b8ab88d8 100644
--- a/src/globals/temp.ts
+++ b/src/globals/temp.ts
@@ -23,9 +23,8 @@ export default class Temp {
}
}
-process.once('beforeExit', async () => {
- // WARN: Jest won't call this: https://github.com/jestjs/jest/issues/10927
- await FsPoly.rm(Temp.getTempDir(), {
+process.once('exit', () => {
+ FsPoly.rmSync(Temp.getTempDir(), {
force: true,
recursive: true,
});
diff --git a/src/igir.ts b/src/igir.ts
index a122ebc7a..1b8257669 100644
--- a/src/igir.ts
+++ b/src/igir.ts
@@ -88,33 +88,44 @@ export default class Igir {
this.logger.trace('Windows has symlink permissions');
}
+ if (this.options.shouldLink() && !this.options.getSymlink()) {
+ const outputDirRoot = this.options.getOutputDirRoot();
+ if (!await FsPoly.canHardlink(outputDirRoot)) {
+ const outputDisk = FsPoly.diskResolved(outputDirRoot);
+ throw new ExpectedError(`${outputDisk} does not support hard-linking`);
+ }
+ }
+
// File cache options
+ const fileCache = new FileCache();
if (this.options.getDisableCache()) {
this.logger.trace('disabling the file cache');
- FileCache.disable();
+ fileCache.disable();
} else {
const cachePath = await this.getCachePath();
if (cachePath !== undefined && process.env.NODE_ENV !== 'test') {
this.logger.trace(`loading the file cache at '${cachePath}'`);
- await FileCache.loadFile(cachePath);
+ await fileCache.loadFile(cachePath);
} else {
this.logger.trace('not using a file for the file cache');
}
}
+ const fileFactory = new FileFactory(fileCache);
// Scan and process input files
- let dats = await this.processDATScanner();
+ let dats = await this.processDATScanner(fileFactory);
const indexedRoms = await this.processROMScanner(
+ fileFactory,
this.determineScanningBitmask(dats),
this.determineScanningChecksumArchives(dats),
);
const roms = indexedRoms.getFiles();
- const patches = await this.processPatchScanner();
+ const patches = await this.processPatchScanner(fileFactory);
// Set up progress bar and input for DAT processing
- const datProcessProgressBar = await this.logger.addProgressBar(chalk.underline('Processing DATs'), ProgressBarSymbol.NONE, dats.length);
+ const datProcessProgressBar = this.logger.addProgressBar(chalk.underline('Processing DATs'), ProgressBarSymbol.NONE, dats.length);
if (dats.length === 0) {
- dats = new DATGameInferrer(this.options, datProcessProgressBar).infer(roms);
+ dats = await new DATGameInferrer(this.options, datProcessProgressBar).infer(roms);
}
const datsToWrittenFiles = new Map();
@@ -125,22 +136,23 @@ export default class Igir {
// Process every DAT
datProcessProgressBar.logTrace(`processing ${dats.length.toLocaleString()} DAT${dats.length !== 1 ? 's' : ''}`);
await async.eachLimit(dats, this.options.getDatThreads(), async (dat, callback) => {
- await datProcessProgressBar.incrementProgress();
+ datProcessProgressBar.incrementProgress();
- const progressBar = await this.logger.addProgressBar(
+ const progressBar = this.logger.addProgressBar(
dat.getNameShort(),
ProgressBarSymbol.WAITING,
dat.getParents().length,
);
- const datWithParents = await new DATParentInferrer(this.options, progressBar).infer(dat);
- const mergedSplitDat = await new DATMergerSplitter(this.options, progressBar)
+ const datWithParents = new DATParentInferrer(this.options, progressBar).infer(dat);
+ const mergedSplitDat = new DATMergerSplitter(this.options, progressBar)
.merge(datWithParents);
- const filteredDat = await new DATFilter(this.options, progressBar).filter(mergedSplitDat);
+ const filteredDat = new DATFilter(this.options, progressBar).filter(mergedSplitDat);
// Generate and filter ROM candidates
const parentsToCandidates = await this.generateCandidates(
progressBar,
+ fileFactory,
filteredDat,
indexedRoms,
patches,
@@ -180,7 +192,7 @@ export default class Igir {
const datStatus = new StatusGenerator(this.options, progressBar)
.generate(filteredDat, parentsToCandidates);
datsStatuses.push(datStatus);
- await progressBar.done([
+ progressBar.done([
datStatus.toConsole(this.options),
dir2DatPath ? `dir2dat: ${dir2DatPath}` : undefined,
fixdatPath ? `Fixdat: ${fixdatPath}` : undefined,
@@ -190,17 +202,17 @@ export default class Igir {
const totalReleaseCandidates = [...parentsToCandidates.values()]
.reduce((sum, rcs) => sum + rcs.length, 0);
if (totalReleaseCandidates > 0) {
- await progressBar.freeze();
+ progressBar.freeze();
} else {
progressBar.delete();
}
- await datProcessProgressBar.incrementDone();
+ datProcessProgressBar.incrementDone();
callback();
});
datProcessProgressBar.logTrace(`done processing ${dats.length.toLocaleString()} DAT${dats.length !== 1 ? 's' : ''}`);
- await datProcessProgressBar.doneItems(dats.length, 'DAT', 'processed');
+ datProcessProgressBar.doneItems(dats.length, 'DAT', 'processed');
datProcessProgressBar.delete();
// Delete moved ROMs
@@ -212,19 +224,23 @@ export default class Igir {
// Generate the report
await this.processReportGenerator(roms, cleanedOutputFiles, datsStatuses);
- await ProgressBarCLI.stop();
+ ProgressBarCLI.stop();
Timer.cancelAll();
}
private async getCachePath(): Promise {
- const defaultFileName = `${Package.NAME}.cache`;
+ const defaultFileName = process.versions.bun
+ // As of v1.1.26, Bun uses a different serializer than V8, making cache files incompatible
+ // @see https://bun.sh/docs/runtime/nodejs-apis
+ ? `${Package.NAME}.bun.cache`
+ : `${Package.NAME}.cache`;
- // Try to use the provided path
+ // First, try to use the provided path
let cachePath = this.options.getCachePath();
if (cachePath !== undefined && await FsPoly.isDirectory(cachePath)) {
cachePath = path.join(cachePath, defaultFileName);
- this.logger.warn(`A directory was provided for cache path instead of a file, using '${cachePath}' instead`);
+ this.logger.warn(`A directory was provided for the cache path instead of a file, using '${cachePath}' instead`);
}
if (cachePath !== undefined) {
if (await FsPoly.isWritable(cachePath)) {
@@ -233,22 +249,36 @@ export default class Igir {
this.logger.warn('Provided cache path isn\'t writable, using the default path');
}
- // Otherwise, use a default path
- return [
+ const cachePathCandidates = [
path.join(path.resolve(Package.DIRECTORY), defaultFileName),
path.join(os.homedir(), defaultFileName),
path.join(process.cwd(), defaultFileName),
]
.filter((filePath) => filePath && !filePath.startsWith(os.tmpdir()))
- .find(async (filePath) => {
- if (await FsPoly.exists(filePath)) {
- return true;
- }
- return FsPoly.isWritable(filePath);
- });
+ .reduce(ArrayPoly.reduceUnique(), []);
+
+ // Next, try to use an already existing path
+ const exists = await Promise.all(
+ cachePathCandidates.map(async (pathCandidate) => FsPoly.exists(pathCandidate)),
+ );
+ const existsCachePath = cachePathCandidates.find((_, idx) => exists[idx]);
+ if (existsCachePath !== undefined) {
+ return existsCachePath;
+ }
+
+ // Next, try to find a writable path
+ const writable = await Promise.all(
+ cachePathCandidates.map(async (pathCandidate) => FsPoly.isWritable(pathCandidate)),
+ );
+ const writableCachePath = cachePathCandidates.find((_, idx) => writable[idx]);
+ if (writableCachePath !== undefined) {
+ return writableCachePath;
+ }
+
+ return undefined;
}
- private async processDATScanner(): Promise {
+ private async processDATScanner(fileFactory: FileFactory): Promise {
if (this.options.shouldDir2Dat()) {
return [];
}
@@ -257,8 +287,8 @@ export default class Igir {
return [];
}
- const progressBar = await this.logger.addProgressBar('Scanning for DATs');
- let dats = await new DATScanner(this.options, progressBar).scan();
+ const progressBar = this.logger.addProgressBar('Scanning for DATs');
+ let dats = await new DATScanner(this.options, progressBar, fileFactory).scan();
if (dats.length === 0) {
throw new ExpectedError('No valid DAT files found!');
}
@@ -275,17 +305,24 @@ export default class Igir {
}
if (this.options.getDatCombine()) {
- await progressBar.reset(1);
+ progressBar.reset(1);
dats = [new DATCombiner(progressBar).combine(dats)];
}
- await progressBar.doneItems(dats.length, 'DAT', 'found');
- await progressBar.freeze();
+ progressBar.doneItems(dats.length, 'DAT', this.options.getDatCombine() ? 'combined' : 'found');
+ progressBar.freeze();
return dats;
}
private determineScanningBitmask(dats: DAT[]): number {
- const minimumChecksum = this.options.getInputMinChecksum() ?? ChecksumBitmask.CRC32;
+ const minimumChecksum = this.options.getInputChecksumMin() ?? ChecksumBitmask.NONE;
+ const maximumChecksum = this.options.getInputChecksumMax()
+ ?? Object.keys(ChecksumBitmask)
+ .filter((bitmask): bitmask is keyof typeof ChecksumBitmask => Number.isNaN(Number(bitmask)))
+ .map((bitmask) => ChecksumBitmask[bitmask])
+ .at(-1)
+ ?? minimumChecksum;
+
let matchChecksum = minimumChecksum;
if (this.options.getPatchFileCount() > 0) {
@@ -307,20 +344,43 @@ export default class Igir {
}
dats.forEach((dat) => {
- const datMinimumBitmask = dat.getRequiredChecksumBitmask();
+ const datMinimumRomBitmask = dat.getRequiredRomChecksumBitmask();
+ Object.keys(ChecksumBitmask)
+ .filter((bitmask): bitmask is keyof typeof ChecksumBitmask => Number.isNaN(Number(bitmask)))
+ // Has not been enabled yet
+ .filter((bitmask) => ChecksumBitmask[bitmask] > minimumChecksum
+ && ChecksumBitmask[bitmask] <= maximumChecksum)
+ .filter((bitmask) => !(matchChecksum & ChecksumBitmask[bitmask]))
+ // Should be enabled for this DAT
+ .filter((bitmask) => datMinimumRomBitmask & ChecksumBitmask[bitmask])
+ .forEach((bitmask) => {
+ matchChecksum |= ChecksumBitmask[bitmask];
+ this.logger.trace(`${dat.getNameShort()}: needs ${bitmask} file checksums for ROMs, enabling`);
+ });
+
+ if (this.options.getExcludeDisks()) {
+ return;
+ }
+ const datMinimumDiskBitmask = dat.getRequiredDiskChecksumBitmask();
Object.keys(ChecksumBitmask)
.filter((bitmask): bitmask is keyof typeof ChecksumBitmask => Number.isNaN(Number(bitmask)))
// Has not been enabled yet
- .filter((bitmask) => ChecksumBitmask[bitmask] > minimumChecksum)
+ .filter((bitmask) => ChecksumBitmask[bitmask] > minimumChecksum
+ && ChecksumBitmask[bitmask] <= maximumChecksum)
.filter((bitmask) => !(matchChecksum & ChecksumBitmask[bitmask]))
// Should be enabled for this DAT
- .filter((bitmask) => datMinimumBitmask & ChecksumBitmask[bitmask])
+ .filter((bitmask) => datMinimumDiskBitmask & ChecksumBitmask[bitmask])
.forEach((bitmask) => {
matchChecksum |= ChecksumBitmask[bitmask];
- this.logger.trace(`${dat.getNameShort()}: needs ${bitmask} file checksums, enabling`);
+ this.logger.trace(`${dat.getNameShort()}: needs ${bitmask} file checksums for disks, enabling`);
});
});
+ if (matchChecksum === ChecksumBitmask.NONE) {
+ matchChecksum |= ChecksumBitmask.CRC32;
+ this.logger.trace('at least one checksum algorithm is required, enabling CRC32 file checksums');
+ }
+
return matchChecksum;
}
@@ -344,44 +404,48 @@ export default class Igir {
}
private async processROMScanner(
+ fileFactory: FileFactory,
checksumBitmask: number,
checksumArchives: boolean,
): Promise {
const romScannerProgressBarName = 'Scanning for ROMs';
- const romProgressBar = await this.logger.addProgressBar(romScannerProgressBarName);
+ const romProgressBar = this.logger.addProgressBar(romScannerProgressBarName);
- const rawRomFiles = await new ROMScanner(this.options, romProgressBar)
+ const rawRomFiles = await new ROMScanner(this.options, romProgressBar, fileFactory)
.scan(checksumBitmask, checksumArchives);
- await romProgressBar.setName('Detecting ROM headers');
- const romFilesWithHeaders = await new ROMHeaderProcessor(this.options, romProgressBar)
- .process(rawRomFiles);
+ romProgressBar.setName('Detecting ROM headers');
+ const romFilesWithHeaders = await new ROMHeaderProcessor(
+ this.options,
+ romProgressBar,
+ fileFactory,
+ ).process(rawRomFiles);
- await romProgressBar.setName('Indexing ROMs');
- const indexedRomFiles = await new ROMIndexer(this.options, romProgressBar)
- .index(romFilesWithHeaders);
+ romProgressBar.setName('Indexing ROMs');
+ const indexedRomFiles = new ROMIndexer(this.options, romProgressBar).index(romFilesWithHeaders);
- await romProgressBar.setName(romScannerProgressBarName); // reset
- await romProgressBar.doneItems(romFilesWithHeaders.length, 'file', 'found');
- await romProgressBar.freeze();
+ romProgressBar.setName(romScannerProgressBarName); // reset
+ romProgressBar.doneItems(romFilesWithHeaders.length, 'file', 'found');
+ romProgressBar.freeze();
return indexedRomFiles;
}
- private async processPatchScanner(): Promise {
+ private async processPatchScanner(fileFactory: FileFactory): Promise {
if (!this.options.getPatchFileCount()) {
return [];
}
- const progressBar = await this.logger.addProgressBar('Scanning for patches');
- const patches = await new PatchScanner(this.options, progressBar).scan();
- await progressBar.doneItems(patches.length, 'patch', 'found');
- await progressBar.freeze();
+ const progressBar = this.logger.addProgressBar('Scanning for patches');
+ const patches = await new PatchScanner(this.options, progressBar, fileFactory).scan();
+ progressBar.doneItems(patches.length, 'patch', 'found');
+ progressBar.freeze();
return patches;
}
private async generateCandidates(
progressBar: ProgressBar,
+ fileFactory: FileFactory,
dat: DAT,
indexedRoms: IndexedFiles,
patches: Patch[],
@@ -392,30 +456,34 @@ export default class Igir {
const patchedCandidates = await new CandidatePatchGenerator(progressBar)
.generate(dat, candidates, patches);
- const preferredCandidates = await new CandidatePreferer(this.options, progressBar)
+ const preferredCandidates = new CandidatePreferer(this.options, progressBar)
.prefer(dat, patchedCandidates);
const extensionCorrectedCandidates = await new CandidateExtensionCorrector(
this.options,
progressBar,
+ fileFactory,
).correct(dat, preferredCandidates);
// Delay calculating checksums for {@link ArchiveFile}s until after {@link CandidatePreferer}
// for efficiency
- const hashedCandidates = await new CandidateArchiveFileHasher(this.options, progressBar)
- .hash(dat, extensionCorrectedCandidates);
+ const hashedCandidates = await new CandidateArchiveFileHasher(
+ this.options,
+ progressBar,
+ fileFactory,
+ ).hash(dat, extensionCorrectedCandidates);
- const postProcessedCandidates = await new CandidatePostProcessor(this.options, progressBar)
+ const postProcessedCandidates = new CandidatePostProcessor(this.options, progressBar)
.process(dat, hashedCandidates);
- const invalidCandidates = await new CandidateValidator(progressBar)
+ const invalidCandidates = new CandidateValidator(progressBar)
.validate(dat, postProcessedCandidates);
if (invalidCandidates.length > 0) {
// Return zero candidates if any candidates failed to validate
return new Map();
}
- await new CandidateMergeSplitValidator(this.options, progressBar)
+ new CandidateMergeSplitValidator(this.options, progressBar)
.validate(dat, postProcessedCandidates);
return new CandidateCombiner(this.options, progressBar)
@@ -457,11 +525,15 @@ export default class Igir {
return;
}
- const progressBar = await this.logger.addProgressBar('Deleting moved files');
+ const progressBar = this.logger.addProgressBar('Deleting moved files');
const deletedFilePaths = await new MovedROMDeleter(progressBar)
.delete(rawRomFiles, movedRomsToDelete, datsToWrittenFiles);
- await progressBar.doneItems(deletedFilePaths.length, 'moved file', 'deleted');
- await progressBar.freeze();
+ progressBar.doneItems(deletedFilePaths.length, 'moved file', 'deleted');
+ if (deletedFilePaths.length > 0) {
+ progressBar.freeze();
+ } else {
+ progressBar.delete();
+ }
}
private async processOutputCleaner(
@@ -475,13 +547,13 @@ export default class Igir {
return [];
}
- const progressBar = await this.logger.addProgressBar('Cleaning output directory');
+ const progressBar = this.logger.addProgressBar('Cleaning output directory');
const uniqueDirsToClean = dirsToClean.reduce(ArrayPoly.reduceUnique(), []);
const writtenFilesToExclude = [...datsToWrittenFiles.values()].flat();
const filesCleaned = await new DirectoryCleaner(this.options, progressBar)
.clean(uniqueDirsToClean, writtenFilesToExclude);
- await progressBar.doneItems(filesCleaned.length, 'file', 'recycled');
- await progressBar.freeze();
+ progressBar.doneItems(filesCleaned.length, 'file', 'recycled');
+ progressBar.freeze();
return filesCleaned;
}
@@ -494,7 +566,7 @@ export default class Igir {
return;
}
- const reportProgressBar = await this.logger.addProgressBar('Generating report', ProgressBarSymbol.WRITING);
+ const reportProgressBar = this.logger.addProgressBar('Generating report', ProgressBarSymbol.WRITING);
await new ReportGenerator(this.options, reportProgressBar).generate(
scannedRomFiles,
cleanedOutputFiles,
diff --git a/src/keyedMutex.ts b/src/keyedMutex.ts
new file mode 100644
index 000000000..6a1188449
--- /dev/null
+++ b/src/keyedMutex.ts
@@ -0,0 +1,58 @@
+import { Mutex } from 'async-mutex';
+
+/**
+ * Wrapper for `async-mutex` {@link Mutex}es to run code exclusively for a key.
+ */
+export default class KeyedMutex {
+ private readonly keyMutexes = new Map();
+
+ private readonly keyMutexesMutex = new Mutex();
+
+ private keyMutexesLru: Set = new Set();
+
+ private readonly maxSize?: number;
+
+ constructor(maxSize?: number) {
+ this.maxSize = maxSize;
+ }
+
+ /**
+ * Run a {@link runnable} exclusively across all keys.
+ */
+ async runExclusiveGlobally(
+ runnable: () => V | Promise,
+ ): Promise {
+ return this.keyMutexesMutex.runExclusive(runnable);
+ }
+
+ /**
+ * Run a {@link runnable} exclusively for the given {@link key}.
+ */
+ async runExclusiveForKey(
+ key: string,
+ runnable: () => V | Promise,
+ ): Promise {
+ const keyMutex = await this.runExclusiveGlobally(() => {
+ if (!this.keyMutexes.has(key)) {
+ this.keyMutexes.set(key, new Mutex());
+
+ // Expire least recently used keys
+ [...this.keyMutexesLru]
+ .filter((lruKey) => !this.keyMutexes.get(lruKey)?.isLocked())
+ .slice(this.maxSize ?? Number.MAX_SAFE_INTEGER)
+ .forEach((lruKey) => {
+ this.keyMutexes.delete(lruKey);
+ this.keyMutexesLru.delete(lruKey);
+ });
+ }
+
+ // Mark this key as recently used
+ this.keyMutexesLru.delete(key);
+ this.keyMutexesLru = new Set([key, ...this.keyMutexesLru]);
+
+ return this.keyMutexes.get(key) as Mutex;
+ });
+
+ return keyMutex.runExclusive(runnable);
+ }
+}
diff --git a/src/modules/argumentsParser.ts b/src/modules/argumentsParser.ts
index cdab0f0fe..2b28a01c3 100644
--- a/src/modules/argumentsParser.ts
+++ b/src/modules/argumentsParser.ts
@@ -15,7 +15,7 @@ import Options, {
FixExtension,
GameSubdirMode,
InputChecksumArchivesMode,
- MergeMode,
+ MergeMode, PreferRevision,
} from '../types/options.js';
import PatchFactory from '../types/patches/patchFactory.js';
@@ -79,7 +79,7 @@ export default class ArgumentsParser {
const groupRomZip = 'zip command options:';
const groupRomLink = 'link command options:';
const groupRomHeader = 'ROM header options:';
- const groupRomSet = 'ROM set options:';
+ const groupRomSet = 'ROM set options (requires DATs):';
const groupRomFiltering = 'ROM filtering options:';
const groupRomPriority = 'One game, one ROM (1G1R) options:';
const groupReport = 'report command options:';
@@ -87,11 +87,10 @@ export default class ArgumentsParser {
// Add every command to a yargs object, recursively, resulting in the ability to specify
// multiple commands
- const commands: [string, string | boolean][] = [
+ const commands = [
['copy', 'Copy ROM files from the input to output directory'],
['move', 'Move ROM files from the input to output directory'],
['link', 'Create links in the output directory to ROM files in the input directory'],
- ['symlink', false],
['extract', 'Extract ROM files in archives when copying or moving'],
['zip', 'Create zip archives of ROMs when copying or moving'],
['test', 'Test ROMs for accuracy after writing them to the output directory'],
@@ -102,9 +101,9 @@ export default class ArgumentsParser {
];
const mutuallyExclusiveCommands = [
// Write commands
- ['copy', 'move', 'link', 'symlink'],
+ ['copy', 'move', 'link'],
// Archive manipulation commands
- ['link', 'symlink', 'extract', 'zip'],
+ ['link', 'extract', 'zip'],
// DAT writing commands
['dir2dat', 'fixdat'],
];
@@ -124,18 +123,10 @@ export default class ArgumentsParser {
return !incompatibleCommands.includes(command);
})
.forEach(([command, description]) => {
- if (typeof description === 'string') {
- yargsObj.command(command, description, (yargsSubObj) => addCommands(
- yargsSubObj,
- [...previousCommands, command],
- ));
- } else {
- // A deprecation message should be printed elsewhere
- yargsObj.command(command, false, (yargsSubObj) => addCommands(
- yargsSubObj,
- [...previousCommands, command],
- ));
- }
+ yargsObj.command(command, description, (yargsSubObj) => addCommands(
+ yargsSubObj,
+ [...previousCommands, command],
+ ));
});
if (previousCommands.length === 0) {
@@ -149,10 +140,6 @@ export default class ArgumentsParser {
middlewareArgv._ = middlewareArgv._.reduce(ArrayPoly.reduceUnique(), []);
}, true)
.check((checkArgv) => {
- if (checkArgv.help) {
- return true;
- }
-
['extract', 'zip'].forEach((command) => {
if (checkArgv._.includes(command) && ['copy', 'move'].every((write) => !checkArgv._.includes(write))) {
throw new ExpectedError(`Command "${command}" also requires the commands copy or move`);
@@ -160,7 +147,7 @@ export default class ArgumentsParser {
});
['test', 'clean'].forEach((command) => {
- if (checkArgv._.includes(command) && ['copy', 'move', 'link', 'symlink'].every((write) => !checkArgv._.includes(write))) {
+ if (checkArgv._.includes(command) && ['copy', 'move', 'link'].every((write) => !checkArgv._.includes(write))) {
throw new ExpectedError(`Command "${command}" requires one of the commands: copy, move, or link`);
}
});
@@ -192,7 +179,7 @@ export default class ArgumentsParser {
requiresArg: true,
})
.check((checkArgv) => {
- const needInput = ['copy', 'move', 'link', 'symlink', 'extract', 'zip', 'test', 'dir2dat', 'fixdat'].filter((command) => checkArgv._.includes(command));
+ const needInput = ['copy', 'move', 'link', 'extract', 'zip', 'test', 'dir2dat', 'fixdat'].filter((command) => checkArgv._.includes(command));
if (!checkArgv.input && needInput.length > 0) {
// TODO(cememr): print help message
throw new ExpectedError(`Missing required argument for command${needInput.length !== 1 ? 's' : ''} ${needInput.join(', ')}: --input `);
@@ -206,7 +193,22 @@ export default class ArgumentsParser {
type: 'array',
requiresArg: true,
})
- .option('input-min-checksum', {
+ .option('input-checksum-quick', {
+ group: groupRomInput,
+ description: 'Only read checksums from archive headers, don\'t decompress to calculate',
+ type: 'boolean',
+ })
+ .check((checkArgv) => {
+ // Re-implement `conflicts: 'input-checksum-min'`, which isn't possible with a default value
+ if (checkArgv['input-checksum-quick'] && checkArgv['input-checksum-min'] !== ChecksumBitmask[ChecksumBitmask.CRC32].toUpperCase()) {
+ throw new ExpectedError('Arguments input-checksum-quick and input-checksum-min are mutually exclusive');
+ }
+ if (checkArgv['input-checksum-quick'] && checkArgv['input-checksum-max']) {
+ throw new ExpectedError('Arguments input-checksum-quick and input-checksum-max are mutually exclusive');
+ }
+ return true;
+ })
+ .option('input-checksum-min', {
group: groupRomInput,
description: 'The minimum checksum level to calculate and use for matching',
choices: Object.keys(ChecksumBitmask)
@@ -217,6 +219,28 @@ export default class ArgumentsParser {
requiresArg: true,
default: ChecksumBitmask[ChecksumBitmask.CRC32].toUpperCase(),
})
+ .option('input-checksum-max', {
+ group: groupRomInput,
+ description: 'The maximum checksum level to calculate and use for matching',
+ choices: Object.keys(ChecksumBitmask)
+ .filter((bitmask) => Number.isNaN(Number(bitmask)))
+ .filter((bitmask) => ChecksumBitmask[bitmask as keyof typeof ChecksumBitmask] > 0)
+ .map((bitmask) => bitmask.toUpperCase()),
+ coerce: ArgumentsParser.getLastValue, // don't allow string[] values
+ requiresArg: true,
+ })
+ .check((checkArgv) => {
+ const options = Options.fromObject(checkArgv);
+ const inputChecksumMin = options.getInputChecksumMin();
+ const inputChecksumMax = options.getInputChecksumMax();
+ if (inputChecksumMin !== undefined
+ && inputChecksumMax !== undefined
+ && inputChecksumMin > inputChecksumMax
+ ) {
+ throw new ExpectedError('Invalid --input-checksum-min & --input-checksum-max, the min must be less than the max');
+ }
+ return true;
+ })
.option('input-checksum-archives', {
group: groupRomInput,
description: 'Calculate checksums of archive files themselves, allowing them to match files in DATs',
@@ -257,15 +281,6 @@ export default class ArgumentsParser {
coerce: ArgumentsParser.readRegexFile,
requiresArg: true,
})
- .option('dat-regex', {
- type: 'string',
- coerce: (val) => {
- this.logger.warn('the \'--dat-regex\' option is deprecated, use \'--dat-name-regex\' instead');
- return ArgumentsParser.readRegexFile(val);
- },
- requiresArg: true,
- hidden: true,
- })
.option('dat-name-regex-exclude', {
group: groupDatInput,
description: 'Regular expression of DAT names to exclude from processing',
@@ -273,15 +288,6 @@ export default class ArgumentsParser {
coerce: ArgumentsParser.readRegexFile,
requiresArg: true,
})
- .option('dat-regex-exclude', {
- type: 'string',
- coerce: (val) => {
- this.logger.warn('the \'--dat-regex-exclude\' option is deprecated, use \'--dat-name-regex-exclude\' instead');
- return ArgumentsParser.readRegexFile(val);
- },
- requiresArg: true,
- hidden: true,
- })
.option('dat-description-regex', {
group: groupDatInput,
description: 'Regular expression of DAT descriptions to process',
@@ -333,17 +339,6 @@ export default class ArgumentsParser {
requiresArg: true,
})
- .option('fixdat', {
- type: 'boolean',
- coerce: (val: boolean) => {
- this.logger.warn('the \'--fixdat\' option is deprecated, use the \'fixdat\' command instead');
- return val;
- },
- implies: 'dat',
- deprecated: true,
- hidden: true,
- })
-
.option('output', {
group: groupRomOutputPath,
alias: 'o',
@@ -437,10 +432,7 @@ export default class ArgumentsParser {
type: 'boolean',
})
.check((checkArgv) => {
- if (checkArgv.help) {
- return true;
- }
- const needOutput = ['copy', 'move', 'link', 'symlink', 'extract', 'zip', 'clean'].filter((command) => checkArgv._.includes(command));
+ const needOutput = ['copy', 'move', 'link', 'extract', 'zip', 'clean'].filter((command) => checkArgv._.includes(command));
if (!checkArgv.output && needOutput.length > 0) {
// TODO(cememr): print help message
throw new ExpectedError(`Missing required argument for command${needOutput.length !== 1 ? 's' : ''} ${needOutput.join(', ')}: --output `);
@@ -468,9 +460,6 @@ export default class ArgumentsParser {
type: 'boolean',
})
.check((checkArgv) => {
- if (checkArgv.help) {
- return true;
- }
const needClean = ['clean-exclude', 'clean-backup', 'clean-dry-run'].filter((option) => checkArgv[option]);
if (!checkArgv._.includes('clean') && needClean.length > 0) {
// TODO(cememr): print help message
@@ -482,7 +471,7 @@ export default class ArgumentsParser {
.option('zip-exclude', {
group: groupRomZip,
alias: 'Z',
- description: 'Glob pattern of files to exclude from zipping',
+ description: 'Glob pattern of ROM filenames to exclude from zipping',
type: 'string',
coerce: ArgumentsParser.getLastValue, // don't allow string[] values
requiresArg: true,
@@ -493,9 +482,6 @@ export default class ArgumentsParser {
type: 'boolean',
})
.check((checkArgv) => {
- if (checkArgv.help) {
- return true;
- }
const needZip = ['zip-exclude', 'zip-dat-name'].filter((option) => checkArgv[option]);
if (!checkArgv._.includes('zip') && needZip.length > 0) {
throw new ExpectedError(`Missing required command for option${needZip.length !== 1 ? 's' : ''} ${needZip.join(', ')}: zip`);
@@ -508,15 +494,6 @@ export default class ArgumentsParser {
description: 'Creates symbolic links instead of hard links',
type: 'boolean',
})
- .middleware((middlewareArgv) => {
- if (middlewareArgv._.includes('symlink')) {
- this.logger.warn('the \'symlink\' command is deprecated, use \'link --symlink\' instead');
- if (middlewareArgv.symlink === undefined) {
- // eslint-disable-next-line no-param-reassign
- middlewareArgv.symlink = true;
- }
- }
- }, true)
.option('symlink-relative', {
group: groupRomLink,
description: 'Create symlinks as relative to the target path, as opposed to absolute',
@@ -524,11 +501,8 @@ export default class ArgumentsParser {
implies: 'symlink',
})
.check((checkArgv) => {
- if (checkArgv.help) {
- return true;
- }
const needLinkCommand = ['symlink'].filter((option) => checkArgv[option]);
- if (!checkArgv._.includes('link') && !checkArgv._.includes('symlink') && needLinkCommand.length > 0) {
+ if (!checkArgv._.includes('link') && needLinkCommand.length > 0) {
throw new ExpectedError(`Missing required command for option${needLinkCommand.length !== 1 ? 's' : ''} ${needLinkCommand.join(', ')}: link`);
}
return true;
@@ -536,7 +510,7 @@ export default class ArgumentsParser {
.option('header', {
group: groupRomHeader,
- description: 'Glob pattern of files to force header processing for',
+ description: 'Glob pattern of input filenames to force header processing for',
type: 'string',
coerce: ArgumentsParser.getLastValue, // don't allow string[] values
requiresArg: true,
@@ -567,10 +541,30 @@ export default class ArgumentsParser {
requiresArg: true,
default: MergeMode[MergeMode.FULLNONMERGED].toLowerCase(),
})
+ .check((checkArgv) => {
+ // Re-implement `implies: 'dat'`, which isn't possible with a default value
+ if (checkArgv['merge-roms'] !== MergeMode[MergeMode.FULLNONMERGED].toLowerCase() && !checkArgv.dat) {
+ throw new ExpectedError('Missing dependent arguments:\n merge-roms -> dat');
+ }
+ return true;
+ })
+ .option('exclude-disks', {
+ group: groupRomSet,
+ description: 'Exclude CHD disks in DATs from processing & writing',
+ type: 'boolean',
+ implies: 'dat',
+ })
+ .option('allow-excess-sets', {
+ group: groupRomSet,
+ description: 'Allow writing archives that have excess files when not extracting or zipping',
+ type: 'boolean',
+ implies: 'dat',
+ })
.option('allow-incomplete-sets', {
group: groupRomSet,
description: 'Allow writing games that don\'t have all of their ROMs',
type: 'boolean',
+ implies: 'dat',
})
.option('filter-regex', {
@@ -604,16 +598,6 @@ export default class ArgumentsParser {
}
return true;
})
- .option('language-filter', {
- type: 'string',
- coerce: (val: string) => {
- this.logger.warn('the \'--language-filter\' option is deprecated, use \'--filter-language\' instead');
- return val.split(',');
- },
- requiresArg: true,
- deprecated: true,
- hidden: true,
- })
.option('filter-region', {
group: groupRomFiltering,
alias: 'R',
@@ -628,16 +612,6 @@ export default class ArgumentsParser {
throw new ExpectedError(`Invalid --filter-region region${invalidRegions.length !== 1 ? 's' : ''}: ${invalidRegions.join(', ')}`);
}
return true;
- })
- .option('region-filter', {
- type: 'string',
- coerce: (val: string) => {
- this.logger.warn('the \'--region-filter\' option is deprecated, use \'--filter-region\' instead');
- return val.split(',');
- },
- requiresArg: true,
- deprecated: true,
- hidden: true,
});
[
['bios', 'BIOS files'],
@@ -664,25 +638,23 @@ export default class ArgumentsParser {
type: 'boolean',
});
([
- ['debug', 'debug ROMs', false],
- ['demo', 'demo ROMs', false],
- ['beta', 'beta ROMs', false],
- ['sample', 'sample ROMs', false],
- ['prototype', 'prototype ROMs', false],
- ['test-roms', 'test ROMs', true],
- ['program', 'program application ROMs', false],
- ['aftermarket', 'aftermarket ROMs', false],
- ['homebrew', 'homebrew ROMs', false],
- ['unverified', 'unverified ROMs', false],
- ['bad', 'bad ROM dumps', false],
- ] satisfies [string, string, boolean][]).forEach(([key, description, hidden]) => {
+ ['debug', 'debug ROMs'],
+ ['demo', 'demo ROMs'],
+ ['beta', 'beta ROMs'],
+ ['sample', 'sample ROMs'],
+ ['prototype', 'prototype ROMs'],
+ ['program', 'program application ROMs'],
+ ['aftermarket', 'aftermarket ROMs'],
+ ['homebrew', 'homebrew ROMs'],
+ ['unverified', 'unverified ROMs'],
+ ['bad', 'bad ROM dumps'],
+ ]).forEach(([key, description]) => {
yargsParser
.option(`no-${key}`, {
group: groupRomFiltering,
description: `Filter out ${description}, opposite of --only-${key}`,
type: 'boolean',
conflicts: [`only-${key}`],
- hidden,
})
.option(`only-${key}`, {
type: 'boolean',
@@ -690,22 +662,6 @@ export default class ArgumentsParser {
hidden: true,
});
});
- yargsParser.middleware((middlewareArgv) => {
- if (middlewareArgv['no-test-roms'] === true) {
- this.logger.warn('the \'--no-test-roms\' option is deprecated, use \'--no-program\' instead');
- if (middlewareArgv.noProgram === undefined) {
- // eslint-disable-next-line no-param-reassign
- middlewareArgv.noProgram = true;
- }
- }
- if (middlewareArgv['only-test-roms'] === true) {
- this.logger.warn('the \'--only-test-roms\' option is deprecated, use \'--only-program\' instead');
- if (middlewareArgv.onlyProgram === undefined) {
- // eslint-disable-next-line no-param-reassign
- middlewareArgv.onlyProgram = true;
- }
- }
- }, true);
yargsParser
.option('single', {
@@ -774,18 +730,14 @@ export default class ArgumentsParser {
}
return true;
})
- .option('prefer-revision-newer', {
+ .option('prefer-revision', {
group: groupRomPriority,
- description: 'Prefer newer ROM revisions over older',
- type: 'boolean',
- conflicts: ['prefer-revision-older'],
- implies: 'single',
- })
- .option('prefer-revision-older', {
- group: groupRomPriority,
- description: 'Prefer older ROM revisions over newer',
- type: 'boolean',
- conflicts: ['prefer-revision-newer'],
+ description: 'Prefer older or newer revisions, versions, or ring codes',
+ choices: Object.keys(PreferRevision)
+ .filter((mode) => Number.isNaN(Number(mode)))
+ .map((mode) => mode.toLowerCase()),
+ coerce: ArgumentsParser.getLastValue, // don't allow string[] values
+ requiresArg: true,
implies: 'single',
})
.option('prefer-retail', {
@@ -794,20 +746,6 @@ export default class ArgumentsParser {
type: 'boolean',
implies: 'single',
})
- .option('prefer-ntsc', {
- group: groupRomPriority,
- description: 'Prefer NTSC ROMs over others',
- type: 'boolean',
- conflicts: 'prefer-pal',
- implies: 'single',
- })
- .option('prefer-pal', {
- group: groupRomPriority,
- description: 'Prefer PAL ROMs over others',
- type: 'boolean',
- conflicts: 'prefer-ntsc',
- implies: 'single',
- })
.option('prefer-parent', {
group: groupRomPriority,
description: 'Prefer parent ROMs over clones',
@@ -871,12 +809,12 @@ export default class ArgumentsParser {
})
.option('disable-cache', {
group: groupHelpDebug,
- description: 'Disable the file checksum cache',
+ description: 'Disable loading or saving the cache file',
type: 'boolean',
})
.option('cache-path', {
group: groupHelpDebug,
- description: 'Location for the file checksum cache file',
+ description: 'Location for the cache file',
type: 'string',
coerce: ArgumentsParser.getLastValue, // don't allow string[] values
requiresArg: true,
@@ -931,7 +869,7 @@ Advanced usage:
{datDescription} The description of the DAT that contains the ROM
{region} The region of the ROM release (e.g. "USA"), each ROM can have multiple
{language} The language of the ROM release (e.g. "En"), each ROM can have multiple
- {gameType} The type of the game (e.g. "Retail", "Demo", "Prototype")
+ {type} The type of the game (e.g. "Retail", "Demo", "Prototype")
{genre} The DAT-defined genre of the game
{inputDirname} The input file's dirname
diff --git a/src/modules/candidateArchiveFileHasher.ts b/src/modules/candidateArchiveFileHasher.ts
index 54a602243..8b90052ee 100644
--- a/src/modules/candidateArchiveFileHasher.ts
+++ b/src/modules/candidateArchiveFileHasher.ts
@@ -1,16 +1,11 @@
-import { Semaphore } from 'async-mutex';
-
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
-import ElasticSemaphore from '../elasticSemaphore.js';
-import Defaults from '../globals/defaults.js';
-import FsPoly from '../polyfill/fsPoly.js';
+import DriveSemaphore from '../driveSemaphore.js';
import DAT from '../types/dats/dat.js';
import Parent from '../types/dats/parent.js';
import ArchiveFile from '../types/files/archives/archiveFile.js';
import FileFactory from '../types/files/fileFactory.js';
import Options from '../types/options.js';
import ReleaseCandidate from '../types/releaseCandidate.js';
-import ROMWithFiles from '../types/romWithFiles.js';
import Module from './module.js';
/**
@@ -19,23 +14,22 @@ import Module from './module.js';
* {@link CandidatePreferer}.
*/
export default class CandidateArchiveFileHasher extends Module {
- private static readonly THREAD_SEMAPHORE = new Semaphore(Number.MAX_SAFE_INTEGER);
-
- // WARN(cemmer): there is an undocumented semaphore max value that can be used, the full
- // 4,700,372,992 bytes of a DVD+R will cause runExclusive() to never run or return.
- private static readonly FILESIZE_SEMAPHORE = new ElasticSemaphore(
- Defaults.MAX_READ_WRITE_CONCURRENT_KILOBYTES,
+ private static readonly DRIVE_SEMAPHORE = new DriveSemaphore(
+ Number.MAX_SAFE_INTEGER,
);
private readonly options: Options;
- constructor(options: Options, progressBar: ProgressBar) {
+ private readonly fileFactory: FileFactory;
+
+ constructor(options: Options, progressBar: ProgressBar, fileFactory: FileFactory) {
super(progressBar, CandidateArchiveFileHasher.name);
this.options = options;
+ this.fileFactory = fileFactory;
// This will be the same value globally, but we can't know the value at file import time
- if (options.getReaderThreads() < CandidateArchiveFileHasher.THREAD_SEMAPHORE.getValue()) {
- CandidateArchiveFileHasher.THREAD_SEMAPHORE.setValue(options.getReaderThreads());
+ if (options.getReaderThreads() < CandidateArchiveFileHasher.DRIVE_SEMAPHORE.getValue()) {
+ CandidateArchiveFileHasher.DRIVE_SEMAPHORE.setValue(options.getReaderThreads());
}
}
@@ -67,8 +61,8 @@ export default class CandidateArchiveFileHasher extends Module {
}
this.progressBar.logTrace(`${dat.getNameShort()}: generating ${archiveFileCount.toLocaleString()} hashed ArchiveFile candidate${archiveFileCount !== 1 ? 's' : ''}`);
- await this.progressBar.setSymbol(ProgressBarSymbol.HASHING);
- await this.progressBar.reset(archiveFileCount);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_HASHING);
+ this.progressBar.reset(archiveFileCount);
const hashedParentsToCandidates = this.hashArchiveFiles(dat, parentsToCandidates);
@@ -91,45 +85,46 @@ export default class CandidateArchiveFileHasher extends Module {
return romWithFiles;
}
- return CandidateArchiveFileHasher.THREAD_SEMAPHORE.runExclusive(async () => {
- const totalKilobytes = await FsPoly.size(inputFile.getFilePath()) / 1024;
- return CandidateArchiveFileHasher.FILESIZE_SEMAPHORE.runExclusive(async () => {
- await this.progressBar.incrementProgress();
+ const outputFile = romWithFiles.getOutputFile();
+ if (inputFile.equals(outputFile)) {
+ // There's no need to calculate the checksum, {@link CandidateWriter} will skip
+ // writing over itself
+ return romWithFiles;
+ }
+
+ return CandidateArchiveFileHasher.DRIVE_SEMAPHORE.runExclusive(
+ inputFile,
+ async () => {
+ this.progressBar.incrementProgress();
const waitingMessage = `${inputFile.toString()} ...`;
this.progressBar.addWaitingMessage(waitingMessage);
this.progressBar.logTrace(`${dat.getNameShort()}: ${parent.getName()}: calculating checksums for: ${inputFile.toString()}`);
- const hashedInputFile = await FileFactory.archiveFileFrom(
+ const hashedInputFile = await this.fileFactory.archiveFileFrom(
inputFile.getArchive(),
inputFile.getChecksumBitmask(),
);
// {@link CandidateGenerator} would have copied undefined values from the input
// file, so we need to modify the expected output file as well for testing
- const hashedOutputFile = romWithFiles.getOutputFile().withProps({
+ const hashedOutputFile = outputFile.withProps({
size: hashedInputFile.getSize(),
crc32: hashedInputFile.getCrc32(),
md5: hashedInputFile.getMd5(),
sha1: hashedInputFile.getSha1(),
sha256: hashedInputFile.getSha256(),
});
- const hashedRomWithFiles = new ROMWithFiles(
- romWithFiles.getRom(),
- hashedInputFile,
- hashedOutputFile,
- );
+ const hashedRomWithFiles = romWithFiles
+ .withInputFile(hashedInputFile)
+ .withOutputFile(hashedOutputFile);
this.progressBar.removeWaitingMessage(waitingMessage);
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
return hashedRomWithFiles;
- }, totalKilobytes);
- });
+ },
+ );
}));
- return new ReleaseCandidate(
- releaseCandidate.getGame(),
- releaseCandidate.getRelease(),
- hashedRomsWithFiles,
- );
+ return releaseCandidate.withRomsWithFiles(hashedRomsWithFiles);
}));
return [parent, hashedReleaseCandidates];
diff --git a/src/modules/candidateCombiner.ts b/src/modules/candidateCombiner.ts
index d934e8919..8020a5f24 100644
--- a/src/modules/candidateCombiner.ts
+++ b/src/modules/candidateCombiner.ts
@@ -8,7 +8,6 @@ import ROM from '../types/dats/rom.js';
import ArchiveEntry from '../types/files/archives/archiveEntry.js';
import Options from '../types/options.js';
import ReleaseCandidate from '../types/releaseCandidate.js';
-import ROMWithFiles from '../types/romWithFiles.js';
import Module from './module.js';
/**
@@ -26,10 +25,10 @@ export default class CandidateCombiner extends Module {
/**
* Combine the candidates.
*/
- async combine(
+ combine(
dat: DAT,
parentsToCandidates: Map,
- ): Promise> {
+ ): Map {
if (!this.options.getZipDatName()) {
return parentsToCandidates;
}
@@ -40,8 +39,8 @@ export default class CandidateCombiner extends Module {
}
this.progressBar.logTrace(`${dat.getNameShort()}: generating consolidated candidate`);
- await this.progressBar.setSymbol(ProgressBarSymbol.COMBINING_ALL);
- await this.progressBar.reset(parentsToCandidates.size);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_COMBINING);
+ this.progressBar.reset(parentsToCandidates.size);
const game = CandidateCombiner.buildGame(dat, parentsToCandidates);
const parent = new Parent(game);
@@ -104,11 +103,7 @@ export default class CandidateCombiner extends Module {
));
}
- return new ROMWithFiles(
- romWithFiles.getRom(),
- romWithFiles.getInputFile(),
- outputEntry,
- );
+ return romWithFiles.withOutputFile(outputEntry);
}));
return new ReleaseCandidate(game, undefined, romsWithFiles);
diff --git a/src/modules/candidateExtensionCorrector.ts b/src/modules/candidateExtensionCorrector.ts
index e31193dd5..8b0cde374 100644
--- a/src/modules/candidateExtensionCorrector.ts
+++ b/src/modules/candidateExtensionCorrector.ts
@@ -7,7 +7,9 @@ import DAT from '../types/dats/dat.js';
import Parent from '../types/dats/parent.js';
import ROM from '../types/dats/rom.js';
import ArchiveEntry from '../types/files/archives/archiveEntry.js';
-import FileCache from '../types/files/fileCache.js';
+import Chd from '../types/files/archives/chd/chd.js';
+import FileFactory from '../types/files/fileFactory.js';
+import FileSignature from '../types/files/fileSignature.js';
import Options, { FixExtension } from '../types/options.js';
import OutputFactory from '../types/outputFactory.js';
import ReleaseCandidate from '../types/releaseCandidate.js';
@@ -24,9 +26,12 @@ export default class CandidateExtensionCorrector extends Module {
private readonly options: Options;
- constructor(options: Options, progressBar: ProgressBar) {
+ private readonly fileFactory: FileFactory;
+
+ constructor(options: Options, progressBar: ProgressBar, fileFactory: FileFactory) {
super(progressBar, CandidateExtensionCorrector.name);
this.options = options;
+ this.fileFactory = fileFactory;
// This will be the same value globally, but we can't know the value at file import time
if (options.getReaderThreads() < CandidateExtensionCorrector.THREAD_SEMAPHORE.getValue()) {
@@ -51,9 +56,14 @@ export default class CandidateExtensionCorrector extends Module {
.flatMap((releaseCandidate) => releaseCandidate.getRomsWithFiles())
.filter((romWithFiles) => this.romNeedsCorrecting(romWithFiles))
.length;
+ if (romsThatNeedCorrecting === 0) {
+ this.progressBar.logTrace(`${dat.getNameShort()}: no output files need their extension corrected`);
+ return parentsToCandidates;
+ }
+
this.progressBar.logTrace(`${dat.getNameShort()}: correcting ${romsThatNeedCorrecting.toLocaleString()} output file extension${romsThatNeedCorrecting !== 1 ? 's' : ''}`);
- await this.progressBar.setSymbol(ProgressBarSymbol.EXTENSION_CORRECTION);
- await this.progressBar.reset(romsThatNeedCorrecting);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_EXTENSION_CORRECTION);
+ this.progressBar.reset(romsThatNeedCorrecting);
const correctedParentsToCandidates = await this.correctExtensions(dat, parentsToCandidates);
@@ -62,6 +72,16 @@ export default class CandidateExtensionCorrector extends Module {
}
private romNeedsCorrecting(romWithFiles: ROMWithFiles): boolean {
+ if (romWithFiles.getRom().getName().trim() === '') {
+ return true;
+ }
+
+ const inputFile = romWithFiles.getInputFile();
+ if (inputFile instanceof ArchiveEntry && inputFile.getArchive() instanceof Chd) {
+ // Files within CHDs never need extension correction
+ return false;
+ }
+
return this.options.getFixExtension() === FixExtension.ALWAYS
|| (this.options.getFixExtension() === FixExtension.AUTO && (
!this.options.usingDats()
@@ -102,18 +122,12 @@ export default class CandidateExtensionCorrector extends Module {
.withEntryPath(correctedOutputPath.entryPath);
}
- return new ROMWithFiles(
- correctedRom,
- romWithFiles.getInputFile(),
- correctedOutputFile,
- );
+ return romWithFiles
+ .withRom(correctedRom)
+ .withOutputFile(correctedOutputFile);
}));
- return new ReleaseCandidate(
- releaseCandidate.getGame(),
- releaseCandidate.getRelease(),
- hashedRomsWithFiles,
- );
+ return releaseCandidate.withRomsWithFiles(hashedRomsWithFiles);
}));
return [parent, hashedReleaseCandidates];
@@ -142,13 +156,18 @@ export default class CandidateExtensionCorrector extends Module {
}
await CandidateExtensionCorrector.THREAD_SEMAPHORE.runExclusive(async () => {
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
const waitingMessage = `${releaseCandidate.getName()} ...`;
this.progressBar.addWaitingMessage(waitingMessage);
this.progressBar.logTrace(`${dat.getNameShort()}: ${parent.getName()}: correcting extension for: ${romWithFiles.getInputFile()
.toString()}`);
- const romSignature = await FileCache.getOrComputeFileSignature(romWithFiles.getInputFile());
+ let romSignature: FileSignature | undefined;
+ try {
+ romSignature = await this.fileFactory.signatureFrom(romWithFiles.getInputFile());
+ } catch (error) {
+ this.progressBar.logError(`${dat.getNameShort()}: failed to correct file extension for '${romWithFiles.getInputFile()}': ${error}`);
+ }
if (romSignature) {
// ROM file signature found, use the appropriate extension
const { dir, name } = path.parse(correctedRom.getName());
@@ -160,7 +179,7 @@ export default class CandidateExtensionCorrector extends Module {
}
this.progressBar.removeWaitingMessage(waitingMessage);
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
});
return correctedRom;
diff --git a/src/modules/candidateGenerator.ts b/src/modules/candidateGenerator.ts
index 1ccf51b0f..478485e6e 100644
--- a/src/modules/candidateGenerator.ts
+++ b/src/modules/candidateGenerator.ts
@@ -13,6 +13,7 @@ import ROM from '../types/dats/rom.js';
import Archive from '../types/files/archives/archive.js';
import ArchiveEntry from '../types/files/archives/archiveEntry.js';
import ArchiveFile from '../types/files/archives/archiveFile.js';
+import Chd from '../types/files/archives/chd/chd.js';
import Zip from '../types/files/archives/zip.js';
import File from '../types/files/file.js';
import IndexedFiles from '../types/indexedFiles.js';
@@ -57,14 +58,14 @@ export default class CandidateGenerator extends Module {
const parents = dat.getParents();
this.progressBar.logTrace(`${dat.getNameShort()}: generating candidates`);
- await this.progressBar.setSymbol(ProgressBarSymbol.GENERATING);
- await this.progressBar.reset(parents.length);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_GENERATING);
+ this.progressBar.reset(parents.length);
// For each parent, try to generate a parent candidate
await Promise.all(parents.map(async (
parent,
) => CandidateGenerator.THREAD_SEMAPHORE.runExclusive(async () => {
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
const waitingMessage = `${parent.getName()} ...`;
this.progressBar.addWaitingMessage(waitingMessage);
@@ -97,7 +98,7 @@ export default class CandidateGenerator extends Module {
output.set(parent, releaseCandidates);
this.progressBar.removeWaitingMessage(waitingMessage);
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
})));
const size = [...output.values()]
@@ -119,13 +120,19 @@ export default class CandidateGenerator extends Module {
): Promise {
const romsToInputFiles = this.getInputFilesForGame(dat, game, indexedFiles);
+ const gameRoms = [
+ ...game.getRoms(),
+ ...(this.options.getExcludeDisks() ? [] : game.getDisks()),
+ ];
+
// For each Game's ROM, find the matching File
const romFiles = await Promise.all(
- game.getRoms().map(async (rom) => {
- if (!romsToInputFiles.has(rom)) {
+ gameRoms.map(async (rom) => {
+ let inputFile = romsToInputFiles.get(rom);
+ if (inputFile === undefined) {
return [rom, undefined];
}
- let inputFile = romsToInputFiles.get(rom) as File;
+
/**
* WARN(cemmer): {@link inputFile} may not be an exact match for {@link rom}. There are two
* situations we can be in:
@@ -147,10 +154,7 @@ export default class CandidateGenerator extends Module {
|| (inputFile.getSha1() !== undefined && inputFile.getSha1() === rom.getSha1())
|| (inputFile.getSha256() !== undefined && inputFile.getSha256() === rom.getSha256()))
// ...and we shouldn't remove the header
- && !this.options.canRemoveHeader(
- dat,
- path.extname(inputFile.getExtractedFilePath()),
- )
+ && !this.options.canRemoveHeader(path.extname(inputFile.getExtractedFilePath()))
) {
// ...then forget the input file's header, so that we don't later remove it
this.progressBar.logTrace(`${dat.getNameShort()}: ${game.getName()}: not removing header, ignoring that one was found for: ${inputFile.toString()}`);
@@ -173,20 +177,21 @@ export default class CandidateGenerator extends Module {
}
/**
- * If the matched input file is from an archive, and we're not zipping or extracting, then
- * treat the file as "raw" so it can be copied/moved as-is.
- * Matches {@link ROMHeaderProcessor.getFileWithHeader}
+ * If the matched input file is from an archive, and we can raw-copy that entire archive,
+ * then treat the file as "raw" so it can be copied/moved as-is.
*/
if (inputFile instanceof ArchiveEntry
- && !this.options.shouldZipFile(rom.getName())
- && !this.options.shouldExtract()
+ && this.shouldGenerateArchiveFile(dat, game, release, rom, romsToInputFiles)
) {
try {
- // Note: we're delaying checksum calculation for now, {@link CandidateArchiveFileHasher}
- // will handle it later
+ // Note: we're delaying checksum calculations for now,
+ // {@link CandidateArchiveFileHasher} will handle it later
inputFile = new ArchiveFile(
inputFile.getArchive(),
- { checksumBitmask: inputFile.getChecksumBitmask() },
+ {
+ size: await fsPoly.size(inputFile.getFilePath()),
+ checksumBitmask: inputFile.getChecksumBitmask(),
+ },
);
} catch (error) {
this.progressBar.logWarn(`${dat.getNameShort()}: ${game.getName()}: ${error}`);
@@ -210,16 +215,17 @@ export default class CandidateGenerator extends Module {
const foundRomsWithFiles = romFiles
.map(([, romWithFiles]) => romWithFiles)
- .filter(ArrayPoly.filterNotNullish);
+ .filter((romWithFiles) => romWithFiles !== undefined);
if (romFiles.length > 0 && foundRomsWithFiles.length === 0) {
// The Game has ROMs, but none were found
return undefined;
}
- // Ignore the Game if not every File is present
const missingRoms = romFiles
.filter(([, romWithFiles]) => !romWithFiles)
.map(([rom]) => rom);
+
+ // Ignore the Game if not every File is present
if (missingRoms.length > 0 && !this.options.getAllowIncompleteSets()) {
if (foundRomsWithFiles.length > 0) {
this.logMissingRomFiles(dat, game, release, foundRomsWithFiles, missingRoms);
@@ -232,6 +238,15 @@ export default class CandidateGenerator extends Module {
return undefined;
}
+ // If the found files have excess and we aren't allowing it, then return no candidate
+ if (!this.options.shouldZip()
+ && !this.options.shouldExtract()
+ && !this.options.getAllowExcessSets()
+ && this.hasExcessFiles(dat, game, foundRomsWithFiles, indexedFiles)
+ ) {
+ return undefined;
+ }
+
return new ReleaseCandidate(game, release, foundRomsWithFiles);
}
@@ -240,7 +255,12 @@ export default class CandidateGenerator extends Module {
game: Game,
indexedFiles: IndexedFiles,
): Map {
- const romsAndInputFiles = game.getRoms().map((rom) => ([
+ const gameRoms = [
+ ...game.getRoms(),
+ ...(this.options.getExcludeDisks() ? [] : game.getDisks()),
+ ];
+
+ const romsAndInputFiles = gameRoms.map((rom) => ([
rom,
indexedFiles.findFiles(rom) ?? [],
])) satisfies [ROM, File[]][];
@@ -259,30 +279,69 @@ export default class CandidateGenerator extends Module {
// Group this Game's ROMs by the input Archives that contain them
const inputArchivesToRoms = romsAndInputFiles.reduce((map, [rom, files]) => {
files
- .filter((file) => file instanceof ArchiveEntry)
- .map((file): Archive => (file as ArchiveEntry).getArchive())
+ .filter((file): file is ArchiveEntry => file instanceof ArchiveEntry)
+ .map((archive): Archive => archive.getArchive())
.forEach((archive) => {
- const roms = map.get(archive) ?? [];
- roms.push(rom);
// We need to filter out duplicate ROMs because of Games that contain duplicate ROMs, e.g.
// optical media games that have the same track multiple times.
- const uniqueRoms = roms.reduce(ArrayPoly.reduceUnique(), []);
- map.set(archive, uniqueRoms);
+ if (!map.has(archive)) {
+ map.set(archive, new Set());
+ }
+ map.get(archive)?.add(rom);
});
return map;
- }, new Map());
+ }, new Map>());
// Filter to the Archives that contain every ROM in this Game
const archivesWithEveryRom = [...inputArchivesToRoms.entries()]
- .filter(([, roms]) => roms.length === game.getRoms().length)
+ .filter(([inputArchive, roms]) => {
+ if ([...roms].map((rom) => rom.hashCode()).join(',') === gameRoms.map((rom) => rom.hashCode()).join(',')) {
+ return true;
+ }
+ // If there is a CHD with every .bin file, and we're raw-copying it, then assume its .cue
+ // file is accurate
+ return inputArchive instanceof Chd
+ && !gameRoms.some((rom) => this.options.shouldZipRom(rom))
+ && !gameRoms.some((rom) => this.options.shouldExtractRom(rom))
+ && CandidateGenerator.onlyCueFilesMissingFromChd(game, [...roms]);
+ })
.map(([archive]) => archive);
- const archiveWithEveryRom = archivesWithEveryRom.at(0);
+
+ const filesByPath = indexedFiles.getFilesByFilePath();
+ const filteredArchivesWithEveryRom = archivesWithEveryRom
+ .sort((a, b) => {
+ // First, prefer the archive with the least number of entries
+ const aEntries = filesByPath.get(a.getFilePath())?.length ?? 0;
+ const bEntries = filesByPath.get(b.getFilePath())?.length ?? 0;
+ if (aEntries !== bEntries) {
+ return aEntries - bEntries;
+ }
+
+ // Then, prefer archives whose filename contains the game name
+ const aGameName = path.basename(a.getFilePath()).includes(game.getName()) ? 1 : 0;
+ const bGameName = path.basename(b.getFilePath()).includes(game.getName()) ? 1 : 0;
+ return aGameName - bGameName;
+ })
+ // Filter out Archives with excess entries
+ .filter((archive) => {
+ const unusedEntryPaths = this.findArchiveUnusedEntryPaths(
+ archive,
+ romsAndInputFiles.flatMap(([, inputFiles]) => inputFiles),
+ indexedFiles,
+ );
+ if (unusedEntryPaths.length > 0) {
+ this.progressBar.logTrace(`${dat.getNameShort()}: ${game.getName()}: not preferring archive that contains every ROM, plus the excess entries:\n${unusedEntryPaths.map((entryPath) => ` ${entryPath}`).join('\n')}`);
+ }
+ return unusedEntryPaths.length === 0;
+ });
+
+ const archiveWithEveryRom = filteredArchivesWithEveryRom.at(0);
// Do nothing if any of...
if (
// The Game has zero or one ROM, therefore, we don't really care where the file comes from,
// and we should respect any previous sorting of the input files
- game.getRoms().length <= 1
+ gameRoms.length <= 1
// No input archive contains every ROM from this Game
|| archiveWithEveryRom === undefined
// We're extracting files, therefore, we don't really care where the file comes from, and we
@@ -298,13 +357,75 @@ export default class CandidateGenerator extends Module {
// For each of this Game's ROMs, find the matching ArchiveEntry from this Archive
return new Map(romsAndInputFiles.map(([rom, inputFiles]) => {
this.progressBar.logTrace(`${dat.getNameShort()}: ${game.getName()}: preferring input archive that contains every ROM: ${archiveWithEveryRom.getFilePath()}`);
- const archiveEntry = inputFiles.find((
- inputFile,
- ) => inputFile.getFilePath() === archiveWithEveryRom.getFilePath()) as File;
- return [rom, archiveEntry];
+ let archiveEntry = inputFiles
+ .find((inputFile) => inputFile.getFilePath() === archiveWithEveryRom.getFilePath());
+
+ if (!archiveEntry
+ && rom.getName().toLowerCase().endsWith('.cue')
+ && archiveWithEveryRom instanceof Chd
+ ) {
+ // We assumed this CHD was fine above, find its .cue file
+ archiveEntry = filesByPath.get(archiveWithEveryRom.getFilePath())
+ ?.find((file) => file.getExtractedFilePath().toLowerCase().endsWith('.cue'));
+ }
+
+ return [rom, archiveEntry as File];
}));
}
+ private shouldGenerateArchiveFile(
+ dat: DAT,
+ game: Game,
+ release: Release | undefined,
+ rom: ROM,
+ romsToInputFiles: Map,
+ ): boolean {
+ if ([...romsToInputFiles.values()].some((inputFile) => inputFile.getFileHeader() !== undefined
+ || inputFile.getPatch() !== undefined)
+ ) {
+ // At least one output file won't exactly match its input file, don't generate an archive
+ // file
+ return false;
+ }
+
+ if (romsToInputFiles.get(rom) instanceof ArchiveEntry
+ && !this.options.shouldZipRom(rom)
+ && !this.options.shouldExtractRom(rom)
+ ) {
+ // This ROM's input file is already archived, and we're not [re-]zipping or extracting, so
+ // we want to leave it as-is. We'll check later if the input archive has excess files.
+ return true;
+ }
+
+ // TODO(cemmer): delay this until after CandidatePatchGenerator
+ if (this.options.getPatchFileCount() === 0
+ && !this.options.getZipDatName()
+ && [...romsToInputFiles.entries()]
+ .every(([inputRom, inputFile]) => inputFile instanceof ArchiveEntry
+ && inputFile.getArchive() instanceof Zip
+ && this.options.shouldZipRom(inputRom)
+ && OutputFactory.getPath(
+ this.options,
+ dat,
+ game,
+ release,
+ inputRom,
+ inputFile,
+ ).entryPath === inputFile.getExtractedFilePath())
+ && [...romsToInputFiles.values()]
+ .map((inputFile) => inputFile.getFilePath())
+ .reduce(ArrayPoly.reduceUnique(), [])
+ .length === 1
+ ) {
+ // Every ROM should be zipped, and every input file is already in the same zip, and the
+ // archive entry paths match, so it's safe to copy the zip as-is
+ return true;
+ }
+
+ // Return false by default
+ return false;
+ }
+
private async getOutputFile(
dat: DAT,
game: Game,
@@ -344,7 +465,7 @@ export default class CandidateGenerator extends Module {
}
// Determine the output file type
- if (this.options.shouldZipFile(rom.getName())) {
+ if (this.options.shouldZipRom(rom) && !(inputFile instanceof ArchiveFile)) {
// Should zip, return an archive entry within an output zip
return ArchiveEntry.entryOf({
archive: new Zip(outputFilePath),
@@ -425,4 +546,102 @@ export default class CandidateGenerator extends Module {
}
return hasConflict;
}
+
+ /**
+ * Given a {@link Game}, return true if all conditions are met:
+ * - The {@link Game} only has .bin and .cue files
+ * - Out of the {@link ROM}s that were found in an input directory for the {@link Game}, every
+ * .bin was found but at least one .cue file is missing
+ * This is only relevant when we are raw-copying CHD files, where it is difficult to ensure that
+ * the .cue file is accurate.
+ */
+ private static onlyCueFilesMissingFromChd(
+ game: Game,
+ foundRoms: ROM[],
+ ): boolean {
+ // Only games with only bin/cue files can have only a cue file missing
+ const allCueBin = game.getRoms()
+ .flat()
+ .every((rom) => ['.bin', '.cue'].includes(path.extname(rom.getName()).toLowerCase()));
+ if (foundRoms.length === 0 || !allCueBin) {
+ return false;
+ }
+
+ const foundRomNames = new Set(foundRoms.map((rom) => rom.getName()));
+ const missingCueRoms = game.getRoms()
+ .filter((rom) => !foundRomNames.has(rom.getName()))
+ .filter((rom) => path.extname(rom.getName()).toLowerCase() === '.cue');
+ const missingNonCueRoms = game.getRoms()
+ .filter((rom) => !foundRomNames.has(rom.getName()))
+ .filter((rom) => path.extname(rom.getName()).toLowerCase() !== '.cue');
+ return missingCueRoms.length > 0 && missingNonCueRoms.length === 0;
+ }
+
+ private hasExcessFiles(
+ dat: DAT,
+ game: Game,
+ romsWithFiles: ROMWithFiles[],
+ indexedFiles: IndexedFiles,
+ ): boolean {
+ // For this Game, find every input file that is an ArchiveEntry
+ const inputArchiveEntries = romsWithFiles
+ // We need to rehydrate information from IndexedFiles because raw-copying/moving archives
+ // would have lost this information
+ .map((romWithFiles) => {
+ const inputFile = romWithFiles.getInputFile();
+ return indexedFiles.findFiles(romWithFiles.getRom())
+ ?.find((foundFile) => foundFile.getFilePath() === inputFile.getFilePath());
+ })
+ .filter((inputFile) => inputFile instanceof ArchiveEntry || inputFile instanceof ArchiveFile);
+ // ...then translate those ArchiveEntries into a list of unique Archives
+ const inputArchives = inputArchiveEntries
+ .map((archiveEntry) => archiveEntry.getArchive())
+ .filter(ArrayPoly.filterUniqueMapped((archive) => archive.getFilePath()));
+
+ for (const inputArchive of inputArchives) {
+ const unusedEntryPaths = this.findArchiveUnusedEntryPaths(
+ inputArchive,
+ inputArchiveEntries,
+ indexedFiles,
+ );
+ if (unusedEntryPaths.length > 0) {
+ this.progressBar.logTrace(`${dat.getNameShort()}: ${game.getName()}: cannot use '${inputArchive.getFilePath()}' as an input file, it has the excess entries:\n${unusedEntryPaths.map((entryPath) => ` ${entryPath}`).join('\n')}`);
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ /**
+ * Given an input {@link archive} and a set of {@link inputFiles} that match to a {@link ROM} from
+ * a {@link Game}, determine if every entry from the {@link archive} was matched.
+ */
+ private findArchiveUnusedEntryPaths(
+ archive: Archive,
+ inputFiles: File[],
+ indexedFiles: IndexedFiles,
+ ): ArchiveEntry[] {
+ if (this.options.shouldZip()
+ || this.options.shouldExtract()
+ || this.options.getAllowExcessSets()
+ ) {
+ // We don't particularly care where input files come from
+ return [];
+ }
+
+ // Find the Archive's entries (all of them, not just ones that match ROMs in this Game)
+ // NOTE(cemmer): we need to use hashCode() because a Game may have duplicate ROMs that all got
+ // matched to the same input file, so not every archive entry may be in {@link inputFiles}
+ const archiveEntryHashCodes = new Set(inputFiles
+ .filter((entry) => entry.getFilePath() === archive.getFilePath())
+ .filter((file): file is ArchiveEntry => file instanceof ArchiveEntry)
+ .map((entry) => entry.hashCode()));
+
+ // Find which of the Archive's entries didn't match to a ROM from this Game
+ return (indexedFiles.getFilesByFilePath().get(archive.getFilePath()) ?? [])
+ .filter((file): file is ArchiveEntry => file instanceof ArchiveEntry)
+ .filter((file) => !(archive instanceof Chd) || !file.getExtractedFilePath().toLowerCase().endsWith('.cue'))
+ .filter((entry) => !archiveEntryHashCodes.has(entry.hashCode()));
+ }
}
diff --git a/src/modules/candidateMergeSplitValidator.ts b/src/modules/candidateMergeSplitValidator.ts
index 475766d5a..ba75642d2 100644
--- a/src/modules/candidateMergeSplitValidator.ts
+++ b/src/modules/candidateMergeSplitValidator.ts
@@ -23,18 +23,18 @@ export default class CandidateMergeSplitValidator extends Module {
/**
* Validate the {@link ReleaseCandidate}s.
*/
- async validate(
+ validate(
dat: DAT,
parentsToCandidates: Map,
- ): Promise {
+ ): string[] {
if (parentsToCandidates.size === 0) {
this.progressBar.logTrace(`${dat.getNameShort()}: no parents to validate merged & split ROM sets for`);
return [];
}
this.progressBar.logTrace(`${dat.getNameShort()}: validating merged & split ROM sets`);
- await this.progressBar.setSymbol(ProgressBarSymbol.VALIDATING);
- await this.progressBar.reset(parentsToCandidates.size);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_VALIDATING);
+ this.progressBar.reset(parentsToCandidates.size);
const datGamesIndexed = dat.getGames().reduce((map, game) => {
map.set(game.getName(), game);
@@ -72,7 +72,7 @@ export default class CandidateMergeSplitValidator extends Module {
) {
const missingDeviceGames = game.getDeviceRefs()
.map((deviceRef) => datGamesIndexed.get(deviceRef.getName()))
- .filter(ArrayPoly.filterNotNullish)
+ .filter((deviceGame) => deviceGame !== undefined)
// Dependent device has ROM files
.filter((deviceGame) => deviceGame.getRoms().length)
.map((deviceGame) => {
@@ -83,7 +83,7 @@ export default class CandidateMergeSplitValidator extends Module {
}
return deviceGame.getName();
})
- .filter(ArrayPoly.filterNotNullish)
+ .filter((deviceGameName) => deviceGameName !== undefined)
.sort();
missingDependencies = [...missingDependencies, ...missingDeviceGames];
}
diff --git a/src/modules/candidatePatchGenerator.ts b/src/modules/candidatePatchGenerator.ts
index 9c3fd591d..439d63ac6 100644
--- a/src/modules/candidatePatchGenerator.ts
+++ b/src/modules/candidatePatchGenerator.ts
@@ -1,7 +1,6 @@
import path from 'node:path';
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
-import ArrayPoly from '../polyfill/arrayPoly.js';
import DAT from '../types/dats/dat.js';
import Game from '../types/dats/game.js';
import Parent from '../types/dats/parent.js';
@@ -37,8 +36,8 @@ export default class CandidatePatchGenerator extends Module {
}
this.progressBar.logTrace(`${dat.getNameShort()}: generating patched candidates`);
- await this.progressBar.setSymbol(ProgressBarSymbol.GENERATING);
- await this.progressBar.reset(parentsToCandidates.size);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_GENERATING);
+ this.progressBar.reset(parentsToCandidates.size);
const crcToPatches = CandidatePatchGenerator.indexPatchesByCrcBefore(patches);
this.progressBar.logTrace(`${dat.getNameShort()}: ${crcToPatches.size} unique patch${crcToPatches.size !== 1 ? 'es' : ''} found`);
@@ -51,10 +50,12 @@ export default class CandidatePatchGenerator extends Module {
private static indexPatchesByCrcBefore(patches: Patch[]): Map {
return patches.reduce((map, patch) => {
- map.set(patch.getCrcBefore(), [
- ...(map.get(patch.getCrcBefore()) ?? []),
- patch,
- ]);
+ const key = patch.getCrcBefore();
+ if (!map.has(key)) {
+ map.set(key, [patch]);
+ } else {
+ map.get(key)?.push(patch);
+ }
return map;
}, new Map());
}
@@ -110,7 +111,7 @@ export default class CandidatePatchGenerator extends Module {
.flatMap((romWithFiles) => romWithFiles.getInputFile())
.filter((inputFile) => inputFile.getCrc32() !== undefined)
.flatMap((inputFile) => crcToPatches.get(inputFile.getCrc32() as string))
- .filter(ArrayPoly.filterNotNullish);
+ .filter((patch) => patch !== undefined);
// No relevant patches found, no new candidates generated
if (releaseCandidatePatches.length === 0) {
diff --git a/src/modules/candidatePostProcessor.ts b/src/modules/candidatePostProcessor.ts
index cb3bd9e25..3a8576892 100644
--- a/src/modules/candidatePostProcessor.ts
+++ b/src/modules/candidatePostProcessor.ts
@@ -22,18 +22,18 @@ export default class CandidatePostProcessor extends Module {
/**
* Post-process the candidates.
*/
- async process(
+ process(
dat: DAT,
parentsToCandidates: Map,
- ): Promise> {
+ ): Map {
if (parentsToCandidates.size === 0) {
this.progressBar.logTrace(`${dat.getNameShort()}: no parents, so no candidates to process`);
return parentsToCandidates;
}
this.progressBar.logTrace(`${dat.getNameShort()}: processing candidates`);
- await this.progressBar.setSymbol(ProgressBarSymbol.GENERATING);
- await this.progressBar.reset(parentsToCandidates.size);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_GENERATING);
+ this.progressBar.reset(parentsToCandidates.size);
// Get the output basename of every ROM
const outputFileBasenames = [...parentsToCandidates.values()]
@@ -78,11 +78,7 @@ export default class CandidatePostProcessor extends Module {
outputFileBasenames,
);
- return new ReleaseCandidate(
- releaseCandidate.getGame(),
- releaseCandidate.getRelease(),
- newRomsWithFiles,
- );
+ return releaseCandidate.withRomsWithFiles(newRomsWithFiles);
}
private mapRomsWithFiles(
@@ -106,11 +102,7 @@ export default class CandidatePostProcessor extends Module {
}
const newOutputFile = romWithFiles.getOutputFile().withFilePath(newOutputPath);
- return new ROMWithFiles(
- romWithFiles.getRom(),
- romWithFiles.getInputFile(),
- newOutputFile,
- );
+ return romWithFiles.withOutputFile(newOutputFile);
});
}
}
diff --git a/src/modules/candidatePreferer.ts b/src/modules/candidatePreferer.ts
index 4a5bfb44d..36f6ea989 100644
--- a/src/modules/candidatePreferer.ts
+++ b/src/modules/candidatePreferer.ts
@@ -2,7 +2,7 @@ import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
import fsPoly from '../polyfill/fsPoly.js';
import DAT from '../types/dats/dat.js';
import Parent from '../types/dats/parent.js';
-import Options from '../types/options.js';
+import Options, { PreferRevision } from '../types/options.js';
import ReleaseCandidate from '../types/releaseCandidate.js';
import Module from './module.js';
@@ -21,10 +21,10 @@ export default class CandidatePreferer extends Module {
/**
* Prefer some candidates.
*/
- async prefer(
+ prefer(
dat: DAT,
parentsToCandidates: Map,
- ): Promise> {
+ ): Map {
this.progressBar.logTrace(`${dat.getNameShort()}: preferring candidates`);
if (parentsToCandidates.size === 0) {
@@ -45,10 +45,10 @@ export default class CandidatePreferer extends Module {
return parentsToCandidates;
}
- await this.progressBar.setSymbol(ProgressBarSymbol.FILTERING);
- await this.progressBar.reset(parentsToCandidates.size);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_FILTERING);
+ this.progressBar.reset(parentsToCandidates.size);
- const output = await this.sortAndFilter(dat, parentsToCandidates);
+ const output = this.sortAndFilter(dat, parentsToCandidates);
const size = [...output.values()]
.flat()
@@ -61,15 +61,15 @@ export default class CandidatePreferer extends Module {
return output;
}
- private async sortAndFilter(
+ private sortAndFilter(
dat: DAT,
parentsToCandidates: Map,
- ): Promise> {
+ ): Map {
const output = new Map();
for (let i = 0; i < [...parentsToCandidates.entries()].length; i += 1) {
const [parent, releaseCandidates] = [...parentsToCandidates.entries()][i];
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
if (releaseCandidates.length > 1) {
// Reduce log spam by only logging parents that can be changed
this.progressBar.logTrace(`${dat.getNameShort()}: ${parent.getName()} (parent): ${releaseCandidates.length.toLocaleString()} candidate${releaseCandidates.length !== 1 ? 's' : ''} before filtering`);
@@ -86,7 +86,7 @@ export default class CandidatePreferer extends Module {
output.set(parent, []);
}
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
}
return output;
@@ -109,8 +109,6 @@ export default class CandidatePreferer extends Module {
|| this.preferRegionsSort(a, b)
|| this.preferRevisionSort(a, b)
|| this.preferRetailSort(a, b)
- || this.preferNTSCSort(a, b)
- || this.preferPALSort(a, b)
|| this.preferParentSort(a, b);
}
@@ -190,9 +188,9 @@ export default class CandidatePreferer extends Module {
}
private preferRevisionSort(a: ReleaseCandidate, b: ReleaseCandidate): number {
- if (this.options.getPreferRevisionNewer()) {
+ if (this.options.getPreferRevision() === PreferRevision.NEWER) {
return b.getGame().getRevision() - a.getGame().getRevision();
- } if (this.options.getPreferRevisionOlder()) {
+ } if (this.options.getPreferRevision() === PreferRevision.OLDER) {
return a.getGame().getRevision() - b.getGame().getRevision();
}
return 0;
@@ -205,20 +203,6 @@ export default class CandidatePreferer extends Module {
return (a.getGame().isRetail() ? 0 : 1) - (b.getGame().isRetail() ? 0 : 1);
}
- private preferNTSCSort(a: ReleaseCandidate, b: ReleaseCandidate): number {
- if (!this.options.getPreferNTSC()) {
- return 0;
- }
- return (a.getGame().isNTSC() ? 0 : 1) - (b.getGame().isNTSC() ? 0 : 1);
- }
-
- private preferPALSort(a: ReleaseCandidate, b: ReleaseCandidate): number {
- if (!this.options.getPreferPAL()) {
- return 0;
- }
- return (a.getGame().isPAL() ? 0 : 1) - (b.getGame().isPAL() ? 0 : 1);
- }
-
private preferParentSort(a: ReleaseCandidate, b: ReleaseCandidate): number {
if (!this.options.getPreferParent()) {
return 0;
diff --git a/src/modules/candidateValidator.ts b/src/modules/candidateValidator.ts
index 39f07a9ee..bf77185f4 100644
--- a/src/modules/candidateValidator.ts
+++ b/src/modules/candidateValidator.ts
@@ -16,18 +16,18 @@ export default class CandidateValidator extends Module {
/**
* Validate the {@link ReleaseCandidate}s.
*/
- async validate(
+ validate(
dat: DAT,
parentsToCandidates: Map,
- ): Promise {
+ ): ReleaseCandidate[] {
if (parentsToCandidates.size === 0) {
this.progressBar.logTrace(`${dat.getNameShort()}: no parents to validate candidates for`);
return [];
}
this.progressBar.logTrace(`${dat.getNameShort()}: validating candidates`);
- await this.progressBar.setSymbol(ProgressBarSymbol.VALIDATING);
- await this.progressBar.reset(parentsToCandidates.size);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_VALIDATING);
+ this.progressBar.reset(parentsToCandidates.size);
const conflictedOutputPaths = this.validateUniqueOutputPaths(dat, parentsToCandidates);
if (conflictedOutputPaths.length > 0) {
@@ -47,7 +47,11 @@ export default class CandidateValidator extends Module {
.reduce((map, releaseCandidate) => {
releaseCandidate.getRomsWithFiles().forEach((romWithFiles) => {
const key = romWithFiles.getOutputFile().getFilePath();
- map.set(key, [...(map.get(key) ?? []), releaseCandidate]);
+ if (!map.has(key)) {
+ map.set(key, [releaseCandidate]);
+ } else {
+ map.get(key)?.push(releaseCandidate);
+ }
});
return map;
}, new Map());
diff --git a/src/modules/candidateWriter.ts b/src/modules/candidateWriter.ts
index cb11e8475..1bceec2dc 100644
--- a/src/modules/candidateWriter.ts
+++ b/src/modules/candidateWriter.ts
@@ -5,6 +5,7 @@ import { Semaphore } from 'async-mutex';
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
import ElasticSemaphore from '../elasticSemaphore.js';
import Defaults from '../globals/defaults.js';
+import KeyedMutex from '../keyedMutex.js';
import ArrayPoly from '../polyfill/arrayPoly.js';
import fsPoly from '../polyfill/fsPoly.js';
import DAT from '../types/dats/dat.js';
@@ -26,6 +27,7 @@ export interface CandidateWriterResults {
* Copy or move output ROM files, if applicable.
*/
export default class CandidateWriter extends Module {
+ // The maximum number of candidates that can be written at once
private static readonly THREAD_SEMAPHORE = new Semaphore(Number.MAX_SAFE_INTEGER);
// WARN(cemmer): there is an undocumented semaphore max value that can be used, the full
@@ -34,6 +36,12 @@ export default class CandidateWriter extends Module {
Defaults.MAX_READ_WRITE_CONCURRENT_KILOBYTES,
);
+ // When moving input files, process input file paths exclusively
+ private static readonly MOVE_MUTEX = new KeyedMutex(1000);
+
+ // When moving input files, keep track of files that have been moved
+ private static readonly FILE_PATH_MOVES = new Map();
+
private readonly options: Options;
private readonly filesQueuedForDeletion: File[] = [];
@@ -88,15 +96,19 @@ export default class CandidateWriter extends Module {
const totalCandidateCount = [...parentsToWritableCandidates.values()].flat().length;
this.progressBar.logTrace(`${dat.getNameShort()}: writing ${totalCandidateCount.toLocaleString()} candidate${totalCandidateCount !== 1 ? 's' : ''}`);
- await this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
- await this.progressBar.reset(parentsToWritableCandidates.size);
+ if (this.options.shouldTest() && !this.options.getOverwrite()) {
+ this.progressBar.setSymbol(ProgressBarSymbol.TESTING);
+ } else {
+ this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
+ }
+ this.progressBar.reset(parentsToWritableCandidates.size);
await Promise.all([...parentsToWritableCandidates.entries()].map(
async ([
parent,
releaseCandidates,
]) => CandidateWriter.THREAD_SEMAPHORE.runExclusive(async () => {
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
this.progressBar.logTrace(`${dat.getNameShort()}: ${parent.getName()} (parent): writing ${releaseCandidates.length.toLocaleString()} candidate${releaseCandidates.length !== 1 ? 's' : ''}`);
for (const releaseCandidate of releaseCandidates) {
@@ -104,7 +116,7 @@ export default class CandidateWriter extends Module {
}
this.progressBar.logTrace(`${dat.getNameShort()}: ${parent.getName()} (parent): done writing ${releaseCandidates.length.toLocaleString()} candidate${releaseCandidates.length !== 1 ? 's' : ''}`);
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
}),
));
@@ -201,6 +213,7 @@ export default class CandidateWriter extends Module {
}
}
+ this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
let written = false;
for (let i = 0; i <= this.options.getWriteRetry(); i += 1) {
written = await this.writeZipFile(
@@ -286,11 +299,6 @@ export default class CandidateWriter extends Module {
}
// Check checksum
- if (expectedFile.getCrc32() === '00000000') {
- this.progressBar.logWarn(`${dat.getNameShort()}: ${releaseCandidate.getName()}: ${expectedFile.toString()}: can't test, expected CRC is unknown`);
- // eslint-disable-next-line no-continue
- continue;
- }
const actualFile = actualEntriesByPath.get(entryPath) as ArchiveEntry;
if (actualFile.getSha256()
&& expectedFile.getSha256()
@@ -312,6 +320,7 @@ export default class CandidateWriter extends Module {
}
if (actualFile.getCrc32()
&& expectedFile.getCrc32()
+ && expectedFile.getCrc32() !== '00000000'
&& actualFile.getCrc32() !== expectedFile.getCrc32()
) {
return `has the CRC32 ${actualFile.getCrc32()}, expected ${expectedFile.getCrc32()}`;
@@ -340,7 +349,7 @@ export default class CandidateWriter extends Module {
outputZip: Zip,
inputToOutputZipEntries: [File, ArchiveEntry][],
): Promise {
- this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: creating zip archive '${outputZip.getFilePath()}' with the entries:\n${inputToOutputZipEntries.map(([input, output]) => ` '${input.toString()}' (${fsPoly.sizeReadable(input.getSize())}) -> '${output.getEntryPath()}'`).join('\n')}`);
+ this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: creating zip archive '${outputZip.getFilePath()}' with the entries:\n${inputToOutputZipEntries.map(([input, output]) => ` '${input.toString()}' (${fsPoly.sizeReadable(input.getSize())}) โ '${output.getEntryPath()}'`).join('\n')}`);
try {
await CandidateWriter.ensureOutputDirExists(outputZip.getFilePath());
@@ -388,10 +397,12 @@ export default class CandidateWriter extends Module {
// the same input archive at the same time, to benefit from batch extraction.
const uniqueInputToOutputEntriesMap = uniqueInputToOutputEntries
.reduce((map, [inputRomFile, outputRomFile]) => {
- map.set(inputRomFile.getFilePath(), [
- ...(map.get(inputRomFile.getFilePath()) ?? []),
- [inputRomFile, outputRomFile],
- ]);
+ const key = inputRomFile.getFilePath();
+ if (!map.has(key)) {
+ map.set(key, [[inputRomFile, outputRomFile]]);
+ } else {
+ map.get(key)?.push([inputRomFile, outputRomFile]);
+ }
return map;
}, new Map());
for (const groupedInputToOutput of uniqueInputToOutputEntriesMap.values()) {
@@ -412,10 +423,18 @@ export default class CandidateWriter extends Module {
inputRomFile: File,
outputRomFile: File,
): Promise {
- // Input and output are the exact same, do nothing
+ // Input and output are the exact same, maybe do nothing
if (outputRomFile.equals(inputRomFile)) {
- this.progressBar.logDebug(`${dat.getNameShort()}: ${releaseCandidate.getName()}: ${outputRomFile}: input and output file is the same, skipping`);
- return;
+ const wasMoved = this.options.shouldMove()
+ && await CandidateWriter.MOVE_MUTEX.runExclusiveForKey(
+ inputRomFile.getFilePath(),
+ () => CandidateWriter.FILE_PATH_MOVES.get(inputRomFile.getFilePath()),
+ ) !== undefined;
+
+ if (!wasMoved) {
+ this.progressBar.logDebug(`${dat.getNameShort()}: ${releaseCandidate.getName()}: ${outputRomFile}: input and output file is the same, skipping`);
+ return;
+ }
}
const outputFilePath = outputRomFile.getFilePath();
@@ -441,9 +460,14 @@ export default class CandidateWriter extends Module {
}
}
+ this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
let written = false;
for (let i = 0; i <= this.options.getWriteRetry(); i += 1) {
- written = await this.writeRawFile(dat, releaseCandidate, inputRomFile, outputFilePath);
+ if (this.options.shouldMove()) {
+ written = await this.moveRawFile(dat, releaseCandidate, inputRomFile, outputFilePath);
+ } else {
+ written = await this.copyRawFile(dat, releaseCandidate, inputRomFile, outputFilePath);
+ }
if (written && !this.options.shouldTest()) {
// Successfully written, unknown if valid
@@ -476,22 +500,65 @@ export default class CandidateWriter extends Module {
this.enqueueFileDeletion(inputRomFile);
}
- private async writeRawFile(
+ private async moveRawFile(
dat: DAT,
releaseCandidate: ReleaseCandidate,
inputRomFile: File,
outputFilePath: string,
): Promise {
- this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: copying file '${inputRomFile.toString()}' (${fsPoly.sizeReadable(inputRomFile.getSize())}) -> '${outputFilePath}'`);
+ // Lock the input file, we can't handle concurrent moves
+ return CandidateWriter.MOVE_MUTEX.runExclusiveForKey(inputRomFile.getFilePath(), async () => {
+ const movedFilePath = CandidateWriter.FILE_PATH_MOVES.get(inputRomFile.getFilePath());
+ if (movedFilePath) {
+ // The file was already moved, we shouldn't move it again
+ return this.copyRawFile(
+ dat,
+ releaseCandidate,
+ inputRomFile.withFilePath(movedFilePath),
+ outputFilePath,
+ );
+ }
+
+ if (inputRomFile instanceof ArchiveEntry
+ || inputRomFile.getFileHeader() !== undefined
+ || inputRomFile.getPatch() !== undefined
+ ) {
+ // The file can't be moved as-is, it needs to get copied
+ return this.copyRawFile(dat, releaseCandidate, inputRomFile, outputFilePath);
+ }
+
+ this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: moving file '${inputRomFile.toString()}' (${fsPoly.sizeReadable(inputRomFile.getSize())}) โ '${outputFilePath}'`);
+
+ try {
+ await CandidateWriter.ensureOutputDirExists(outputFilePath);
+
+ await fsPoly.mv(inputRomFile.getFilePath(), outputFilePath);
+ CandidateWriter.FILE_PATH_MOVES.set(inputRomFile.getFilePath(), outputFilePath);
+ return true;
+ } catch (error) {
+ this.progressBar.logError(`${dat.getNameShort()}: ${releaseCandidate.getName()}: failed to move file '${inputRomFile.toString()}' โ '${outputFilePath}': ${error}`);
+ return false;
+ }
+ });
+ }
+
+ private async copyRawFile(
+ dat: DAT,
+ releaseCandidate: ReleaseCandidate,
+ inputRomFile: File,
+ outputFilePath: string,
+ ): Promise {
+ this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: ${inputRomFile instanceof ArchiveEntry ? 'extracting' : 'copying'} file '${inputRomFile.toString()}' (${fsPoly.sizeReadable(inputRomFile.getSize())}) โ '${outputFilePath}'`);
try {
await CandidateWriter.ensureOutputDirExists(outputFilePath);
+
const tempRawFile = await fsPoly.mktemp(outputFilePath);
await inputRomFile.extractAndPatchToFile(tempRawFile);
await fsPoly.mv(tempRawFile, outputFilePath);
return true;
} catch (error) {
- this.progressBar.logError(`${dat.getNameShort()}: ${releaseCandidate.getName()}: ${outputFilePath}: failed to copy from ${inputRomFile.toString()}: ${error}`);
+ this.progressBar.logError(`${dat.getNameShort()}: ${releaseCandidate.getName()}: failed to ${inputRomFile instanceof ArchiveEntry ? 'extract' : 'copy'} file '${inputRomFile.toString()}' โ '${outputFilePath}': ${error}`);
return false;
}
}
@@ -505,14 +572,15 @@ export default class CandidateWriter extends Module {
this.progressBar.logTrace(`${dat.getNameShort()}: ${releaseCandidate.getName()}: ${outputFilePath}: testing raw file`);
// Check checksum
- if (expectedFile.getCrc32() === '00000000') {
- this.progressBar.logWarn(`${dat.getNameShort()}: ${releaseCandidate.getName()}: ${outputFilePath}: can't test, expected CRC is unknown`);
- return undefined;
+ let actualFile: File;
+ try {
+ actualFile = await File.fileOf(
+ { filePath: outputFilePath },
+ expectedFile.getChecksumBitmask(),
+ );
+ } catch (error) {
+ return `failed to parse: ${error}`;
}
- const actualFile = await File.fileOf(
- { filePath: outputFilePath },
- expectedFile.getChecksumBitmask(),
- );
if (actualFile.getSha256()
&& expectedFile.getSha256()
&& actualFile.getSha256() !== expectedFile.getSha256()
@@ -533,6 +601,7 @@ export default class CandidateWriter extends Module {
}
if (actualFile.getCrc32()
&& expectedFile.getCrc32()
+ && expectedFile.getCrc32() !== '00000000'
&& actualFile.getCrc32() !== expectedFile.getCrc32()
) {
return `has the CRC32 ${actualFile.getCrc32()}, expected ${expectedFile.getCrc32()}`;
@@ -626,6 +695,7 @@ export default class CandidateWriter extends Module {
await fsPoly.rm(linkPath, { force: true });
}
+ this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
for (let i = 0; i <= this.options.getWriteRetry(); i += 1) {
const written = await this.writeRawLink(dat, releaseCandidate, targetPath, linkPath);
@@ -667,10 +737,10 @@ export default class CandidateWriter extends Module {
try {
await CandidateWriter.ensureOutputDirExists(linkPath);
if (this.options.getSymlink()) {
- this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: creating symlink '${targetPath}' -> '${linkPath}'`);
+ this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: creating symlink '${targetPath}' โ '${linkPath}'`);
await fsPoly.symlink(targetPath, linkPath);
} else {
- this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: creating hard link '${targetPath}' -> '${linkPath}'`);
+ this.progressBar.logInfo(`${dat.getNameShort()}: ${releaseCandidate.getName()}: creating hard link '${targetPath}' โ '${linkPath}'`);
await fsPoly.hardlink(targetPath, linkPath);
}
return true;
diff --git a/src/modules/datFilter.ts b/src/modules/datFilter.ts
index 28cc7dfbd..22bf06c8c 100644
--- a/src/modules/datFilter.ts
+++ b/src/modules/datFilter.ts
@@ -20,7 +20,7 @@ export default class DATFilter extends Module {
/**
* Create a new DAT after filtering.
*/
- async filter(dat: DAT): Promise {
+ filter(dat: DAT): DAT {
// Return early if there aren't any games
if (dat.getGames().length === 0) {
this.progressBar.logTrace(`${dat.getNameShort()}: no games to filter`);
@@ -28,8 +28,8 @@ export default class DATFilter extends Module {
}
this.progressBar.logTrace(`${dat.getNameShort()}: filtering DAT`);
- await this.progressBar.setSymbol(ProgressBarSymbol.FILTERING);
- await this.progressBar.reset(dat.getGames().length);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_FILTERING);
+ this.progressBar.reset(dat.getGames().length);
const filteredGames = dat.getParents().flatMap((parent) => {
const games = parent.getGames().filter((game) => this.filterGame(game));
diff --git a/src/modules/datGameInferrer.ts b/src/modules/datGameInferrer.ts
index 849df88a2..d015fac78 100644
--- a/src/modules/datGameInferrer.ts
+++ b/src/modules/datGameInferrer.ts
@@ -1,5 +1,8 @@
+import fs from 'node:fs';
import path from 'node:path';
+import util from 'node:util';
+import { parse } from '@gplane/cue';
import moment from 'moment';
import ProgressBar from '../console/progressBar.js';
@@ -34,7 +37,7 @@ export default class DATGameInferrer extends Module {
/**
* Infer {@link Game}s from input files.
*/
- infer(romFiles: File[]): DAT[] {
+ async infer(romFiles: File[]): Promise {
this.progressBar.logTrace(`inferring DATs for ${romFiles.length.toLocaleString()} ROM${romFiles.length !== 1 ? 's' : ''}`);
const normalizedInputPaths = this.options.getInputPaths()
@@ -50,21 +53,24 @@ export default class DATGameInferrer extends Module {
.filter((inputPath) => normalizedPath.startsWith(inputPath));
(matchedInputPaths.length > 0 ? matchedInputPaths : [DATGameInferrer.DEFAULT_DAT_NAME])
.forEach((inputPath) => {
- const datRomFiles = [...(map.get(inputPath) ?? []), file];
- map.set(inputPath, datRomFiles);
+ if (!map.has(inputPath)) {
+ map.set(inputPath, [file]);
+ } else {
+ map.get(inputPath)?.push(file);
+ }
});
return map;
}, new Map());
this.progressBar.logTrace(`inferred ${inputPathsToRomFiles.size.toLocaleString()} DAT${inputPathsToRomFiles.size !== 1 ? 's' : ''}`);
- const dats = [...inputPathsToRomFiles.entries()]
- .map(([inputPath, datRomFiles]) => DATGameInferrer.createDAT(inputPath, datRomFiles));
+ const dats = await Promise.all([...inputPathsToRomFiles.entries()]
+ .map(async ([inputPath, datRomFiles]) => this.createDAT(inputPath, datRomFiles)));
this.progressBar.logTrace('done inferring DATs');
return dats;
}
- private static createDAT(inputPath: string, romFiles: File[]): DAT {
+ private async createDAT(inputPath: string, romFiles: File[]): Promise {
const datName = path.basename(inputPath);
const date = moment().format('YYYYMMDD-HHmmss');
const header = new Header({
@@ -80,50 +86,47 @@ export default class DATGameInferrer extends Module {
].join('\n'),
});
- // For all non-archived files, group files of the same filename without extension together
- const gameNamesToRawFiles = romFiles
- .filter((file) => !(file instanceof ArchiveEntry))
- .reduce((map, file) => {
- const gameName = DATGameInferrer.getGameName(file);
- map.set(gameName, [...(map.get(gameName) ?? []), file]);
- return map;
- }, new Map());
+ let remainingRomFiles = romFiles;
+ let gameNamesToRomFiles: [string, File[]][] = [];
- // For archives, assume the entire archive is one game
- const archivePathsToArchiveEntries = romFiles
- .filter((file) => file instanceof ArchiveEntry)
- .reduce((map, file) => {
- const archivePath = file.getFilePath();
- map.set(archivePath, [...(map.get(archivePath) ?? []), file]);
- return map;
- }, new Map[]>());
- const gameNamesToArchiveEntries = [...archivePathsToArchiveEntries.values()]
- .map((archiveEntries) => {
- const gameName = DATGameInferrer.getGameName(archiveEntries[0]);
- return [gameName, archiveEntries] satisfies [string, ArchiveEntry[]];
- });
+ // For each inference strategy
+ const inferFunctions = [
+ this.inferArchiveEntries,
+ this.inferBinCueFiles,
+ this.inferGdiFiles,
+ this.inferRawFiles,
+ ];
+ for (const inferFunction of inferFunctions) {
+ // Infer the games and their files)
+ const result = await inferFunction.bind(this)(remainingRomFiles);
- const games = [
- ...gameNamesToRawFiles.entries(),
- ...gameNamesToArchiveEntries,
- ]
- .map(([gameName, gameRomFiles]) => {
- const roms = gameRomFiles
- .map((romFile) => new ROM({
- name: path.basename(romFile.getExtractedFilePath()),
- size: romFile.getSize(),
- crc32: romFile.getCrc32(),
- md5: romFile.getMd5(),
- sha1: romFile.getSha1(),
- sha256: romFile.getSha256(),
- }))
- .filter(ArrayPoly.filterUniqueMapped((rom) => rom.getName()));
- return new Game({
- name: gameName,
- description: gameName,
- rom: roms,
- });
- })
+ // Update the list of results
+ gameNamesToRomFiles = [...gameNamesToRomFiles, ...result];
+
+ // Remove the consumed files from further inference
+ const consumedFiles = new Set(result
+ .flatMap(([, resultFiles]) => resultFiles)
+ .map((file) => file.toString()));
+ remainingRomFiles = remainingRomFiles.filter((file) => !consumedFiles.has(file.toString()));
+ }
+
+ const games = gameNamesToRomFiles.map(([gameName, gameRomFiles]) => {
+ const roms = gameRomFiles
+ .map((romFile) => new ROM({
+ name: path.basename(romFile.getExtractedFilePath()),
+ size: romFile.getSize(),
+ crc32: romFile.getCrc32(),
+ md5: romFile.getMd5(),
+ sha1: romFile.getSha1(),
+ sha256: romFile.getSha256(),
+ }))
+ .filter(ArrayPoly.filterUniqueMapped((rom) => rom.getName()));
+ return new Game({
+ name: gameName,
+ description: gameName,
+ rom: roms,
+ });
+ })
// Filter out duplicate games
.filter(ArrayPoly.filterUniqueMapped((game) => game.hashCode()));
@@ -143,4 +146,139 @@ export default class DATGameInferrer extends Module {
.replace(/(\.[a-z0-9]+)+$/, '')
.trim();
}
+
+ private inferArchiveEntries(romFiles: File[]): [string, ArchiveEntry[]][] {
+ this.progressBar.logTrace(`inferring games from archives from ${romFiles.length.toLocaleString()} file${romFiles.length !== 1 ? 's' : ''}`);
+
+ // For archives, assume the entire archive is one game
+ const archivePathsToArchiveEntries = romFiles
+ .filter((file) => file instanceof ArchiveEntry)
+ .reduce((map, file) => {
+ const archivePath = file.getFilePath();
+ if (!map.has(archivePath)) {
+ map.set(archivePath, [file]);
+ } else {
+ map.get(archivePath)?.push(file);
+ }
+ return map;
+ }, new Map[]>());
+
+ const results = [...archivePathsToArchiveEntries.values()]
+ .map((archiveEntries) => {
+ const gameName = DATGameInferrer.getGameName(archiveEntries[0]);
+ return [gameName, archiveEntries] satisfies [string, ArchiveEntry[]];
+ });
+
+ this.progressBar.logTrace(`inferred ${results.length.toLocaleString()} games from archives`);
+ return results;
+ }
+
+ private async inferBinCueFiles(romFiles: File[]): Promise<[string, File[]][]> {
+ const rawFiles = romFiles.filter((file) => !(file instanceof ArchiveEntry));
+ this.progressBar.logTrace(`inferring games from cue files from ${rawFiles.length.toLocaleString()} non-archive${rawFiles.length !== 1 ? 's' : ''}`);
+
+ const rawFilePathsToFiles = rawFiles
+ .reduce((map, file) => {
+ map.set(file.getFilePath(), file);
+ return map;
+ }, new Map());
+
+ const results = (await Promise.all(rawFiles
+ .filter((file) => file.getExtractedFilePath().toLowerCase().endsWith('.cue'))
+ .map(async (cueFile): Promise<[string, File[]] | undefined> => {
+ try {
+ const cueData = await util.promisify(fs.readFile)(cueFile.getFilePath());
+
+ const cueSheet = parse(cueData.toString(), {
+ fatal: true,
+ }).sheet;
+
+ const binFiles = cueSheet.files
+ .map((binFile) => path.join(path.dirname(cueFile.getFilePath()), binFile.name))
+ .map((binFilePath) => rawFilePathsToFiles.get(binFilePath))
+ .filter((file) => file !== undefined);
+ if (binFiles.length === 0) {
+ return undefined;
+ }
+
+ const gameName = DATGameInferrer.getGameName(cueFile);
+ return [gameName, [cueFile, ...binFiles]];
+ } catch {
+ return undefined;
+ }
+ })))
+ .filter((result) => result !== undefined);
+
+ this.progressBar.logTrace(`inferred ${results.length.toLocaleString()} games from cue files`);
+ return results;
+ }
+
+ private async inferGdiFiles(romFiles: File[]): Promise<[string, File[]][]> {
+ const rawFiles = romFiles.filter((file) => !(file instanceof ArchiveEntry));
+ this.progressBar.logTrace(`inferring games from gdi files from ${rawFiles.length.toLocaleString()} non-archive${rawFiles.length !== 1 ? 's' : ''}`);
+
+ const rawFilePathsToFiles = rawFiles
+ .reduce((map, file) => {
+ map.set(file.getFilePath(), file);
+ return map;
+ }, new Map());
+
+ const results = (await Promise.all(rawFiles
+ .filter((file) => file.getExtractedFilePath().toLowerCase().endsWith('.gdi'))
+ .map(async (gdiFile): Promise<[string, File[]] | undefined> => {
+ try {
+ const cueData = await util.promisify(fs.readFile)(gdiFile.getFilePath());
+
+ const { name: filePrefix } = path.parse(gdiFile.getFilePath());
+ const gdiContents = `${cueData.toString()
+ .split(/\r?\n/)
+ .filter((line) => line)
+ // Replace the chdman-generated track files with TOSEC-style track filenames
+ .map((line) => line
+ .replace(filePrefix, 'track')
+ .replace(/"/g, ''))
+ .join('\r\n')}\r\n`;
+
+ const trackFilePaths = gdiContents.trim()
+ .split(/\r?\n/)
+ .slice(1)
+ .map((line) => line.split(' ')[4]);
+ const trackFiles = trackFilePaths
+ .map((trackFilePath) => path.join(path.dirname(gdiFile.getFilePath()), trackFilePath))
+ .map((trackFilePath) => rawFilePathsToFiles.get(trackFilePath))
+ .filter((file) => file !== undefined);
+ if (trackFiles.length === 0) {
+ return undefined;
+ }
+
+ const gameName = DATGameInferrer.getGameName(gdiFile);
+ return [gameName, [gdiFile, ...trackFiles]];
+ } catch {
+ return undefined;
+ }
+ })))
+ .filter((result) => result !== undefined);
+
+ this.progressBar.logTrace(`inferred ${results.length.toLocaleString()} games from cue files`);
+ return results;
+ }
+
+ private inferRawFiles(romFiles: File[]): [string, File[]][] {
+ this.progressBar.logTrace(`inferring games from raw files from ${romFiles.length.toLocaleString()} file${romFiles.length !== 1 ? 's' : ''}`);
+
+ const results = romFiles
+ .filter((file) => !(file instanceof ArchiveEntry))
+ .reduce((map, file) => {
+ const gameName = DATGameInferrer.getGameName(file);
+ if (!map.has(gameName)) {
+ map.set(gameName, [file]);
+ } else {
+ map.get(gameName)?.push(file);
+ }
+ return map;
+ }, new Map());
+
+ this.progressBar.logTrace(`inferred ${results.size.toLocaleString()} games from raw files`);
+ return [...results.entries()];
+ }
}
diff --git a/src/modules/datMergerSplitter.ts b/src/modules/datMergerSplitter.ts
index bec1f68ae..69e90e52f 100644
--- a/src/modules/datMergerSplitter.ts
+++ b/src/modules/datMergerSplitter.ts
@@ -23,7 +23,7 @@ export default class DATMergerSplitter extends Module {
/**
* Un-merge, split, or merge the {@link Game}s within a {@link DAT}.
*/
- async merge(dat: DAT): Promise {
+ merge(dat: DAT): DAT {
// Don't do anything if no type provided
if (this.options.getMergeRoms() === undefined) {
this.progressBar.logTrace(`${dat.getNameShort()}: no ROM merge option provided, doing nothing`);
@@ -42,8 +42,8 @@ export default class DATMergerSplitter extends Module {
}, new Map());
this.progressBar.logTrace(`${dat.getNameShort()}: merging & splitting ${dat.getGames().length.toLocaleString()} game${dat.getGames().length !== 1 ? 's' : ''}`);
- await this.progressBar.setSymbol(ProgressBarSymbol.MERGE_SPLIT);
- await this.progressBar.reset(dat.getGames().length);
+ this.progressBar.setSymbol(ProgressBarSymbol.DAT_MERGE_SPLIT);
+ this.progressBar.reset(dat.getGames().length);
const newGames = dat.getParents()
.flatMap((parent) => this.mergeParent(dat, parent, gameNamesToGames));
@@ -65,6 +65,9 @@ export default class DATMergerSplitter extends Module {
// Get rid of duplicate ROMs. MAME will sometimes duplicate a file with the exact same
// name, size, and checksum but with a different "region" (e.g. neogeo).
.filter(ArrayPoly.filterUniqueMapped((rom) => rom.getName())),
+ disk: game.getDisks()
+ // Get rid of ROMs that haven't been dumped yet
+ .filter((disk) => disk.getStatus() !== 'nodump'),
}));
// 'full' types expect device ROMs to be included
@@ -81,7 +84,7 @@ export default class DATMergerSplitter extends Module {
.reduce(ArrayPoly.reduceUnique(), [])
// Get ROMs from the DeviceRef
.map((deviceRefName) => gameNamesToGames.get(deviceRefName))
- .filter(ArrayPoly.filterNotNullish)
+ .filter((deviceGame) => deviceGame !== undefined)
.flatMap((deviceGame) => deviceGame.getRoms()
.filter((rom) => rom.getStatus() !== 'nodump')),
...game.getRoms(),
@@ -115,33 +118,33 @@ export default class DATMergerSplitter extends Module {
}
return game.withProps({
- rom: DATMergerSplitter.diffGameRoms(biosGame, game),
+ rom: DATMergerSplitter.diffGameRoms(biosGame.getRoms(), game.getRoms()),
});
});
}
- // 'split' and 'merged' types should exclude ROMs found in their parent
+ // 'split' and 'merged' types should exclude ROMs & disks found in their parent
if (this.options.getMergeRoms() === MergeMode.SPLIT
|| this.options.getMergeRoms() === MergeMode.MERGED
) {
- games = games
- .map((game) => {
- if (!game.getParent()) {
- // This game doesn't have a parent
- return game;
- }
+ games = games.map((game) => {
+ if (!game.getParent()) {
+ // This game doesn't have a parent
+ return game;
+ }
- const parentGame = gameNamesToGames.get(game.getParent());
- if (!parentGame) {
- // Invalid cloneOf attribute, parent not found
- this.progressBar.logTrace(`${dat.getNameShort()}: ${game.getName()} references an invalid parent: ${game.getParent()}`);
- return game;
- }
+ const parentGame = gameNamesToGames.get(game.getParent());
+ if (!parentGame) {
+ // Invalid cloneOf attribute, parent not found
+ this.progressBar.logTrace(`${dat.getNameShort()}: ${game.getName()} references an invalid parent: ${game.getParent()}`);
+ return game;
+ }
- return game.withProps({
- rom: DATMergerSplitter.diffGameRoms(parentGame, game),
- });
+ return game.withProps({
+ rom: DATMergerSplitter.diffGameRoms(parentGame.getRoms(), game.getRoms()),
+ disk: DATMergerSplitter.diffGameRoms(parentGame.getDisks(), game.getDisks()),
});
+ });
}
const parentGame = games.find((game) => game.isParent());
@@ -173,13 +176,13 @@ export default class DATMergerSplitter extends Module {
})];
}
- private static diffGameRoms(parent: Game, child: Game): ROM[] {
- const parentRomNamesToHashCodes = parent.getRoms().reduce((map, rom) => {
+ private static diffGameRoms(parentRoms: ROM[], childRoms: ROM[]): ROM[] {
+ const parentRomNamesToHashCodes = parentRoms.reduce((map, rom) => {
map.set(rom.getName(), rom.hashCode());
return map;
}, new Map());
- return child.getRoms().filter((rom) => {
+ return childRoms.filter((rom) => {
const parentName = rom.getMerge() ?? rom.getName();
const parentHashCode = parentRomNamesToHashCodes.get(parentName);
if (!parentHashCode) {
diff --git a/src/modules/datParentInferrer.ts b/src/modules/datParentInferrer.ts
index 21adbb86d..5cdcac2f3 100644
--- a/src/modules/datParentInferrer.ts
+++ b/src/modules/datParentInferrer.ts
@@ -20,7 +20,7 @@ export default class DATParentInferrer extends Module {
/**
* Infer {@link Parent}s from {@link Game}s.
*/
- async infer(dat: DAT): Promise {
+ infer(dat: DAT): DAT {
if (dat.hasParentCloneInfo() && !this.options.getDatIgnoreParentClone()) {
this.progressBar.logTrace(`${dat.getNameShort()}: DAT has parent/clone info, skipping`);
return dat;
@@ -32,15 +32,19 @@ export default class DATParentInferrer extends Module {
}
this.progressBar.logTrace(`${dat.getNameShort()}: inferring parents for ${dat.getGames().length.toLocaleString()} game${dat.getGames().length !== 1 ? 's' : ''}`);
- await this.progressBar.setSymbol(ProgressBarSymbol.GROUPING_SIMILAR);
- await this.progressBar.reset(dat.getGames().length);
+ this.progressBar.setSymbol(ProgressBarSymbol.DAT_GROUPING_SIMILAR);
+ this.progressBar.reset(dat.getGames().length);
// Group games by their stripped names
const strippedNamesToGames = dat.getGames().reduce((map, game) => {
let strippedGameName = game.getName();
strippedGameName = DATParentInferrer.stripGameRegionAndLanguage(strippedGameName);
strippedGameName = DATParentInferrer.stripGameVariants(strippedGameName);
- map.set(strippedGameName, [...(map.get(strippedGameName) ?? []), game]);
+ if (!map.has(strippedGameName)) {
+ map.set(strippedGameName, [game]);
+ } else {
+ map.get(strippedGameName)?.push(game);
+ }
return map;
}, new Map());
const groupedGames = [...strippedNamesToGames.entries()]
@@ -60,6 +64,7 @@ export default class DATParentInferrer extends Module {
// ***** Regions *****
.replace(new RegExp(`\\(((${Internationalization.REGION_CODES.join('|')})[,+-]? ?)+\\)`, 'i'), '')
.replace(new RegExp(`\\(((${Internationalization.REGION_NAMES.join('|')})[,+-]? ?)+\\)`, 'i'), '')
+ .replace(/\(Latin America\)/i, '')
// ***** Languages *****
.replace(new RegExp(`\\(((${Internationalization.LANGUAGES.join('|')})[,+-]? ?)+\\)`, 'i'), '')
// ***** Cleanup *****
@@ -73,6 +78,7 @@ export default class DATParentInferrer extends Module {
.replace(/\(Alt( [a-z0-9. ]*)?\)/i, '')
.replace(/\([^)]*Collector's Edition\)/i, '')
.replace(/\(Extra Box\)/i, '')
+ .replace(/ - European Version/i, '')
.replace(/\(Fukkokuban\)/i, '') // "reprint"
.replace(/\([^)]*Genteiban\)/i, '') // "limited edition"
.replace(/\(Limited[^)]+Edition\)/i, '')
@@ -152,8 +158,8 @@ export default class DATParentInferrer extends Module {
// Nintendo - Super Nintendo Entertainment System
.replace(/\(NP\)/i, '') // "Nintendo Power"
// Sega - Dreamcast
- .replace(/\[[0-9]+S\]/, '') // boxcode
- .replace(/\[[0-9]+MM?[0-9]+(, [0-9]+MM?[0-9]+)*\]/, '')
+ .replace(/\[([0-9A-Z ]+(, )?)+\]$/, '') // TOSEC boxcode
+ .replace(/\[(compilation|data identical to retail|fixed version|keyboard|limited edition|req\. microphone|scrambled|unscrambled|white label)\]/ig, '') // TOSEC
.replace(/for Dreamcast/i, '')
// Sega - Mega Drive / Genesis
.replace(/\(MP\)/i, '') // "MegaPlay version"
diff --git a/src/modules/datScanner.ts b/src/modules/datScanner.ts
index 1565502c3..506281699 100644
--- a/src/modules/datScanner.ts
+++ b/src/modules/datScanner.ts
@@ -5,12 +5,17 @@ import { parse } from '@fast-csv/parse';
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
import DriveSemaphore from '../driveSemaphore.js';
-import ArrayPoly from '../polyfill/arrayPoly.js';
import bufferPoly from '../polyfill/bufferPoly.js';
import fsPoly from '../polyfill/fsPoly.js';
-import CMProParser, { DATProps, GameProps, ROMProps } from '../types/dats/cmpro/cmProParser.js';
+import CMProParser, {
+ DATProps,
+ DiskProps,
+ GameProps,
+ ROMProps,
+} from '../types/dats/cmpro/cmProParser.js';
import DAT from '../types/dats/dat.js';
import DATObject, { DATObjectProps } from '../types/dats/datObject.js';
+import Disk from '../types/dats/disk.js';
import Game from '../types/dats/game.js';
import Header from '../types/dats/logiqx/header.js';
import LogiqxDAT from '../types/dats/logiqx/logiqxDat.js';
@@ -19,8 +24,10 @@ import ROM from '../types/dats/rom.js';
import SoftwareListDAT from '../types/dats/softwarelist/softwareListDat.js';
import SoftwareListsDAT from '../types/dats/softwarelist/softwareListsDat.js';
import ExpectedError from '../types/expectedError.js';
+import ArchiveEntry from '../types/files/archives/archiveEntry.js';
import File from '../types/files/file.js';
import { ChecksumBitmask } from '../types/files/fileChecksums.js';
+import FileFactory from '../types/files/fileFactory.js';
import Options from '../types/options.js';
import Scanner from './scanner.js';
@@ -38,8 +45,8 @@ type SmdbRow = {
* representation.
*/
export default class DATScanner extends Scanner {
- constructor(options: Options, progressBar: ProgressBar) {
- super(options, progressBar, DATScanner.name);
+ constructor(options: Options, progressBar: ProgressBar, fileFactory: FileFactory) {
+ super(options, progressBar, fileFactory, DATScanner.name);
}
/**
@@ -47,17 +54,17 @@ export default class DATScanner extends Scanner {
*/
async scan(): Promise {
this.progressBar.logTrace('scanning DAT files');
- await this.progressBar.setSymbol(ProgressBarSymbol.SEARCHING);
- await this.progressBar.reset(0);
+ this.progressBar.setSymbol(ProgressBarSymbol.FILE_SCANNING);
+ this.progressBar.reset(0);
- const datFilePaths = await this.options.scanDatFilesWithoutExclusions(async (increment) => {
- await this.progressBar.incrementTotal(increment);
+ const datFilePaths = await this.options.scanDatFilesWithoutExclusions((increment) => {
+ this.progressBar.incrementTotal(increment);
});
if (datFilePaths.length === 0) {
return [];
}
this.progressBar.logTrace(`found ${datFilePaths.length.toLocaleString()} DAT file${datFilePaths.length !== 1 ? 's' : ''}`);
- await this.progressBar.reset(datFilePaths.length);
+ this.progressBar.reset(datFilePaths.length);
this.progressBar.logTrace('enumerating DAT archives');
const datFiles = await this.getUniqueFilesFromPaths(
@@ -65,9 +72,10 @@ export default class DATScanner extends Scanner {
this.options.getReaderThreads(),
ChecksumBitmask.NONE,
);
- await this.progressBar.reset(datFiles.length);
+ this.progressBar.reset(datFiles.length);
const downloadedDats = await this.downloadDats(datFiles);
+ this.progressBar.reset(downloadedDats.length);
const parsedDats = await this.parseDatFiles(downloadedDats);
this.progressBar.logTrace('done scanning DAT files');
@@ -80,7 +88,7 @@ export default class DATScanner extends Scanner {
}
this.progressBar.logTrace('downloading DATs from URLs');
- await this.progressBar.setSymbol(ProgressBarSymbol.DOWNLOADING);
+ this.progressBar.setSymbol(ProgressBarSymbol.DAT_DOWNLOADING);
return (await Promise.all(datFiles.map(async (datFile) => {
if (!datFile.isURL()) {
@@ -89,6 +97,7 @@ export default class DATScanner extends Scanner {
try {
this.progressBar.logTrace(`${datFile.toString()}: downloading`);
+ // TODO(cemmer): these never get deleted?
const downloadedDatFile = await datFile.downloadToTempPath('dat');
this.progressBar.logTrace(`${datFile.toString()}: downloaded to '${downloadedDatFile.toString()}'`);
return await this.getFilesFromPaths(
@@ -105,12 +114,12 @@ export default class DATScanner extends Scanner {
// Parse each file into a DAT
private async parseDatFiles(datFiles: File[]): Promise {
this.progressBar.logTrace(`parsing ${datFiles.length.toLocaleString()} DAT file${datFiles.length !== 1 ? 's' : ''}`);
- await this.progressBar.setSymbol(ProgressBarSymbol.PARSING_CONTENTS);
+ this.progressBar.setSymbol(ProgressBarSymbol.DAT_PARSING);
return (await new DriveSemaphore(this.options.getReaderThreads()).map(
datFiles,
async (datFile) => {
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
const waitingMessage = `${datFile.toString()} ...`;
this.progressBar.addWaitingMessage(waitingMessage);
@@ -121,7 +130,7 @@ export default class DATScanner extends Scanner {
this.progressBar.logWarn(`${datFile.toString()}: failed to parse DAT file: ${error}`);
}
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
this.progressBar.removeWaitingMessage(waitingMessage);
if (dat && this.shouldFilterOut(dat)) {
@@ -130,7 +139,7 @@ export default class DATScanner extends Scanner {
return dat;
},
))
- .filter(ArrayPoly.filterNotNullish)
+ .filter((dat) => dat !== undefined)
.map((dat) => this.sanitizeDat(dat))
.sort((a, b) => a.getNameShort().localeCompare(b.getNameShort()));
}
@@ -138,7 +147,10 @@ export default class DATScanner extends Scanner {
private async parseDatFile(datFile: File): Promise {
let dat: DAT | undefined;
- if (!dat && await fsPoly.isExecutable(datFile.getFilePath())) {
+ if (!dat
+ && !(datFile instanceof ArchiveEntry)
+ && await fsPoly.isExecutable(datFile.getFilePath())
+ ) {
dat = await this.parseMameListxml(datFile);
}
@@ -196,7 +208,7 @@ export default class DATScanner extends Scanner {
output += chunk.toString();
});
- proc.on('exit', (code) => {
+ proc.on('close', (code) => {
if (code !== null && code > 0) {
reject(new Error(`exit code ${code}`));
return;
@@ -333,6 +345,8 @@ export default class DATScanner extends Scanner {
}
const games = cmproDatGames.flatMap((game) => {
+ const gameName = game.name ?? game.comment;
+
let gameRoms: ROMProps[] = [];
if (game.rom) {
if (Array.isArray(game.rom)) {
@@ -341,16 +355,29 @@ export default class DATScanner extends Scanner {
gameRoms = [game.rom];
}
}
- const gameName = game.name ?? game.comment;
+ const roms = gameRoms.map((entry) => new ROM({
+ name: entry.name ?? '',
+ size: Number.parseInt(entry.size ?? '0', 10),
+ crc32: entry.crc,
+ md5: entry.md5,
+ sha1: entry.sha1,
+ }));
- const roms = gameRoms
- .map((entry) => new ROM({
- name: entry.name ?? '',
- size: Number.parseInt(entry.size ?? '0', 10),
- crc32: entry.crc,
- md5: entry.md5,
- sha1: entry.sha1,
- }));
+ let gameDisks: DiskProps[] = [];
+ if (game.disk) {
+ if (Array.isArray(game.disk)) {
+ gameDisks = game.disk;
+ } else {
+ gameDisks = [game.disk];
+ }
+ }
+ const disks = gameDisks.map((entry) => new Disk({
+ name: entry.name ?? '',
+ size: Number.parseInt(entry.size ?? '0', 10),
+ crc32: entry.crc,
+ md5: entry.md5,
+ sha1: entry.sha1,
+ }));
return new Game({
name: gameName,
@@ -365,6 +392,7 @@ export default class DATScanner extends Scanner {
genre: game.genre?.toString(),
release: undefined,
rom: roms,
+ disk: disks,
});
});
@@ -480,7 +508,7 @@ export default class DATScanner extends Scanner {
const games = dat.getGames()
.map((game) => {
const roms = game.getRoms()
- // ROMs have to have and at least one non-empty checksum
+ // Games have to have at least one ROM with a non-empty checksum
.filter((rom) => this.options.shouldDir2Dat() || (
(rom.getCrc32() === undefined || rom.getCrc32() !== '00000000')
&& (rom.getMd5() === undefined || rom.getMd5() !== 'd41d8cd98f00b204e9800998ecf8427e')
diff --git a/src/modules/dir2DatCreator.ts b/src/modules/dir2DatCreator.ts
index 53a366811..b7b222dbd 100644
--- a/src/modules/dir2DatCreator.ts
+++ b/src/modules/dir2DatCreator.ts
@@ -34,8 +34,8 @@ export default class Dir2DatCreator extends Module {
}
this.progressBar.logTrace(`${dat.getNameShort()}: writing dir2dat`);
- await this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
- await this.progressBar.reset(1);
+ this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
+ this.progressBar.reset(1);
const datDir = this.options.shouldWrite()
? OutputFactory.getDir(this.options, dat)
@@ -52,7 +52,11 @@ export default class Dir2DatCreator extends Module {
.flat()
.reduce((map, releaseCandidate) => {
const key = releaseCandidate.getGame();
- map.set(key, [...(map.get(key) ?? []), releaseCandidate]);
+ if (!map.has(key)) {
+ map.set(key, [releaseCandidate]);
+ } else {
+ map.get(key)?.push(releaseCandidate);
+ }
return map;
}, new Map());
const gamesFromCandidates = [...gamesToCandidates.entries()]
diff --git a/src/modules/directoryCleaner.ts b/src/modules/directoryCleaner.ts
index 190cd5b22..af7c31dfa 100644
--- a/src/modules/directoryCleaner.ts
+++ b/src/modules/directoryCleaner.ts
@@ -35,15 +35,15 @@ export default class DirectoryCleaner extends Module {
}
this.progressBar.logTrace('cleaning files in output');
- await this.progressBar.setSymbol(ProgressBarSymbol.SEARCHING);
- await this.progressBar.reset(0);
+ this.progressBar.setSymbol(ProgressBarSymbol.FILE_SCANNING);
+ this.progressBar.reset(0);
// If there is nothing to clean, then don't do anything
const filesToClean = await this.options.scanOutputFilesWithoutCleanExclusions(
dirsToClean,
filesToExclude,
- async (increment) => {
- await this.progressBar.incrementTotal(increment);
+ (increment) => {
+ this.progressBar.incrementTotal(increment);
},
);
if (filesToClean.length === 0) {
@@ -51,11 +51,11 @@ export default class DirectoryCleaner extends Module {
return [];
}
- await this.progressBar.setSymbol(ProgressBarSymbol.RECYCLING);
+ this.progressBar.setSymbol(ProgressBarSymbol.RECYCLING);
try {
this.progressBar.logTrace(`cleaning ${filesToClean.length.toLocaleString()} file${filesToClean.length !== 1 ? 's' : ''}`);
- await this.progressBar.reset(filesToClean.length);
+ this.progressBar.reset(filesToClean.length);
if (this.options.getCleanDryRun()) {
this.progressBar.logInfo(`paths skipped from cleaning (dry run):\n${filesToClean.map((filePath) => ` ${filePath}`).join('\n')}`);
} else {
@@ -74,7 +74,7 @@ export default class DirectoryCleaner extends Module {
try {
let emptyDirs = await DirectoryCleaner.getEmptyDirs(dirsToClean);
while (emptyDirs.length > 0) {
- await this.progressBar.reset(emptyDirs.length);
+ this.progressBar.reset(emptyDirs.length);
this.progressBar.logTrace(`cleaning ${emptyDirs.length.toLocaleString()} empty director${emptyDirs.length !== 1 ? 'ies' : 'y'}`);
if (this.options.getCleanDryRun()) {
this.progressBar.logInfo(`paths skipped from cleaning (dry run):\n${emptyDirs.map((filePath) => ` ${filePath}`).join('\n')}`);
@@ -102,7 +102,7 @@ export default class DirectoryCleaner extends Module {
} catch (error) {
this.progressBar.logWarn(`failed to recycle ${filePathsChunk.length} path${filePathsChunk.length !== 1 ? 's' : ''}: ${error}`);
}
- await this.progressBar.update(i);
+ this.progressBar.update(i);
}
// ...but if that doesn't work, delete the leftovers
@@ -110,15 +110,19 @@ export default class DirectoryCleaner extends Module {
const existingFilePathsCheck = await Promise.all(filePaths
.map(async (filePath) => existSemaphore.runExclusive(async () => fsPoly.exists(filePath))));
const existingFilePaths = filePaths.filter((filePath, idx) => existingFilePathsCheck.at(idx));
+ if (existingFilePaths.length > 0) {
+ this.progressBar.setSymbol(ProgressBarSymbol.DELETING);
+ }
for (let i = 0; i < existingFilePaths.length; i += Defaults.OUTPUT_CLEANER_BATCH_SIZE) {
const filePathsChunk = existingFilePaths.slice(i, i + Defaults.OUTPUT_CLEANER_BATCH_SIZE);
this.progressBar.logInfo(`deleting cleaned path${filePathsChunk.length !== 1 ? 's' : ''}:\n${filePathsChunk.map((filePath) => ` ${filePath}`).join('\n')}`);
- try {
- await Promise.all(filePathsChunk
- .map(async (filePath) => fsPoly.rm(filePath, { force: true })));
- } catch (error) {
- this.progressBar.logWarn(`failed to delete ${filePathsChunk.length} path${filePathsChunk.length !== 1 ? 's' : ''}: ${error}`);
- }
+ await Promise.all(filePathsChunk.map(async (filePath) => {
+ try {
+ await fsPoly.rm(filePath, { force: true });
+ } catch (error) {
+ this.progressBar.logError(`${filePath}: failed to delete: ${error}`);
+ }
+ }));
}
}
@@ -144,7 +148,7 @@ export default class DirectoryCleaner extends Module {
} catch (error) {
this.progressBar.logWarn(`failed to move ${filePath} -> ${backupPath}: ${error}`);
}
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
});
}));
}
diff --git a/src/modules/fixdatCreator.ts b/src/modules/fixdatCreator.ts
index 0ec411e15..39a731796 100644
--- a/src/modules/fixdatCreator.ts
+++ b/src/modules/fixdatCreator.ts
@@ -38,8 +38,8 @@ export default class FixdatCreator extends Module {
}
this.progressBar.logTrace(`${originalDat.getNameShort()}: generating a fixdat`);
- await this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
- await this.progressBar.reset(1);
+ this.progressBar.setSymbol(ProgressBarSymbol.WRITING);
+ this.progressBar.reset(1);
// Create an easily searchable index of every ROM that has a ReleaseCandidate
const writtenRomHashCodes = new Set([...parentsToCandidates.values()]
@@ -60,7 +60,7 @@ export default class FixdatCreator extends Module {
try {
fixdatDir = this.getDatOutputDirRoot(originalDat);
} catch (error) {
- this.progressBar.logWarn(`failed to: ${error}`);
+ this.progressBar.logWarn(`${originalDat.getNameShort()}: failed to get output directory: ${error}`);
}
}
if (!await fsPoly.exists(fixdatDir)) {
diff --git a/src/modules/module.ts b/src/modules/module.ts
index 6ca10e390..cf57b3074 100644
--- a/src/modules/module.ts
+++ b/src/modules/module.ts
@@ -7,6 +7,7 @@ export default abstract class Module {
protected readonly progressBar: ProgressBar;
protected constructor(progressBar: ProgressBar, loggerPrefix: string) {
- this.progressBar = progressBar.withLoggerPrefix(loggerPrefix);
+ this.progressBar = progressBar;
+ this.progressBar.setLoggerPrefix(loggerPrefix);
}
}
diff --git a/src/modules/movedRomDeleter.ts b/src/modules/movedRomDeleter.ts
index 27fc5fe80..d9ad7c6c1 100644
--- a/src/modules/movedRomDeleter.ts
+++ b/src/modules/movedRomDeleter.ts
@@ -1,4 +1,7 @@
+import { Semaphore } from 'async-mutex';
+
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
+import Defaults from '../globals/defaults.js';
import ArrayPoly from '../polyfill/arrayPoly.js';
import fsPoly from '../polyfill/fsPoly.js';
import DAT from '../types/dats/dat.js';
@@ -31,8 +34,8 @@ export default class MovedROMDeleter extends Module {
}
this.progressBar.logTrace('deleting moved ROMs');
- await this.progressBar.setSymbol(ProgressBarSymbol.FILTERING);
- await this.progressBar.reset(movedRoms.length);
+ this.progressBar.setSymbol(ProgressBarSymbol.CANDIDATE_FILTERING);
+ this.progressBar.reset(movedRoms.length);
const fullyConsumedFiles = this.filterOutPartiallyConsumedArchives(movedRoms, inputRoms);
@@ -41,21 +44,31 @@ export default class MovedROMDeleter extends Module {
datsToWrittenFiles,
);
- await this.progressBar.setSymbol(ProgressBarSymbol.DELETING);
- await this.progressBar.reset(filePathsToDelete.length);
- this.progressBar.logTrace(`deleting ${filePathsToDelete.length.toLocaleString()} moved file${filePathsToDelete.length !== 1 ? 's' : ''}`);
-
- await Promise.all(filePathsToDelete.map(async (filePath) => {
- this.progressBar.logInfo(`deleting moved file: ${filePath}`);
- try {
- await fsPoly.rm(filePath, { force: true });
- } catch {
- this.progressBar.logError(`${filePath}: failed to delete`);
- }
- }));
+ const existSemaphore = new Semaphore(Defaults.OUTPUT_CLEANER_BATCH_SIZE);
+ const existingFilePathsCheck = await Promise.all(filePathsToDelete
+ .map(async (filePath) => existSemaphore.runExclusive(async () => fsPoly.exists(filePath))));
+ const existingFilePaths = filePathsToDelete
+ .filter((filePath, idx) => existingFilePathsCheck.at(idx));
+
+ this.progressBar.setSymbol(ProgressBarSymbol.DELETING);
+ this.progressBar.reset(existingFilePaths.length);
+ this.progressBar.logTrace(`deleting ${existingFilePaths.length.toLocaleString()} moved file${existingFilePaths.length !== 1 ? 's' : ''}`);
+
+ const filePathChunks = existingFilePaths
+ .reduce(ArrayPoly.reduceChunk(Defaults.OUTPUT_CLEANER_BATCH_SIZE), []);
+ for (const filePathChunk of filePathChunks) {
+ this.progressBar.logInfo(`deleting moved file${filePathChunk.length !== 1 ? 's' : ''}:\n${filePathChunk.map((filePath) => ` ${filePath}`).join('\n')}`);
+ await Promise.all(filePathChunk.map(async (filePath) => {
+ try {
+ await fsPoly.rm(filePath, { force: true });
+ } catch (error) {
+ this.progressBar.logError(`${filePath}: failed to delete: ${error}`);
+ }
+ }));
+ }
this.progressBar.logTrace('done deleting moved ROMs');
- return filePathsToDelete;
+ return existingFilePaths;
}
/**
@@ -123,7 +136,7 @@ export default class MovedROMDeleter extends Module {
return filePath;
})
- .filter(ArrayPoly.filterNotNullish);
+ .filter((filePath) => filePath !== undefined);
}
private static groupFilesByFilePath(files: File[]): Map {
diff --git a/src/modules/patchScanner.ts b/src/modules/patchScanner.ts
index 3c6b4b701..9e1577571 100644
--- a/src/modules/patchScanner.ts
+++ b/src/modules/patchScanner.ts
@@ -1,8 +1,8 @@
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
import DriveSemaphore from '../driveSemaphore.js';
-import ArrayPoly from '../polyfill/arrayPoly.js';
import File from '../types/files/file.js';
import { ChecksumBitmask } from '../types/files/fileChecksums.js';
+import FileFactory from '../types/files/fileFactory.js';
import Options from '../types/options.js';
import Patch from '../types/patches/patch.js';
import PatchFactory from '../types/patches/patchFactory.js';
@@ -12,8 +12,8 @@ import Scanner from './scanner.js';
* Scan for {@link Patch}es and parse them into the correct supported type.
*/
export default class PatchScanner extends Scanner {
- constructor(options: Options, progressBar: ProgressBar) {
- super(options, progressBar, PatchScanner.name);
+ constructor(options: Options, progressBar: ProgressBar, fileFactory: FileFactory) {
+ super(options, progressBar, fileFactory, PatchScanner.name);
}
/**
@@ -21,26 +21,26 @@ export default class PatchScanner extends Scanner {
*/
async scan(): Promise {
this.progressBar.logTrace('scanning patch files');
- await this.progressBar.setSymbol(ProgressBarSymbol.SEARCHING);
- await this.progressBar.reset(0);
+ this.progressBar.setSymbol(ProgressBarSymbol.FILE_SCANNING);
+ this.progressBar.reset(0);
- const patchFilePaths = await this.options.scanPatchFilesWithoutExclusions(async (increment) => {
- await this.progressBar.incrementTotal(increment);
+ const patchFilePaths = await this.options.scanPatchFilesWithoutExclusions((increment) => {
+ this.progressBar.incrementTotal(increment);
});
this.progressBar.logTrace(`found ${patchFilePaths.length.toLocaleString()} patch file${patchFilePaths.length !== 1 ? 's' : ''}`);
- await this.progressBar.reset(patchFilePaths.length);
+ this.progressBar.reset(patchFilePaths.length);
- const files = await this.getUniqueFilesFromPaths(
+ const patchFiles = await this.getUniqueFilesFromPaths(
patchFilePaths,
this.options.getReaderThreads(),
ChecksumBitmask.NONE,
);
- await this.progressBar.reset(files.length);
+ this.progressBar.reset(patchFiles.length);
const patches = (await new DriveSemaphore(this.options.getReaderThreads()).map(
- files,
+ patchFiles,
async (file) => {
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
const waitingMessage = `${file.toString()} ...`;
this.progressBar.addWaitingMessage(waitingMessage);
@@ -50,11 +50,12 @@ export default class PatchScanner extends Scanner {
this.progressBar.logWarn(`${file.toString()}: failed to parse patch: ${error}`);
return undefined;
} finally {
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
this.progressBar.removeWaitingMessage(waitingMessage);
}
},
- )).filter(ArrayPoly.filterNotNullish);
+ ))
+ .filter((patch) => patch !== undefined);
this.progressBar.logTrace('done scanning patch files');
return patches;
diff --git a/src/modules/reportGenerator.ts b/src/modules/reportGenerator.ts
index 81bf4abbb..08563db2f 100644
--- a/src/modules/reportGenerator.ts
+++ b/src/modules/reportGenerator.ts
@@ -87,7 +87,7 @@ export default class ReportGenerator extends Module {
this.progressBar.logTrace(`wrote ${datStatuses.length.toLocaleString()} CSV row${datStatuses.length !== 1 ? 's' : ''}: ${reportPath}`);
this.progressBar.logTrace('done generating report');
- await this.progressBar.done(reportPath);
- await this.progressBar.freeze();
+ this.progressBar.done(reportPath);
+ this.progressBar.freeze();
}
}
diff --git a/src/modules/romHeaderProcessor.ts b/src/modules/romHeaderProcessor.ts
index de9b6fcda..3471a1019 100644
--- a/src/modules/romHeaderProcessor.ts
+++ b/src/modules/romHeaderProcessor.ts
@@ -2,7 +2,7 @@ import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
import DriveSemaphore from '../driveSemaphore.js';
import ArchiveEntry from '../types/files/archives/archiveEntry.js';
import File from '../types/files/file.js';
-import FileCache from '../types/files/fileCache.js';
+import FileFactory from '../types/files/fileFactory.js';
import ROMHeader from '../types/files/romHeader.js';
import Options from '../types/options.js';
import Module from './module.js';
@@ -14,9 +14,15 @@ import Module from './module.js';
export default class ROMHeaderProcessor extends Module {
private readonly options: Options;
- constructor(options: Options, progressBar: ProgressBar) {
+ private readonly fileFactory: FileFactory;
+
+ private readonly driveSemaphore: DriveSemaphore;
+
+ constructor(options: Options, progressBar: ProgressBar, fileFactory: FileFactory) {
super(progressBar, ROMHeaderProcessor.name);
this.options = options;
+ this.fileFactory = fileFactory;
+ this.driveSemaphore = new DriveSemaphore(this.options.getReaderThreads());
}
/**
@@ -27,14 +33,25 @@ export default class ROMHeaderProcessor extends Module {
return inputRomFiles;
}
- this.progressBar.logTrace('processing file headers');
- await this.progressBar.setSymbol(ProgressBarSymbol.DETECTING_HEADERS);
- await this.progressBar.reset(inputRomFiles.length);
+ const filesThatNeedProcessing = inputRomFiles
+ .filter((inputFile) => this.fileNeedsProcessing(inputFile))
+ .length;
+ if (filesThatNeedProcessing === 0) {
+ this.progressBar.logTrace('no ROMs need their header processed');
+ return inputRomFiles;
+ }
+
+ this.progressBar.logTrace(`processing headers in ${filesThatNeedProcessing.toLocaleString()} ROM${filesThatNeedProcessing !== 1 ? 's' : ''}`);
+ this.progressBar.setSymbol(ProgressBarSymbol.ROM_HEADER_DETECTION);
+ this.progressBar.reset(filesThatNeedProcessing);
+
+ const parsedFiles = await Promise.all(inputRomFiles.map(async (inputFile) => {
+ if (!this.fileNeedsProcessing(inputFile)) {
+ return inputFile;
+ }
- const parsedFiles = await new DriveSemaphore(this.options.getReaderThreads()).map(
- inputRomFiles,
- async (inputFile) => {
- await this.progressBar.incrementProgress();
+ return this.driveSemaphore.runExclusive(inputFile, async () => {
+ this.progressBar.incrementProgress();
const waitingMessage = `${inputFile.toString()} ...`;
this.progressBar.addWaitingMessage(waitingMessage);
@@ -47,11 +64,11 @@ export default class ROMHeaderProcessor extends Module {
}
this.progressBar.removeWaitingMessage(waitingMessage);
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
return fileWithHeader;
- },
- );
+ });
+ }));
const headeredRomsCount = parsedFiles.filter((romFile) => romFile.getFileHeader()).length;
this.progressBar.logTrace(`found headers in ${headeredRomsCount.toLocaleString()} ROM${headeredRomsCount !== 1 ? 's' : ''}`);
@@ -60,7 +77,7 @@ export default class ROMHeaderProcessor extends Module {
return parsedFiles;
}
- private async getFileWithHeader(inputFile: File): Promise {
+ private fileNeedsProcessing(inputFile: File): boolean {
/**
* If the input file is from an archive, and we're not zipping or extracting, then we have no
* chance to remove the header, so we shouldn't bother detecting one.
@@ -70,23 +87,21 @@ export default class ROMHeaderProcessor extends Module {
&& !this.options.shouldZip()
&& !this.options.shouldExtract()
) {
- return inputFile;
+ return false;
}
- // Should get FileHeader from File, try to
- if (ROMHeader.headerFromFilename(inputFile.getExtractedFilePath()) !== undefined
- || this.options.shouldReadFileForHeader(inputFile.getExtractedFilePath())
- ) {
- this.progressBar.logTrace(`${inputFile.toString()}: reading potentially headered file by file contents`);
- const headerForFileStream = await FileCache.getOrComputeFileHeader(inputFile);
- if (headerForFileStream) {
- this.progressBar.logTrace(`${inputFile.toString()}: found header by file contents: ${headerForFileStream.getHeaderedFileExtension()}`);
- return inputFile.withFileHeader(headerForFileStream);
- }
- this.progressBar.logTrace(`${inputFile.toString()}: didn't find header by file contents`);
- }
+ return ROMHeader.headerFromFilename(inputFile.getExtractedFilePath()) !== undefined
+ || this.options.shouldReadFileForHeader(inputFile.getExtractedFilePath());
+ }
- // Should not get FileHeader
+ private async getFileWithHeader(inputFile: File): Promise {
+ this.progressBar.logTrace(`${inputFile.toString()}: reading potentially headered file by file contents`);
+ const headerForFileStream = await this.fileFactory.headerFrom(inputFile);
+ if (headerForFileStream) {
+ this.progressBar.logTrace(`${inputFile.toString()}: found header by file contents: ${headerForFileStream.getHeaderedFileExtension()}`);
+ return inputFile.withFileHeader(headerForFileStream);
+ }
+ this.progressBar.logTrace(`${inputFile.toString()}: didn't find header by file contents`);
return inputFile;
}
}
diff --git a/src/modules/romIndexer.ts b/src/modules/romIndexer.ts
index 7fd488122..ec83d7121 100644
--- a/src/modules/romIndexer.ts
+++ b/src/modules/romIndexer.ts
@@ -3,8 +3,10 @@ import path from 'node:path';
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
import FsPoly from '../polyfill/fsPoly.js';
import ArchiveEntry from '../types/files/archives/archiveEntry.js';
+import Chd from '../types/files/archives/chd/chd.js';
+import Maxcso from '../types/files/archives/maxcso/maxcso.js';
import Rar from '../types/files/archives/rar.js';
-import SevenZip from '../types/files/archives/sevenZip.js';
+import SevenZip from '../types/files/archives/sevenZip/sevenZip.js';
import Tar from '../types/files/archives/tar.js';
import Zip from '../types/files/archives/zip.js';
import File from '../types/files/file.js';
@@ -27,10 +29,10 @@ export default class ROMIndexer extends Module {
/**
* Index files.
*/
- async index(files: File[]): Promise {
+ index(files: File[]): IndexedFiles {
this.progressBar.logTrace(`indexing ${files.length.toLocaleString()} file${files.length !== 1 ? 's' : ''}`);
- await this.progressBar.setSymbol(ProgressBarSymbol.INDEXING);
- await this.progressBar.reset(files.length);
+ this.progressBar.setSymbol(ProgressBarSymbol.ROM_INDEXING);
+ this.progressBar.reset(files.length);
// Index the files
const result = IndexedFiles.fromFiles(files);
@@ -45,12 +47,12 @@ export default class ROMIndexer extends Module {
private sortMap(checksumsToFiles: ChecksumsToFiles): void {
const outputDir = path.resolve(this.options.getOutputDirRoot());
- const outputDirDisk = FsPoly.disksSync().find((mount) => outputDir.startsWith(mount));
+ const outputDirDisk = FsPoly.diskResolved(outputDir);
[...checksumsToFiles.values()]
.forEach((files) => files
.sort((fileOne, fileTwo) => {
- // Prefer un-archived files
+ // Prefer un-archived files because they're less expensive to process
const fileOneArchived = ROMIndexer.archiveEntryPriority(fileOne);
const fileTwoArchived = ROMIndexer.archiveEntryPriority(fileTwo);
if (fileOneArchived !== fileTwoArchived) {
@@ -61,6 +63,7 @@ export default class ROMIndexer extends Module {
// This is in case the output file is invalid and we're trying to overwrite it with
// something else. Otherwise, we'll just attempt to overwrite the invalid output file with
// itself, still resulting in an invalid output file.
+ // TODO(cemmer): only do this when overwriting files in some way?
const fileOneInOutput = path.resolve(fileOne.getFilePath()).startsWith(outputDir) ? 1 : 0;
const fileTwoInOutput = path.resolve(fileTwo.getFilePath()).startsWith(outputDir) ? 1 : 0;
if (fileOneInOutput !== fileTwoInOutput) {
@@ -69,6 +72,7 @@ export default class ROMIndexer extends Module {
// Then, prefer files that are on the same disk for fs efficiency see {@link FsPoly#mv}
if (outputDirDisk) {
+ // TODO(cemmer): only do this when not copying files?
const fileOneInOutputDisk = path.resolve(fileOne.getFilePath())
.startsWith(outputDirDisk) ? 0 : 1;
const fileTwoInOutputDisk = path.resolve(fileTwo.getFilePath())
@@ -97,6 +101,10 @@ export default class ROMIndexer extends Module {
return 3;
} if (file.getArchive() instanceof SevenZip) {
return 4;
+ } if (file.getArchive() instanceof Maxcso) {
+ return 5;
+ } if (file.getArchive() instanceof Chd) {
+ return 6;
}
return 99;
}
diff --git a/src/modules/romScanner.ts b/src/modules/romScanner.ts
index f84f9f46f..9e625a5e6 100644
--- a/src/modules/romScanner.ts
+++ b/src/modules/romScanner.ts
@@ -1,6 +1,7 @@
import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js';
import File from '../types/files/file.js';
import { ChecksumBitmask } from '../types/files/fileChecksums.js';
+import FileFactory from '../types/files/fileFactory.js';
import Options from '../types/options.js';
import Scanner from './scanner.js';
@@ -9,8 +10,8 @@ import Scanner from './scanner.js';
* representation.
*/
export default class ROMScanner extends Scanner {
- constructor(options: Options, progressBar: ProgressBar) {
- super(options, progressBar, ROMScanner.name);
+ constructor(options: Options, progressBar: ProgressBar, fileFactory: FileFactory) {
+ super(options, progressBar, fileFactory, ROMScanner.name);
}
/**
@@ -21,14 +22,15 @@ export default class ROMScanner extends Scanner {
checksumArchives = false,
): Promise {
this.progressBar.logTrace('scanning ROM files');
- await this.progressBar.setSymbol(ProgressBarSymbol.SEARCHING);
- await this.progressBar.reset(0);
+ this.progressBar.setSymbol(ProgressBarSymbol.FILE_SCANNING);
+ this.progressBar.reset(0);
- const romFilePaths = await this.options.scanInputFilesWithoutExclusions(async (increment) => {
- await this.progressBar.incrementTotal(increment);
+ const romFilePaths = await this.options.scanInputFilesWithoutExclusions((increment) => {
+ this.progressBar.incrementTotal(increment);
});
this.progressBar.logTrace(`found ${romFilePaths.length.toLocaleString()} ROM file${romFilePaths.length !== 1 ? 's' : ''}`);
- await this.progressBar.reset(romFilePaths.length);
+ this.progressBar.setSymbol(ProgressBarSymbol.ROM_HASHING);
+ this.progressBar.reset(romFilePaths.length);
const files = await this.getFilesFromPaths(
romFilePaths,
diff --git a/src/modules/scanner.ts b/src/modules/scanner.ts
index 191a4a797..03bf7eb2b 100644
--- a/src/modules/scanner.ts
+++ b/src/modules/scanner.ts
@@ -1,11 +1,15 @@
+import { CHDInfo, CHDType } from 'chdman';
+
import ProgressBar from '../console/progressBar.js';
import DriveSemaphore from '../driveSemaphore.js';
-import ElasticSemaphore from '../elasticSemaphore.js';
-import Defaults from '../globals/defaults.js';
import ArrayPoly from '../polyfill/arrayPoly.js';
import fsPoly from '../polyfill/fsPoly.js';
import ArchiveEntry from '../types/files/archives/archiveEntry.js';
+import Chd from '../types/files/archives/chd/chd.js';
+import Gzip from '../types/files/archives/sevenZip/gzip.js';
+import Tar from '../types/files/archives/tar.js';
import File from '../types/files/file.js';
+import { ChecksumBitmask } from '../types/files/fileChecksums.js';
import FileFactory from '../types/files/fileFactory.js';
import Options from '../types/options.js';
import Module from './module.js';
@@ -14,17 +18,19 @@ import Module from './module.js';
* The base class for every input file scanner class.
*/
export default abstract class Scanner extends Module {
- // WARN(cemmer): there is an undocumented semaphore max value that can be used, the full
- // 4,700,372,992 bytes of a DVD+R will cause runExclusive() to never run or return.
- private static readonly FILESIZE_SEMAPHORE = new ElasticSemaphore(
- Defaults.MAX_READ_WRITE_CONCURRENT_KILOBYTES,
- );
-
protected readonly options: Options;
- protected constructor(options: Options, progressBar: ProgressBar, loggerPrefix: string) {
+ private readonly fileFactory: FileFactory;
+
+ protected constructor(
+ options: Options,
+ progressBar: ProgressBar,
+ fileFactory: FileFactory,
+ loggerPrefix: string,
+ ) {
super(progressBar, loggerPrefix);
this.options = options;
+ this.fileFactory = fileFactory;
}
protected async getFilesFromPaths(
@@ -36,14 +42,15 @@ export default abstract class Scanner extends Module {
return (await new DriveSemaphore(threads).map(
filePaths,
async (inputFile) => {
- await this.progressBar.incrementProgress();
+ this.progressBar.incrementProgress();
const waitingMessage = `${inputFile} ...`;
this.progressBar.addWaitingMessage(waitingMessage);
const files = await this.getFilesFromPath(inputFile, checksumBitmask, checksumArchives);
+ await this.logWarnings(files);
this.progressBar.removeWaitingMessage(waitingMessage);
- await this.progressBar.incrementDone();
+ this.progressBar.incrementDone();
return files;
},
)).flat();
@@ -65,36 +72,62 @@ export default abstract class Scanner extends Module {
checksumArchives = false,
): Promise {
try {
- const totalKilobytes = await fsPoly.size(filePath) / 1024;
- const files = await Scanner.FILESIZE_SEMAPHORE.runExclusive(
- async () => {
- if (await fsPoly.isSymlink(filePath)) {
- const realFilePath = await fsPoly.readlinkResolved(filePath);
- if (!await fsPoly.exists(realFilePath)) {
- this.progressBar.logWarn(`${filePath}: broken symlink, '${realFilePath}' doesn't exist`);
- return [];
- }
- }
-
- const filesFromPath = await FileFactory.filesFrom(filePath, checksumBitmask);
-
- const fileIsArchive = filesFromPath.some((file) => file instanceof ArchiveEntry);
- if (checksumArchives && fileIsArchive) {
- filesFromPath.push(await FileFactory.fileFrom(filePath, checksumBitmask));
- }
-
- return filesFromPath;
- },
- totalKilobytes,
+ if (await fsPoly.isSymlink(filePath)) {
+ const realFilePath = await fsPoly.readlinkResolved(filePath);
+ if (!await fsPoly.exists(realFilePath)) {
+ this.progressBar.logWarn(`${filePath}: broken symlink, '${realFilePath}' doesn't exist`);
+ return [];
+ }
+ }
+
+ const filesFromPath = await this.fileFactory.filesFrom(
+ filePath,
+ checksumBitmask,
+ this.options.getInputChecksumQuick() ? ChecksumBitmask.NONE : checksumBitmask,
);
- if (files.length === 0) {
+ const fileIsArchive = filesFromPath.some((file) => file instanceof ArchiveEntry);
+ if (checksumArchives && fileIsArchive) {
+ filesFromPath.push(await this.fileFactory.fileFrom(filePath, checksumBitmask));
+ }
+
+ if (filesFromPath.length === 0) {
this.progressBar.logWarn(`${filePath}: found no files in path`);
}
- return files;
+ return filesFromPath;
} catch (error) {
this.progressBar.logError(`${filePath}: failed to parse file: ${error}`);
return [];
}
}
+
+ private async logWarnings(files: File[]): Promise {
+ if (this.options.getInputChecksumQuick()) {
+ const archiveWithoutChecksums = files
+ .filter((file) => file instanceof ArchiveEntry)
+ .map((archiveEntry) => archiveEntry.getArchive())
+ .find((archive) => archive instanceof Gzip || archive instanceof Tar);
+ if (archiveWithoutChecksums !== undefined) {
+ this.progressBar.logWarn(`${archiveWithoutChecksums.getFilePath()}: quick checksums will skip ${archiveWithoutChecksums.getExtension()} files`);
+ return;
+ }
+
+ const chdInfos = await Promise.all(files
+ .filter((file) => file instanceof ArchiveEntry)
+ .map((archiveEntry) => archiveEntry.getArchive())
+ .filter((archive) => archive instanceof Chd)
+ .map(async (chd) => ([chd, await chd.getInfo()] satisfies [Chd, CHDInfo])));
+
+ const cdRom = chdInfos.find(([, info]) => info.type === CHDType.CD_ROM);
+ if (cdRom !== undefined) {
+ this.progressBar.logWarn(`${cdRom[0].getFilePath()}: quick checksums will skip .cue/.bin files in CD-ROM CHDs`);
+ return;
+ }
+
+ const gdRom = chdInfos.find(([, info]) => info.type === CHDType.GD_ROM);
+ if (gdRom !== undefined) {
+ this.progressBar.logWarn(`${gdRom[0].getFilePath()}: quick checksums will skip .gdi/.bin/.raw files in GD-ROM CHDs`);
+ }
+ }
+ }
}
diff --git a/src/polyfill/arrayPoly.ts b/src/polyfill/arrayPoly.ts
index 063a01647..9ecc430e8 100644
--- a/src/polyfill/arrayPoly.ts
+++ b/src/polyfill/arrayPoly.ts
@@ -1,22 +1,4 @@
export default class ArrayPoly {
- /**
- * Filter out nullish values from an array, in a way that TypeScript can understand how the
- * resulting element type has changed. For example, TypeScript (as of v5.1.3) will tell you that:
- *
- *
- * [1, 2, undefined, 4, undefined].filter((val) => val);
- *
- * [1, 2, undefined, 4, undefined].filter(ArrayPoly.filterNotNullish);
- *
- */
- public static filterNotNullish(value: T | null | undefined): value is T {
- return value !== null && value !== undefined;
- }
-
/**
* Filter elements in an array to only unique values, using the result of a mapper function to
* test for equality. Usage:
@@ -29,7 +11,12 @@ export default class ArrayPoly {
mapper: (arg: T) => V,
): (value: T, idx: number, values: T[]) => boolean {
const seenMappedValues = new Set();
- return (value: T): boolean => {
+ return (value, idx, values): boolean => {
+ if (values.length <= 1) {
+ // Arrays that are empty or only have one element are already unique
+ return true;
+ }
+
const mapped = mapper(value);
if (!seenMappedValues.has(mapped)) {
seenMappedValues.add(mapped);
@@ -76,6 +63,11 @@ export default class ArrayPoly {
*/
public static reduceUnique(): (previous: T[], current: T, idx: number, array: T[]) => T[] {
return (previous: T[], current: T, idx: number, array: T[]): T[] => {
+ if (array.length <= 1) {
+ // Arrays that are empty or only have one element are already unique
+ return array;
+ }
+
if (idx === 0) {
return [...new Set(array)];
}
diff --git a/src/polyfill/fsPoly.ts b/src/polyfill/fsPoly.ts
index a387e830e..5718c60c5 100644
--- a/src/polyfill/fsPoly.ts
+++ b/src/polyfill/fsPoly.ts
@@ -8,9 +8,9 @@ import util from 'node:util';
import { isNotJunk } from 'junk';
import nodeDiskInfo from 'node-disk-info';
+import { Memoize } from 'typescript-memoize';
import ExpectedError from '../types/expectedError.js';
-import ArrayPoly from './arrayPoly.js';
export type FsWalkCallback = (increment: number) => void;
@@ -18,13 +18,31 @@ export default class FsPoly {
static readonly FILE_READING_CHUNK_SIZE = 64 * 1024; // 64KiB, Node.js v22 default
// Assume that all drives we're reading from or writing to were already mounted at startup
- public static readonly DRIVES = nodeDiskInfo.getDiskInfoSync();
+ private static readonly DRIVES = nodeDiskInfo.getDiskInfoSync();
- static async canSymlink(tempDir: string): Promise {
- const source = await this.mktemp(path.join(tempDir, 'source'));
+ static async canHardlink(dirPath: string): Promise {
+ const source = await this.mktemp(path.join(dirPath, 'source'));
try {
await this.touch(source);
- const target = await this.mktemp(path.join(tempDir, 'target'));
+ const target = await this.mktemp(path.join(dirPath, 'target'));
+ try {
+ await this.hardlink(source, target);
+ return await this.exists(target);
+ } finally {
+ await this.rm(target, { force: true });
+ }
+ } catch {
+ return false;
+ } finally {
+ await this.rm(source, { force: true });
+ }
+ }
+
+ static async canSymlink(dirPath: string): Promise {
+ const source = await this.mktemp(path.join(dirPath, 'source'));
+ try {
+ await this.touch(source);
+ const target = await this.mktemp(path.join(dirPath, 'target'));
try {
await this.symlink(source, target);
return await this.exists(target);
@@ -78,10 +96,17 @@ export default class FsPoly {
return (await Promise.all(
readDir.map(async (filePath) => (await this.isDirectory(filePath) ? filePath : undefined)),
- )).filter(ArrayPoly.filterNotNullish);
+ ))
+ .filter((childDir) => childDir !== undefined);
}
- static disksSync(): string[] {
+ static diskResolved(filePath: string): string | undefined {
+ const filePathResolved = path.resolve(filePath);
+ return this.disksSync().find((mountPath) => filePathResolved.startsWith(mountPath));
+ }
+
+ @Memoize()
+ private static disksSync(): string[] {
return FsPoly.DRIVES
.filter((drive) => drive.available > 0)
.map((drive) => drive.mounted)
@@ -127,6 +152,19 @@ export default class FsPoly {
}
}
+ static isDirectorySync(pathLike: string): boolean {
+ try {
+ const lstat = fs.lstatSync(pathLike);
+ if (lstat.isSymbolicLink()) {
+ const link = this.readlinkResolvedSync(pathLike);
+ return this.isDirectorySync(link);
+ }
+ return lstat.isDirectory();
+ } catch {
+ return false;
+ }
+ }
+
static async isExecutable(pathLike: PathLike): Promise {
try {
await fs.promises.access(pathLike, fs.constants.X_OK);
@@ -136,6 +174,14 @@ export default class FsPoly {
}
}
+ static async isHardlink(pathLike: PathLike): Promise {
+ try {
+ return (await this.stat(pathLike)).nlink > 1;
+ } catch {
+ return false;
+ }
+ }
+
static isSamba(filePath: string): boolean {
const normalizedPath = filePath.replace(/[\\/]/g, path.sep);
if (normalizedPath.startsWith(`${path.sep}${path.sep}`) && normalizedPath !== os.devNull) {
@@ -163,6 +209,14 @@ export default class FsPoly {
}
}
+ static isSymlinkSync(pathLike: PathLike): boolean {
+ try {
+ return fs.lstatSync(pathLike).isSymbolicLink();
+ } catch {
+ return false;
+ }
+ }
+
static async isWritable(filePath: string): Promise {
const exists = await this.exists(filePath);
try {
@@ -172,7 +226,7 @@ export default class FsPoly {
return false;
} finally {
if (!exists) {
- await this.rm(filePath);
+ await this.rm(filePath, { force: true });
}
}
}
@@ -227,23 +281,6 @@ export default class FsPoly {
}
static async mv(oldPath: string, newPath: string, attempt = 1): Promise {
- /**
- * WARN(cemmer): {@link fs.rename} appears to be VERY memory intensive when copying across
- * drives! Instead, we'll use stream piping to keep memory usage low.
- */
- if (this.onDifferentDrives(oldPath, newPath)) {
- const read = fs.createReadStream(oldPath, {
- highWaterMark: this.FILE_READING_CHUNK_SIZE,
- });
- await new Promise((resolve, reject) => {
- const write = fs.createWriteStream(newPath);
- write.on('close', resolve);
- write.on('error', reject);
- read.pipe(write);
- });
- return this.rm(oldPath, { force: true });
- }
-
try {
return await fs.promises.rename(oldPath, newPath);
} catch (error) {
@@ -253,11 +290,11 @@ export default class FsPoly {
}
// Backoff with jitter
- if (attempt >= 3) {
+ if (attempt >= 5) {
throw error;
}
await new Promise((resolve) => {
- setTimeout(resolve, Math.random() * (2 ** (attempt - 1) * 100));
+ setTimeout(resolve, Math.random() * (2 ** (attempt - 1) * 10));
});
// Attempt to resolve Windows' "EBUSY: resource busy or locked"
@@ -267,14 +304,10 @@ export default class FsPoly {
}
private static onDifferentDrives(one: string, two: string): boolean {
- const oneResolved = path.resolve(one);
- const twoResolved = path.resolve(two);
- if (path.dirname(oneResolved) === path.dirname(twoResolved)) {
+ if (path.dirname(one) === path.dirname(two)) {
return false;
}
- const driveMounts = this.disksSync();
- return driveMounts.find((mount) => oneResolved.startsWith(mount))
- !== driveMounts.find((mount) => twoResolved.startsWith(mount));
+ return this.diskResolved(one) !== this.diskResolved(two);
}
static async readlink(pathLike: PathLike): Promise {
@@ -284,6 +317,13 @@ export default class FsPoly {
return fs.promises.readlink(pathLike);
}
+ static readlinkSync(pathLike: PathLike): string {
+ if (!this.isSymlinkSync(pathLike)) {
+ throw new ExpectedError(`can't readlink of non-symlink: ${pathLike}`);
+ }
+ return fs.readlinkSync(pathLike);
+ }
+
static async readlinkResolved(link: string): Promise {
const source = await this.readlink(link);
if (path.isAbsolute(source)) {
@@ -292,6 +332,14 @@ export default class FsPoly {
return path.join(path.dirname(link), source);
}
+ static readlinkResolvedSync(link: string): string {
+ const source = this.readlinkSync(link);
+ if (path.isAbsolute(source)) {
+ return source;
+ }
+ return path.join(path.dirname(link), source);
+ }
+
static async realpath(pathLike: PathLike): Promise {
if (!await this.exists(pathLike)) {
throw new ExpectedError(`can't get realpath of non-existent path: ${pathLike}`);
@@ -311,7 +359,7 @@ export default class FsPoly {
if (optionsWithRetry?.force) {
return;
}
- throw new Error(`can't rm, path doesn't exist: ${pathLike}`);
+ throw new ExpectedError(`can't rm, path doesn't exist: ${pathLike}`);
}
if (await this.isDirectory(pathLike)) {
@@ -324,6 +372,31 @@ export default class FsPoly {
}
}
+ static rmSync(pathLike: string, options: RmOptions = {}): void {
+ const optionsWithRetry = {
+ maxRetries: 2,
+ ...options,
+ };
+
+ try {
+ fs.accessSync(pathLike);
+ } catch {
+ if (optionsWithRetry?.force) {
+ return;
+ }
+ throw new ExpectedError(`can't rmSync, path doesn't exist: ${pathLike}`);
+ }
+
+ if (this.isDirectorySync(pathLike)) {
+ fs.rmSync(pathLike, {
+ ...optionsWithRetry,
+ recursive: true,
+ });
+ } else {
+ fs.unlinkSync(pathLike);
+ }
+ }
+
/**
* Note: this will follow symlinks and get the size of the target.
*/
@@ -382,22 +455,6 @@ export default class FsPoly {
await file.close();
}
- static touchSync(filePath: string): void {
- const dirname = path.dirname(filePath);
- if (!fs.existsSync(dirname)) {
- fs.mkdirSync(dirname, { recursive: true });
- }
-
- // Create the file if it doesn't already exist
- const file = fs.openSync(filePath, 'a');
-
- // Ensure the file's `atime` and `mtime` are updated
- const date = new Date();
- fs.futimesSync(file, date, date);
-
- fs.closeSync(file);
- }
-
static async walk(pathLike: PathLike, callback?: FsWalkCallback): Promise {
let output: string[] = [];
@@ -409,23 +466,30 @@ export default class FsPoly {
return [];
}
- if (callback) {
- callback(entries.length);
- }
-
- // TODO(cemmer): `Promise.all()` this?
- for (const entry of entries) {
+ const entryIsDirectory = await Promise.all(entries.map(async (entry) => {
const fullPath = path.join(pathLike.toString(), entry.name);
- if (entry.isDirectory() || (entry.isSymbolicLink() && await this.isDirectory(fullPath))) {
- const subDirFiles = await this.walk(fullPath);
- output = [...output, ...subDirFiles];
- if (callback) {
- callback(subDirFiles.length - 1);
- }
- } else {
- output = [...output, fullPath];
+ return entry.isDirectory() || (entry.isSymbolicLink() && await this.isDirectory(fullPath));
+ }));
+
+ // Depth-first search directories first
+ const directories = entries
+ .filter((entry, idx) => entryIsDirectory[idx])
+ .map((entry) => path.join(pathLike.toString(), entry.name));
+ for (const directory of directories) {
+ const subDirFiles = await this.walk(directory);
+ if (callback) {
+ callback(subDirFiles.length);
}
+ output = [...output, ...subDirFiles];
+ }
+
+ const files = entries
+ .filter((entry, idx) => !entryIsDirectory[idx])
+ .map((entry) => path.join(pathLike.toString(), entry.name));
+ if (callback) {
+ callback(files.length);
}
+ output = [...output, ...files];
return output;
}
diff --git a/src/polyfill/timePoly.ts b/src/polyfill/timePoly.ts
new file mode 100644
index 000000000..2dc414fda
--- /dev/null
+++ b/src/polyfill/timePoly.ts
@@ -0,0 +1,14 @@
+export default {
+ hrtimeMillis(timeMillis?: number): number {
+ const hrtime = timeMillis !== undefined
+ ? [
+ // ms -> s
+ Math.floor(timeMillis / 1000),
+ // ms -> ns
+ (timeMillis - Math.floor(timeMillis / 1000) * 1000) * 1_000_000,
+ ] satisfies [number, number] : undefined;
+
+ const [sec, nano] = process.hrtime(hrtime);
+ return sec * 1000 + Math.floor(nano / 1_000_000);
+ },
+};
diff --git a/src/polyfill/urlPoly.ts b/src/polyfill/urlPoly.ts
index 1d31d803d..1c9c34086 100644
--- a/src/polyfill/urlPoly.ts
+++ b/src/polyfill/urlPoly.ts
@@ -4,9 +4,9 @@ export default {
*/
canParse(input: string, base?: string): boolean {
try {
- // eslint-disable-next-line no-new
- new URL(input, base);
- return true;
+ const url = new URL(input, base);
+ // Try to detect and ignore Windows drive letters
+ return process.platform !== 'win32' || url.protocol.length > 2;
} catch {
return false;
}
diff --git a/src/timer.ts b/src/timer.ts
index c7b9ebcc5..48b13a7e8 100644
--- a/src/timer.ts
+++ b/src/timer.ts
@@ -8,22 +8,31 @@ export default class Timer {
private readonly timeoutId: NodeJS.Timeout;
- private constructor(
+ private constructor(timeoutId: NodeJS.Timeout) {
+ this.timeoutId = timeoutId;
+ Timer.TIMERS.add(this);
+ }
+
+ static setTimeout(
runnable: (...args: unknown[]) => void,
timeoutMillis: number,
- ) {
- this.timeoutId = setTimeout(() => {
+ ): Timer {
+ const timer = new Timer(setTimeout(() => {
runnable();
- Timer.TIMERS.delete(this);
- }, timeoutMillis);
- Timer.TIMERS.add(this);
+ Timer.TIMERS.delete(timer);
+ }, timeoutMillis));
+ return timer;
}
- static setTimeout(
+ static setInterval(
runnable: (...args: unknown[]) => void,
timeoutMillis: number,
): Timer {
- return new Timer(runnable, timeoutMillis);
+ const timer = new Timer(setInterval(() => {
+ runnable();
+ Timer.TIMERS.delete(timer);
+ }, timeoutMillis));
+ return timer;
}
/**
diff --git a/src/types/cache.ts b/src/types/cache.ts
index dccc88f4e..56cf61f6f 100644
--- a/src/types/cache.ts
+++ b/src/types/cache.ts
@@ -1,22 +1,18 @@
import fs from 'node:fs';
import path from 'node:path';
-import util from 'node:util';
+import * as v8 from 'node:v8';
import * as zlib from 'node:zlib';
-import { Mutex } from 'async-mutex';
+import { E_CANCELED, Mutex } from 'async-mutex';
+import KeyedMutex from '../keyedMutex.js';
import FsPoly from '../polyfill/fsPoly.js';
import Timer from '../timer.js';
-interface CacheData {
- data: string,
-}
-
export interface CacheProps {
filePath?: string,
fileFlushMillis?: number,
saveOnExit?: boolean,
- maxSize?: number,
}
/**
@@ -25,17 +21,9 @@ export interface CacheProps {
export default class Cache {
private static readonly BUFFER_ENCODING: BufferEncoding = 'binary';
- private static readonly KEY_MUTEXES_MAX_COUNT = 1000;
-
- private keyOrder: Set = new Set();
-
private keyValues = new Map();
- private readonly keyMutexes = new Map();
-
- private keyMutexesLru: Set = new Set();
-
- private readonly keyMutexesMutex = new Mutex();
+ private readonly keyedMutex = new KeyedMutex(1000);
private hasChanged: boolean = false;
@@ -45,7 +33,7 @@ export default class Cache {
readonly fileFlushMillis?: number;
- readonly maxSize?: number;
+ private readonly saveMutex = new Mutex();
constructor(props?: CacheProps) {
this.filePath = props?.filePath;
@@ -56,14 +44,13 @@ export default class Cache {
await this.save();
});
}
- this.maxSize = props?.maxSize;
}
/**
* Return if a key exists in the cache, waiting for any existing operations to complete first.
*/
public async has(key: string): Promise {
- return this.lockKey(key, () => this.keyValues.has(key));
+ return this.keyedMutex.runExclusiveForKey(key, () => this.keyValues.has(key));
}
/**
@@ -84,7 +71,7 @@ export default class Cache {
* Get the value of a key in the cache, waiting for any existing operations to complete first.
*/
public async get(key: string): Promise {
- return this.lockKey(key, () => this.keyValues.get(key));
+ return this.keyedMutex.runExclusiveForKey(key, () => this.keyValues.get(key));
}
/**
@@ -96,7 +83,7 @@ export default class Cache {
runnable: (key: string) => V | Promise,
shouldRecompute?: (value: V) => boolean | Promise,
): Promise {
- return this.lockKey(key, async () => {
+ return this.keyedMutex.runExclusiveForKey(key, async () => {
if (this.keyValues.has(key)) {
const existingValue = this.keyValues.get(key) as V;
if (shouldRecompute === undefined || !await shouldRecompute(existingValue)) {
@@ -114,20 +101,14 @@ export default class Cache {
* Set the value of a key in the cache.
*/
public async set(key: string, val: V): Promise {
- return this.lockKey(key, () => this.setUnsafe(key, val));
+ return this.keyedMutex.runExclusiveForKey(key, () => this.setUnsafe(key, val));
}
private setUnsafe(key: string, val: V): void {
- if (this.maxSize !== undefined && !this.keyValues.has(key)) {
- this.keyOrder.add(key);
- }
+ const oldVal = this.keyValues.get(key);
this.keyValues.set(key, val);
- this.saveWithTimeout();
-
- // Evict old values (FIFO)
- if (this.maxSize !== undefined && this.keyValues.size > this.maxSize) {
- const staleKey = this.keyOrder.keys().next().value;
- this.deleteUnsafe(staleKey);
+ if (val !== oldVal) {
+ this.saveWithTimeout();
}
}
@@ -143,47 +124,16 @@ export default class Cache {
}
// Note: avoiding lockKey() because it could get expensive with many keys to delete
- await this.keyMutexesMutex.runExclusive(() => {
+ await this.keyedMutex.runExclusiveGlobally(() => {
keysToDelete.forEach((k) => this.deleteUnsafe(k));
});
}
private deleteUnsafe(key: string): void {
- this.keyOrder.delete(key);
this.keyValues.delete(key);
- this.keyMutexes.delete(key);
- this.keyMutexesLru.delete(key);
this.saveWithTimeout();
}
- private async lockKey(key: string, runnable: () => (R | Promise)): Promise {
- // Get a mutex for `key`
- const keyMutex = await this.keyMutexesMutex.runExclusive(() => {
- if (!this.keyMutexes.has(key)) {
- this.keyMutexes.set(key, new Mutex());
- this.keyMutexesLru = new Set([key, ...this.keyMutexesLru]);
-
- // Expire least recently used keys
- [...this.keyMutexesLru]
- .filter((lruKey) => !this.keyMutexes.get(lruKey)?.isLocked())
- .slice(Cache.KEY_MUTEXES_MAX_COUNT)
- .forEach((lruKey) => {
- this.keyMutexes.delete(lruKey);
- this.keyMutexesLru.delete(lruKey);
- });
- }
-
- // Mark this key as recently used
- this.keyMutexesLru.delete(key);
- this.keyMutexesLru = new Set([key, ...this.keyMutexesLru]);
-
- return this.keyMutexes.get(key) as Mutex;
- });
-
- // Only allow one concurrent fetch/compute for `key`
- return keyMutex.runExclusive(async () => runnable());
- }
-
/**
* Load the cache from a file.
*/
@@ -194,17 +144,15 @@ export default class Cache {
}
try {
- const cacheData = JSON.parse(
- await fs.promises.readFile(this.filePath, { encoding: Cache.BUFFER_ENCODING }),
- ) as CacheData;
- const compressed = Buffer.from(cacheData.data, Cache.BUFFER_ENCODING);
- const decompressed = await util.promisify(zlib.inflate)(compressed);
- const keyValuesObject = JSON.parse(decompressed.toString(Cache.BUFFER_ENCODING));
+ const compressed = await fs.promises.readFile(this.filePath);
+ if (compressed.length === 0) {
+ return this;
+ }
+ // NOTE(cemmer): util.promisify(zlib.inflate) seems to have issues not throwing correctly
+ const decompressed = zlib.inflateSync(compressed);
+ const keyValuesObject = v8.deserialize(decompressed);
const keyValuesEntries = Object.entries(keyValuesObject) as [string, V][];
this.keyValues = new Map(keyValuesEntries);
- if (this.maxSize !== undefined) {
- this.keyOrder = new Set(Object.keys(keyValuesObject));
- }
} catch { /* ignored */ }
return this;
@@ -226,37 +174,58 @@ export default class Cache {
* Save the cache to a file.
*/
public async save(): Promise {
- // Clear any existing timeout
- if (this.saveToFileTimeout !== undefined) {
- this.saveToFileTimeout.cancel();
- this.saveToFileTimeout = undefined;
- }
+ try {
+ await this.saveMutex.runExclusive(async () => {
+ // Clear any existing timeout
+ if (this.saveToFileTimeout !== undefined) {
+ this.saveToFileTimeout.cancel();
+ this.saveToFileTimeout = undefined;
+ }
- if (this.filePath === undefined || !this.hasChanged) {
- return;
- }
+ if (this.filePath === undefined || !this.hasChanged) {
+ return;
+ }
- const keyValuesObject = Object.fromEntries(this.keyValues);
- const decompressed = JSON.stringify(keyValuesObject);
- const compressed = await util.promisify(zlib.deflate)(decompressed);
- const cacheData = {
- data: compressed.toString(Cache.BUFFER_ENCODING),
- } satisfies CacheData;
-
- // Ensure the directory exists
- const dirPath = path.dirname(this.filePath);
- if (!await FsPoly.exists(dirPath)) {
- await FsPoly.mkdir(dirPath, { recursive: true });
- }
+ const keyValuesObject = Object.fromEntries(this.keyValues);
+ const decompressed = v8.serialize(keyValuesObject);
+ // NOTE(cemmer): util.promisify(zlib.deflate) seems to have issues not throwing correctly
+ const compressed = zlib.deflateSync(decompressed);
- // Write to a temp file first, then overwrite the old cache file
- const tempFile = await FsPoly.mktemp(this.filePath);
- await FsPoly.writeFile(
- tempFile,
- JSON.stringify(cacheData),
- { encoding: Cache.BUFFER_ENCODING },
- );
- await FsPoly.mv(tempFile, this.filePath);
- this.hasChanged = false;
+ // Ensure the directory exists
+ const dirPath = path.dirname(this.filePath);
+ if (!await FsPoly.exists(dirPath)) {
+ await FsPoly.mkdir(dirPath, { recursive: true });
+ }
+
+ // Write to a temp file first
+ const tempFile = await FsPoly.mktemp(this.filePath);
+ await FsPoly.writeFile(
+ tempFile,
+ compressed,
+ { encoding: Cache.BUFFER_ENCODING },
+ );
+
+ // Validate the file was written correctly
+ const tempFileCache = await new Cache({ filePath: tempFile }).load();
+ if (tempFileCache.size() !== Object.keys(keyValuesObject).length) {
+ // The written file is bad, don't use it
+ await FsPoly.rm(tempFile, { force: true });
+ return;
+ }
+
+ // Overwrite the real file with the temp file
+ try {
+ await FsPoly.mv(tempFile, this.filePath);
+ } catch {
+ return;
+ }
+ this.hasChanged = false;
+ this.saveMutex.cancel(); // cancel all waiting locks, we just saved
+ });
+ } catch (error) {
+ if (error !== E_CANCELED) {
+ throw error;
+ }
+ }
}
}
diff --git a/src/types/datStatus.ts b/src/types/datStatus.ts
index fcaa3c078..e29ee8679 100644
--- a/src/types/datStatus.ts
+++ b/src/types/datStatus.ts
@@ -144,9 +144,11 @@ export default class DATStatus {
}
private static append(map: Map, romType: ROMType, val: T): void {
- const arr = map.get(romType) ?? [];
- arr.push(val);
- map.set(romType, arr);
+ if (!map.has(romType)) {
+ map.set(romType, [val]);
+ } else {
+ map.get(romType)?.push(val);
+ }
}
getDATName(): string {
@@ -159,7 +161,7 @@ export default class DATStatus {
...this.incompleteRomTypesToReleaseCandidates.values(),
]
.flat()
- .filter(ArrayPoly.filterNotNullish)
+ .filter((releaseCandidate) => releaseCandidate !== undefined)
.flatMap((releaseCandidate) => releaseCandidate.getRomsWithFiles())
.map((romWithFiles) => romWithFiles.getInputFile());
}
@@ -170,8 +172,8 @@ export default class DATStatus {
anyGamesFound(options: Options): boolean {
return DATStatus.getAllowedTypes(options)
.reduce((result, romType) => {
- const foundReleaseCandidates = (
- this.foundRomTypesToReleaseCandidates.get(romType) ?? []).length;
+ const foundReleaseCandidates = this.foundRomTypesToReleaseCandidates
+ .get(romType)?.length ?? 0;
return result || foundReleaseCandidates > 0;
}, false);
}
@@ -361,7 +363,7 @@ export default class DATStatus {
): T[] {
return DATStatus.getAllowedTypes(options)
.flatMap((type) => romTypesToValues.get(type))
- .filter(ArrayPoly.filterNotNullish)
+ .filter((value) => value !== undefined)
.reduce(ArrayPoly.reduceUnique(), [])
.sort();
}
@@ -377,6 +379,6 @@ export default class DATStatus {
options.getOnlyRetail() || (!options.getOnlyBios() && !options.getOnlyDevice())
? ROMType.RETAIL : undefined,
ROMType.PATCHED,
- ].filter(ArrayPoly.filterNotNullish);
+ ].filter((romType) => romType !== undefined);
}
}
diff --git a/src/types/dats/cmpro/cmProParser.ts b/src/types/dats/cmpro/cmProParser.ts
index d36de7633..71bc86a85 100644
--- a/src/types/dats/cmpro/cmProParser.ts
+++ b/src/types/dats/cmpro/cmProParser.ts
@@ -35,26 +35,26 @@ export interface GameProps extends CMProObject {
sample?: SampleProps | SampleProps[],
// NON-STANDARD PROPERTIES
comment?: string,
- serial?: string,
- publisher?: string,
- releaseyear?: string,
- releasemonth?: string,
- developer?: string,
- users?: string,
- esrbrating?: string,
+ // serial?: string,
+ // publisher?: string,
+ // releaseyear?: string,
+ // releasemonth?: string,
+ // developer?: string,
+ // users?: string,
+ // esrbrating?: string,
genre?: string,
}
export interface ROMProps extends CMProObject {
name?: string,
- merge?: string,
+ // merge?: string,
size?: string,
crc?: string,
- flags?: string,
+ // flags?: string,
md5?: string,
sha1?: string,
// NON-STANDARD PROPERTIES
- serial?: string,
+ // serial?: string,
}
export interface DiskProps extends ROMProps {}
diff --git a/src/types/dats/dat.ts b/src/types/dats/dat.ts
index 9d1b4aeb7..342902cfa 100644
--- a/src/types/dats/dat.ts
+++ b/src/types/dats/dat.ts
@@ -107,33 +107,7 @@ export default abstract class DAT {
return FsPoly.makeLegal(filename.trim());
}
- /**
- * Does a DAT explicitly contain headered ROMs. It is possible for a DAT to be both non-headered
- * and non-headerless.
- */
- isHeadered(): boolean {
- // No-Intro "headerless" DATs have this field set
- if (this.getHeader().getClrMamePro()?.getHeader()) {
- return false;
- }
-
- return this.getName().match(/\(headered\)/i) !== null;
- }
-
- /**
- * Does a DAT explicitly contain headerless ROMs. It is possible for a DAT to be both non-headered
- * and non-headerless.
- */
- isHeaderless(): boolean {
- // No-Intro "headerless" DATs have this field set
- if (this.getHeader().getClrMamePro()?.getHeader()) {
- return true;
- }
-
- return this.getName().match(/\(headerless\)/i) !== null;
- }
-
- getRequiredChecksumBitmask(): number {
+ getRequiredRomChecksumBitmask(): number {
let checksumBitmask = 0;
this.getGames().forEach((game) => game.getRoms().forEach((rom) => {
if (rom.getCrc32() && rom.getSize()) {
@@ -149,6 +123,22 @@ export default abstract class DAT {
return checksumBitmask;
}
+ getRequiredDiskChecksumBitmask(): number {
+ let checksumBitmask = 0;
+ this.getGames().forEach((game) => game.getDisks().forEach((disk) => {
+ if (disk.getCrc32() && disk.getSize()) {
+ checksumBitmask |= ChecksumBitmask.CRC32;
+ } else if (disk.getMd5()) {
+ checksumBitmask |= ChecksumBitmask.MD5;
+ } else if (disk.getSha1()) {
+ checksumBitmask |= ChecksumBitmask.SHA1;
+ } else if (disk.getSha256()) {
+ checksumBitmask |= ChecksumBitmask.SHA256;
+ }
+ }));
+ return checksumBitmask;
+ }
+
/**
* Serialize this {@link DAT} to the file contents of an XML file.
*/
diff --git a/src/types/dats/disk.ts b/src/types/dats/disk.ts
index 55b530ee1..4cd60a95e 100644
--- a/src/types/dats/disk.ts
+++ b/src/types/dats/disk.ts
@@ -1,11 +1,10 @@
-import { Expose } from 'class-transformer';
+import ROM, { ROMProps } from './rom.js';
-interface DiskOptions {
- readonly name?: string;
- readonly sha1?: string;
- readonly md5?: string;
- readonly merge?: string;
- readonly status?: 'baddump' | 'nodump' | 'good' | 'verified';
+interface DiskProps extends Omit {
+ size?: number,
+ // region?: string,
+ // index?: number,
+ // writable?: 'yes' | 'no',
}
/**
@@ -13,27 +12,11 @@ interface DiskOptions {
* SHA1 do not both need to be specified in the data file:"
* @see http://www.logiqx.com/DatFAQs/CMPro.php
*/
-export default class Disk implements DiskOptions {
- @Expose({ name: 'name' })
- readonly name: string;
-
- @Expose({ name: 'sha1' })
- readonly sha1: string;
-
- @Expose({ name: 'md5' })
- readonly md5: string;
-
- @Expose({ name: 'merge' })
- readonly merge: string;
-
- @Expose({ name: 'status' })
- readonly status: 'baddump' | 'nodump' | 'good' | 'verified';
-
- constructor(options?: DiskOptions) {
- this.name = options?.name ?? '';
- this.sha1 = options?.sha1 ?? '';
- this.md5 = options?.md5 ?? '';
- this.merge = options?.merge ?? '';
- this.status = options?.status ?? 'good';
+export default class Disk extends ROM implements DiskProps {
+ constructor(props?: DiskProps) {
+ super(props ? {
+ ...props,
+ size: props?.size ?? 0,
+ } : undefined);
}
}
diff --git a/src/types/dats/game.ts b/src/types/dats/game.ts
index b2a65970f..33c6a8ec0 100644
--- a/src/types/dats/game.ts
+++ b/src/types/dats/game.ts
@@ -4,6 +4,7 @@ import { Expose, Transform, Type } from 'class-transformer';
import ArrayPoly from '../../polyfill/arrayPoly.js';
import Internationalization from '../internationalization.js';
+import Disk from './disk.js';
import Release from './release.js';
import ROM from './rom.js';
@@ -67,7 +68,7 @@ export interface GameProps {
// readonly manufacturer?: string,
readonly release?: Release | Release[],
readonly rom?: ROM | ROM[],
- // readonly disk?: Disk | Disk[],
+ readonly disk?: Disk | Disk[],
}
/**
@@ -124,6 +125,11 @@ export default class Game implements GameProps {
@Transform(({ value }) => value || [])
readonly rom?: ROM | ROM[];
+ @Expose()
+ @Type(() => Disk)
+ @Transform(({ value }) => value || [])
+ readonly disk?: Disk | Disk[];
+
constructor(props?: GameProps) {
this.name = props?.name ?? '';
this.category = props?.category ?? '';
@@ -134,8 +140,9 @@ export default class Game implements GameProps {
this.romOf = props?.romOf;
this.sampleOf = props?.sampleOf;
this.genre = props?.genre;
- this.release = props?.release ?? [];
- this.rom = props?.rom ?? [];
+ this.release = props?.release;
+ this.rom = props?.rom;
+ this.disk = props?.disk;
}
/**
@@ -212,6 +219,15 @@ export default class Game implements GameProps {
return [];
}
+ getDisks(): Disk[] {
+ if (Array.isArray(this.disk)) {
+ return this.disk;
+ } if (this.disk) {
+ return [this.disk];
+ }
+ return [];
+ }
+
// Computed getters
getRevision(): number {
@@ -578,7 +594,7 @@ export default class Game implements GameProps {
const releaseLanguages = this.getReleases()
.map((release) => release.getLanguage())
- .filter(ArrayPoly.filterNotNullish);
+ .filter((language) => language !== undefined);
if (releaseLanguages.length > 0) {
return releaseLanguages;
}
@@ -615,7 +631,7 @@ export default class Game implements GameProps {
.map((lang) => lang.toUpperCase())
.map((lang) => Internationalization.LANGUAGE_OPTIONS
.find((langOpt) => langOpt.long?.toUpperCase() === lang.toUpperCase())?.short)
- .filter(ArrayPoly.filterNotNullish)
+ .filter((lang) => lang !== undefined)
.filter((lang) => Internationalization.LANGUAGES.includes(lang)) // is known
.reduce(ArrayPoly.reduceUnique(), []);
if (threeMatchesParsed.length > 0) {
@@ -637,7 +653,7 @@ export default class Game implements GameProps {
}
return undefined;
})
- .filter(ArrayPoly.filterNotNullish);
+ .filter((language) => language !== undefined);
}
// Immutable setters
diff --git a/src/types/files/archives/archive.ts b/src/types/files/archives/archive.ts
index 8eece28b7..cc1c3f95c 100644
--- a/src/types/files/archives/archive.ts
+++ b/src/types/files/archives/archive.ts
@@ -34,9 +34,14 @@ export default abstract class Archive {
): Promise {
const tempFile = await fsPoly.mktemp(path.join(
Temp.getTempDir(),
- path.basename(entryPath),
+ fsPoly.makeLegal(path.basename(entryPath) || path.parse(this.getFilePath()).name),
));
+ const tempDir = path.dirname(tempFile);
+ if (!await fsPoly.exists(tempDir)) {
+ await fsPoly.mkdir(tempDir, { recursive: true });
+ }
+
try {
await this.extractEntryToFile(entryPath, tempFile);
return await callback(tempFile);
diff --git a/src/types/files/archives/archiveEntry.ts b/src/types/files/archives/archiveEntry.ts
index e3439c59a..845bb72be 100644
--- a/src/types/files/archives/archiveEntry.ts
+++ b/src/types/files/archives/archiveEntry.ts
@@ -31,12 +31,14 @@ export default class ArchiveEntry extends File implements Arc
filePath: archiveEntryProps.archive.getFilePath(),
});
this.archive = archiveEntryProps.archive;
- this.entryPath = path.normalize(archiveEntryProps.entryPath);
+ this.entryPath = archiveEntryProps.entryPath
+ ? path.normalize(archiveEntryProps.entryPath)
+ : archiveEntryProps.entryPath;
}
static async entryOf(
archiveEntryProps: ArchiveEntryProps,
- checksumBitmask: number = ChecksumBitmask.CRC32,
+ checksumBitmask: number = ChecksumBitmask.NONE,
): Promise> {
let finalSize = archiveEntryProps.size;
let finalCrcWithHeader = archiveEntryProps.crc32;
@@ -151,7 +153,8 @@ export default class ArchiveEntry extends File implements Arc
}
getExtractedFilePath(): string {
- return this.entryPath;
+ // Note: {@link Chd} will stuff some extra metadata in the entry path, chop it out
+ return this.entryPath.split('|')[0];
}
getEntryPath(): string {
@@ -219,6 +222,9 @@ export default class ArchiveEntry extends File implements Arc
}
withFilePath(filePath: string): ArchiveEntry {
+ if (this.getArchive().getFilePath() === filePath) {
+ return this;
+ }
return new ArchiveEntry({
...this,
archive: this.getArchive().withFilePath(filePath),
@@ -271,9 +277,9 @@ export default class ArchiveEntry extends File implements Arc
toString(): string {
if (this.getSymlinkSource()) {
- return `${this.getFilePath()}|${this.getEntryPath()} -> ${this.getSymlinkSource()}|${this.getEntryPath()}`;
+ return `${this.getFilePath()}|${this.getExtractedFilePath()} -> ${this.getSymlinkSource()}|${this.getExtractedFilePath()}`;
}
- return `${this.getFilePath()}|${this.getEntryPath()}`;
+ return `${this.getFilePath()}|${this.getExtractedFilePath()}`;
}
equals(other: File): boolean {
diff --git a/src/types/files/archives/chd/chd.ts b/src/types/files/archives/chd/chd.ts
new file mode 100644
index 000000000..2de8ac00b
--- /dev/null
+++ b/src/types/files/archives/chd/chd.ts
@@ -0,0 +1,208 @@
+import fs from 'node:fs';
+import path from 'node:path';
+import { Readable } from 'node:stream';
+import util from 'node:util';
+
+import { Mutex } from 'async-mutex';
+import chdman, { CHDInfo, CHDType } from 'chdman';
+import { Memoize } from 'typescript-memoize';
+
+import Temp from '../../../../globals/temp.js';
+import FsPoly from '../../../../polyfill/fsPoly.js';
+import ExpectedError from '../../../expectedError.js';
+import File from '../../file.js';
+import { ChecksumBitmask } from '../../fileChecksums.js';
+import Archive from '../archive.js';
+import ArchiveEntry from '../archiveEntry.js';
+import ChdBinCueParser from './chdBinCueParser.js';
+import ChdGdiParser from './chdGdiParser.js';
+
+export default class Chd extends Archive {
+ private tempSingletonHandles = 0;
+
+ private readonly tempSingletonMutex = new Mutex();
+
+ private tempSingletonDirPath?: string;
+
+ private tempSingletonFilePath?: string;
+
+ // eslint-disable-next-line class-methods-use-this
+ protected new(filePath: string): Archive {
+ return new Chd(filePath);
+ }
+
+ static getExtensions(): string[] {
+ return ['.chd'];
+ }
+
+ // eslint-disable-next-line class-methods-use-this
+ getExtension(): string {
+ return Chd.getExtensions()[0];
+ }
+
+ async getArchiveEntries(checksumBitmask: number): Promise[]> {
+ const info = await this.getInfo();
+
+ if (checksumBitmask === ChecksumBitmask.NONE) {
+ // Doing a quick scan
+ return this.getArchiveEntriesSingleFile(info, checksumBitmask);
+ }
+
+ if (info.type === CHDType.CD_ROM) {
+ return ChdBinCueParser.getArchiveEntriesBinCue(this, checksumBitmask);
+ } if (info.type === CHDType.GD_ROM) {
+ // TODO(cemmer): allow parsing GD-ROM to bin/cue https://github.com/mamedev/mame/issues/11903
+ return ChdGdiParser.getArchiveEntriesGdRom(this, checksumBitmask);
+ }
+
+ return this.getArchiveEntriesSingleFile(info, checksumBitmask);
+ }
+
+ private async getArchiveEntriesSingleFile(
+ info: CHDInfo,
+ checksumBitmask: number,
+ ): Promise[]> {
+ // MAME DAT s use the data+metadata SHA1 (vs. just the data SHA1)
+ const rawEntry = await ArchiveEntry.entryOf({
+ archive: this,
+ entryPath: '',
+ sha1: info.sha1,
+ // There isn't a way for us to calculate these other checksums, so fill it in with garbage
+ size: 0,
+ crc32: checksumBitmask & ChecksumBitmask.CRC32 ? 'x'.repeat(8) : undefined,
+ md5: checksumBitmask & ChecksumBitmask.MD5 ? 'x'.repeat(32) : undefined,
+ sha256: checksumBitmask & ChecksumBitmask.SHA256 ? 'x'.repeat(64) : undefined,
+ }, checksumBitmask);
+
+ const extractedEntry = await ArchiveEntry.entryOf({
+ archive: this,
+ entryPath: '',
+ size: info.logicalSize,
+ /**
+ * NOTE(cemmer): the "data SHA1" equals the original input file in these tested cases:
+ * - PSP .iso -> .chd with createdvd (and NOT createcd)
+ */
+ sha1: info.dataSha1,
+ }, checksumBitmask);
+
+ return [rawEntry, extractedEntry];
+ }
+
+ async extractEntryToFile(
+ entryPath: string,
+ extractedFilePath: string,
+ ): Promise {
+ return this.extractEntryToStreamCached(
+ entryPath,
+ async (stream) => new Promise((resolve, reject) => {
+ const writeStream = fs.createWriteStream(extractedFilePath);
+ writeStream.on('close', resolve);
+ writeStream.on('error', reject);
+ stream.pipe(writeStream);
+ }),
+ );
+ }
+
+ private async extractEntryToStreamCached(
+ entryPath: string,
+ callback: (stream: Readable) => (Promise
+
-With `igir` you can manage a ROM collection of any size:
+With Igir you can manage a ROM collection of any size:
-- ๐ Scan for DATs, ROMs, and ROM patches - including those in archives (see [scanning](https://igir.io/input/file-scanning) & [archive docs](https://igir.io/input/reading-archives))
+- ๐ Scan for DATs, ROMs, and ROM patchesโincluding those in archives (see [scanning](https://igir.io/input/file-scanning) & [archive docs](https://igir.io/input/reading-archives))
- ๐ Organize ROM files by console (see [DAT docs](https://igir.io/dats/overview))
- ๐ช Name ROM files consistently, including the right extension (see [DAT docs](https://igir.io/dats/overview))
- โ๏ธ Filter out duplicate ROMs, or ROMs in languages you don't understand (see [filtering docs](https://igir.io/roms/filtering-preferences))
- ๐๏ธ Extract or archive ROMs in mass (see [archive docs](https://igir.io/output/writing-archives))
- ๐ฉน Patch ROMs automatically in mass (see [scanning](https://igir.io/input/file-scanning) & [patching docs](https://igir.io/roms/patching))
-- ๐ฉ Parse ROMs with headers, and optionally remove them (see [header docs](https://igir.io/roms/headers))
+- ๐ฉ Parse ROMs with headers and optionally remove them (see [header docs](https://igir.io/roms/headers))
- โ๏ธ Build & re-build (un-merge, split, or merge) MAME ROM sets (see [arcade docs](https://igir.io/usage/arcade))
-- ๐ฎ Report on what ROMs are present or missing for each console, and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/dats/overview))
+- ๐ฎ Report on what ROMs are present or missing for each console and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/dats/overview))
-## How do I run `igir`?
+## How do I run Igir?
Either download the latest version for your OS from the [releases page](https://github.com/emmercm/igir/releases/latest), or if you have Node.js installed you can use [`npx`](https://docs.npmjs.com/cli/v9/commands/npx) to always run the latest version from the command line:
diff --git a/codecov.yml b/codecov.yml
index 3b7f31352..9f77df147 100644
--- a/codecov.yml
+++ b/codecov.yml
@@ -9,5 +9,5 @@ coverage:
patch: off
project:
default:
- target: 94%
+ target: 93%
threshold: 1%
diff --git a/docs/advanced/internals.md b/docs/advanced/internals.md
index ce366f5ad..e7875180f 100644
--- a/docs/advanced/internals.md
+++ b/docs/advanced/internals.md
@@ -1,10 +1,10 @@
# Internal Operations
-Information about the inner workings of `igir`.
+Information about the inner workings of Igir.
## Order of operations
-`igir` runs these steps in the following order:
+Igir runs these steps in the following order:
1. Scan each DAT input path for every file and parse them, if provided (`--dat