zk
e363bd2532
* docs: migrate from mkdocs to mdbook - Rename old mkdocs docs/ to old-docs/ for reference - Set up new docs/ with mdbook (book.toml + src/ tree) - Mirror full nav structure from mkdocs.yml into SUMMARY.md - Add Justfile-docs with docs-serve, docs-build, docs-check, docs-install recipes - Import Justfile-docs from root Justfile - Rewrite .github/workflows/docs.yml: build step uses taiki-e/install-action to install mdbook, split into separate build + deploy jobs, PR builds run the check step too * ci(docs): pin actions to SHAs, install mdbook via release binary * ci(docs): install mdbook from apt instead of curling a release binary * dev: replace mkdocs python stack with mdbook in dev image * ci(docs): apt only works on Debian; restore release binary install for Ubuntu CI * docs: fix duplicate file entries in SUMMARY.md * docs: remove docs-install recipe * docs: remove site-url from book.toml to fix asset loading * dev: install mdbook from upstream release binary, not Debian apt The Debian package (0.4.x+ds) strips bundled font assets, leaving the built site without fonts/fonts.css. Use the upstream tarball (same as CI) so the theme is complete. Handles x86_64 and aarch64. * docs: vendor mdbook tarballs in dev/mdbook/, install from there Dockerfile.dev uses COPY + tar to install the right arch at build time. CI extracts the x86_64 tarball directly from the checkout. No network access required for either — and no stripped-fonts Debian package. * fix: unwritten
3.1 KiB
Importing DryScores
This page goes over hydrating a DryScore into a fully fledged Tachi score, and then goes over how it is imported.
!!! warning Performance has been squeezed out of this process rather aggressively, as such, some of the more 'obvious' ways to do things have been ignored for performance gains.
!!! info
The code that handles this logic is found in
src/lib/score-import/framework/score-importing/score-importing.ts.
Return Format
The ImportIterableDatapoint returns
ImportProcessingInfo or null. The format for the former
can be found in tachi-common, and should be read before
following this document!
As a rough outline:
{
// whether this import worked or not
success: boolean;
// ScoreImported implies success: true, and all others imply success: false.
type: "ScoreImported" | "SongOrChartNotFound" // ... and more;
// An error message.
message: String | null;
// Some errors return some data about the error, or things like scoreImported returns the score that was imported.
content: see_implementation
}
Dealing with converter returns
As mentioned in Parsers and Converters, converters will return a DryScore, and its matching chart and song on success.
However, failures are also an expected throw from a
converter function. We handle these throws by logging
them dependent on their severity, and returning
that this score has failed to be imported properly
in the ImportProcessingInfo format.
If the converter was successful, we now have a DryScore, song and chart to work with.
Hydration
Our first step is to turn that DryScore into a 'real score'.
!!! info
The code for this is found in HydrateScore.
Before anything, we calculated the ScoreID for this score. This is used to dedupe scores, and is a checksum of the following properties.
- userID
- chartID
- lamp
- grade
- score
- percent
We fill out all the properties that can be calculated
about the score, such as the aptly named calculatedData,
and things like gradeIndex and lampIndex, which are
just the index of the grade/lamp string in the set of grade/lamps.
Other properties, such as timeAdded (the time this data was inserted into the database) can be trivially attached onto the score.
Insertion
Now that we have a real Tachi score, we can insert it into the database.
The obvious solution here is something like:
await db.scores.insert(scoreDocument);
But this has performance implications for large sets of scores.
To solve these performance issues, we use a queue to insert scores.
ScoreDocuments are appended to a queue of 500. If the queue hits 501 scores, the queue is bulk-written to the database. At the end of hydrating and queueing all scores, the queue is flushed one last time.
This simple optimisation provides large performance benefits.
Final Returns
Importing each datapoint gives us an ImportProcessingInfo
object OR null. We want to return only the former,
as null is for skipped scores.
We filter out all the nulls and flush the queue before
returning the array of ImportProcessingInfo objects
that were returned from our import process.