mirror of
https://github.com/zkldi/Tachi.git
synced 2026-10-04 21:08:09 +03:00
feat: first overhaul
This commit is contained in:
zkldi
@@ -0,0 +1,22 @@
|
||||
# Direct Manual
|
||||
|
||||
Direct Manual is a way of POSTing BATCH-MANUAL content straight to the server.
|
||||
|
||||
In short, this lets you implement your own IRs, with absolutely no input
|
||||
from me.
|
||||
|
||||
## Flow
|
||||
|
||||
You need to send a POST request to `/ir/direct-manual/import`, with the body
|
||||
of the request being a batch manual document.
|
||||
|
||||
You can set the `X-User-Intent` Header to true if the import was done with
|
||||
user-intent (i.e. they clicked a button to fire this request, rather than it
|
||||
being automated in the background).
|
||||
|
||||
!!! warning
|
||||
The response from `/ir/direct-manual/import` *MAY* be deferred if the `tachi-server` instance uses score import workers. If this happens **202** will be returned as the status code. You **MUST** then poll the import status to find out what happened to it (such as it failing.)
|
||||
|
||||
## That's it!
|
||||
|
||||
Seriously -- that's it.
|
||||
@@ -0,0 +1,186 @@
|
||||
# What is BATCH-MANUAL?
|
||||
|
||||
BATCH-MANUAL is a JSON format that Tachi accepts.
|
||||
This format can be submitted as [a file](../../api/routes/import.md#import-scores-from-a-file)
|
||||
using the `file/batch-manual` [Import Type](../import/import-types.md), or it can be submitted as a
|
||||
[HTTP request body](./direct-manual.md).
|
||||
|
||||
*****
|
||||
|
||||
## Motivation
|
||||
|
||||
Instead of Tachi writing new support for every kind of
|
||||
possible export, and bothering other service providers
|
||||
to write exports, we could write a generic format we accept
|
||||
and then users with a bit of scripting knowledge can import
|
||||
their own scores.
|
||||
|
||||
This has the additional advantage of allowing extremely
|
||||
obscure imports, and reduces the workload on Tachi.
|
||||
|
||||
## Format
|
||||
|
||||
The format is incredibly simple JSON.
|
||||
|
||||
It is comprised of two base keys, `meta` and `scores`.
|
||||
|
||||
!!! note
|
||||
These keys were originally called `head` and `body` in Kamaitachi. You will have to update
|
||||
existing batch-manual code.
|
||||
|
||||
### Meta
|
||||
|
||||
The `meta` key contains metadata, and looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"game": "iidx",
|
||||
"playtype": "SP",
|
||||
"service": "foobar"
|
||||
}
|
||||
```
|
||||
|
||||
The fields have the following values:
|
||||
|
||||
| Property | Type | Description |
|
||||
| :: | :: | :: |
|
||||
| `game` | Any Game Identifier | The game this import is for. |
|
||||
| `playtype` | Any Playtype for the above game. | The playtype this import is for. |
|
||||
| `service` | String | A humanised string to explain where these scores are from. This must be between 2 and 15 characters. |
|
||||
| `version` (Optional) | String | Optionally, you can specify a version of the game this import is for. This should be used when conflicting versions of songs exist, or when this import is rather old. Most of the time, this does not need to be present. |
|
||||
|
||||
### Scores
|
||||
|
||||
The `scores` property is an array of Batch Manual Scores. An example
|
||||
score is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"score": 500,
|
||||
"lamp": "HARD CLEAR",
|
||||
"matchType": "songTitle",
|
||||
"identifier": "5.1.1.",
|
||||
"difficulty": "ANOTHER",
|
||||
"timeAchieved": 1624324467489
|
||||
}
|
||||
```
|
||||
|
||||
The properties are described as this:
|
||||
|
||||
| Property | Type | Description |
|
||||
| :: | :: | :: |
|
||||
| `score` | Number | The score for this, well, score. This should use the default scoring algorithm for this game. |
|
||||
| `lamp` | Lamp | The lamp for this score. This should be one of the lamps as described in the config for your game + playtype. |
|
||||
| `percent` (Conditional) | Number | Only appears for `jubeat`. This should be set to the Percent for this score. In jubeat's case, this is your Music Rate. |
|
||||
| `matchType` | "songTitle" \| "ddrSongHash" \| "tachiSongID" \| "bmsChartHash" \| "inGameID" \| "uscChartHash" | This determines how `identifier` will be used to match your scores' chart with Tachi's database of songs and charts. |
|
||||
| `identifier` | String | A string that Tachi uses to identify what chart this is for. How this is used depends on the `matchType`. |
|
||||
| `difficulty` (Conditional) | String | If `matchType` is "tachiSongID", "inGameID", "ddrSongHash" or "songTitle", this field must be present, and describe the difficulty of the chart this score is for. |
|
||||
| `timeAchieved` (Optional) | integer \| null | This is *when* the score was achieved in unix milliseconds. This should be provided if possible, as Tachi uses it for a LOT of features. |
|
||||
| `comment` (Optional) | string \| null | A comment from the user about this score. |
|
||||
| `judgements` (Optional) | Record<Game Judgement, integer> | This should be a record of the judgements for your game + playtype, and the integer indicating how often they occured. |
|
||||
| `hitMeta` (Optional) | See [Game Specific Hit Meta](../../schemas/score.md#game-specific) | This can be a partial record of various `hitMeta` props for this game. |
|
||||
| `scoreMeta` (Optional) | See [Game Specific Score Meta](../../schemas/score.md#game-specific) | This can be a partial record of various `scoreMeta` props for this game. |
|
||||
|
||||
!!! warning
|
||||
`identifier` should always be a string. Even if it's something like a numeric ID! Tachi will handle this.
|
||||
|
||||
!!! warning
|
||||
`timeAchieved` is in **UNIX MILLISECONDS**. Most programming languages use unix seconds. You might have to
|
||||
multiply your timestamps by 1000.
|
||||
|
||||
#### Match Type
|
||||
|
||||
There are many match types, and they all use identifier
|
||||
in a different way.
|
||||
|
||||
- songTitle
|
||||
|
||||
As the name implies, this searches for a song who's title
|
||||
resembles `identifier`. **THIS IS NOT FUZZY MATCHING**,
|
||||
and is by far the least reliable way to send scores to
|
||||
Tachi. This is kept for compatibility purposes with poor quality APIs.
|
||||
|
||||
This match type *necessitates* that `difficulty` be defined
|
||||
and set to a valid difficulty for this game + playtype.
|
||||
|
||||
- tachiSongID
|
||||
|
||||
This uses `identifier` as if it were an integer, to match
|
||||
songs based on the `id` field of a tachi song.
|
||||
|
||||
This match type *necessitates* that `difficulty` be defined
|
||||
and set to a valid difficulty for this game + playtype.
|
||||
|
||||
- bmsChartHash
|
||||
|
||||
As the name implies, this looks for the chart hash BMS
|
||||
uses. This can be either the MD5 hash or the SHA256 hash,
|
||||
both will match.
|
||||
|
||||
This match type can only be used for BMS.
|
||||
|
||||
- uscChartHash
|
||||
|
||||
This looks for the chart SHA1 that USC uses. As expected, this
|
||||
can only be used for USC.
|
||||
|
||||
- inGameID
|
||||
|
||||
This uses the in-game-ID for this **SONG**. You, therefore,
|
||||
**MUST** specify the difficulty for this chart aswell.
|
||||
|
||||
This is supported for the following games:
|
||||
|
||||
- IIDX
|
||||
- Pop'n Music
|
||||
- Jubeat
|
||||
- CHUNITHM
|
||||
- GITADORA
|
||||
- maimai
|
||||
- MUSECA
|
||||
|
||||
- sdvxInGameID
|
||||
|
||||
This uses the in-game-ID for this SDVX song. You must specify
|
||||
the difficulty for this chart aswell.
|
||||
|
||||
The reason SDVX gets its own special `matchType` is because this
|
||||
matchType supports `difficulty: "ANY_INF"`. This special difficulty
|
||||
means that it will check for any of `INF/GRV/HVN/VVD/XCD` for this song.
|
||||
|
||||
This is useful for services that store all of those as the same difficulty.
|
||||
|
||||
## Example
|
||||
|
||||
A final example of a simple BATCH MANUAL format
|
||||
looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"meta": {
|
||||
"game": "iidx",
|
||||
"playtype": "SP",
|
||||
"service": "My Service"
|
||||
},
|
||||
"scores": [{
|
||||
"score": 500,
|
||||
"lamp": "HARD CLEAR",
|
||||
"matchType": "songTitle",
|
||||
"identifier": "5.1.1.",
|
||||
"difficulty": "ANOTHER"
|
||||
}, {
|
||||
"score": 123,
|
||||
"lamp": "FAILED",
|
||||
"matchType": "tachiSongID",
|
||||
"identifier": "1",
|
||||
"difficulty": "HYPER",
|
||||
"comment": "This score sucked!",
|
||||
"hitMeta": {
|
||||
"bp": 5
|
||||
},
|
||||
"scoreMeta": {
|
||||
"random": "MIRROR"
|
||||
}
|
||||
}]
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,8 @@
|
||||
# About
|
||||
|
||||
As you'd expect, this section contains implementation details. Although the entire
|
||||
codebase reference is technically about implementation details, this part is
|
||||
for miscellaneous implementation details that wouldn't fit anywhere else.
|
||||
|
||||
*****
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# Game Configuration
|
||||
|
||||
Tachi has two sets of game configurations.
|
||||
|
||||
The first is on the game level, which contains things
|
||||
like the humanised name for the game (i.e. `iidx -> beatmania IIDX`).
|
||||
|
||||
The second is for each individual game + playtype
|
||||
combination, which contains things like the list of
|
||||
lamps for the game.
|
||||
|
||||
!!! help
|
||||
This page is unfinished.
|
||||
|
||||
It would be really appreciated if someone formatted
|
||||
the configurations for every game! For the time being,
|
||||
the below documentation is just the raw configuration
|
||||
for each game. Please see [Contributing to Tachi](../../contributing).
|
||||
|
||||
*****
|
||||
|
||||
This documentation is unfinished. It is probably easier for you to [just read the config.ts file](https://github.com/TNG-dev/Tachi/tree/staging/common/src/config/config.ts)
|
||||
@@ -0,0 +1,18 @@
|
||||
# Goal ID implementation
|
||||
|
||||
Goal IDs exist to dedupe goals when a user creates
|
||||
a new goal. For example, if a user wants to create a
|
||||
HARD CLEAR Mei goal, but one already exists, we should
|
||||
not insert two [Goal Documents](../../schemas/goal.md)
|
||||
representing the same thing.
|
||||
|
||||
*****
|
||||
|
||||
## Hashing
|
||||
|
||||
To create the Goal ID, we perform a [json stable hash](https://github.com/zkldi/fast-json-stable-hash) on the
|
||||
`game`, `playtype`, `charts` and `criteria` of this field.
|
||||
|
||||
This enforces that goals are unique on those fields.
|
||||
|
||||
The above hash is then returned, prefixed with `G`.
|
||||
@@ -0,0 +1,115 @@
|
||||
# Goals, Quests, Questlines
|
||||
|
||||
This page covers pretty much everything you need to know about interacting with goals
|
||||
and quests.
|
||||
|
||||
Although these features are fairly straightforward on the surface, their implementation is a bit complex at points. We'll get to it.
|
||||
|
||||
## What is a Goal?
|
||||
|
||||
A goal is a set of instructions that are evaluated for a player if they are subscribed to it (and it is relevant to their current import, see [Getting Relevant Goals](../import/goals.md)).
|
||||
|
||||
!!! example
|
||||
An example goal would be something like "AAA 100 charts in the X Folder".
|
||||
|
||||
Users must subscribe to goals for them to be evaluated when they import scores. This is
|
||||
abstracted away from the user, as the operation to create a goal will automatically subscribe
|
||||
them to the same goal.
|
||||
|
||||
### Identification
|
||||
|
||||
Goals are identified by a checksum of their instructions. If two users make the goal to "AAA Freedom Dive", the goalID will be the same.
|
||||
|
||||
This allows us to do things like see what goals are popular, and reduce the general duplication on the database.
|
||||
|
||||
## User Subscriptions
|
||||
|
||||
Users may create and subscribe to goals. This is done with the [UGPT Create Goals Endpoint](TODO). If they create a goal that already exists, they are just subscribed to that goal.
|
||||
|
||||
If they create a goal that does not already exist, it is created, and they are subscribed.
|
||||
|
||||
Users may unsubscribe from goals that they no longer care about getting pinged for.
|
||||
|
||||
Subscriptions to goals are stored in a [GoalSubscriptionDocument](../../schemas/goal-sub.md).
|
||||
This document is uniquely identified by the joining of the `goalID` with the `userID`.
|
||||
|
||||
### Instant Direct Achievements
|
||||
|
||||
If a user creates/subscribes to a goal that they would instantly achieve, they are denied
|
||||
from adding goal directly.
|
||||
|
||||
### Evaluating A User's Progress
|
||||
|
||||
Goals have two main components in terms of evaluation -- `progress` and `outOf`. If the progress is greater than or equal to the `outOf`, the goal is marked as achieved.
|
||||
|
||||
Both of these values are numbers, and are not intended for human consumption. For example, the goal 'HARD CLEAR Freedom Dive' would be represented by an `outOf` of `6`, which is the internal lampIndex for a BMS HARD CLEAR.
|
||||
|
||||
For human representation, these values are formatted and stored in the Goal Subscription Document as `progressHuman` and `outOfHuman`, respectively. These values are prettified for human users, and leverage game-specific formatting in some cases.
|
||||
|
||||
!!! note
|
||||
Achieved goals are still calculated -- a user may actually find themselves un-achieving goals in some stranger cases.
|
||||
|
||||
If their goal is something like AAA 10% of a folder, and the folder gets larger, they
|
||||
may find themselves losing the achieved status on this goal.
|
||||
|
||||
!!! warning
|
||||
Goal Subscriptions are capped by the MAX_GOAL_SUBSCRIPTIONS [conf.json5](../setup/config.md) property, and defaults to 1000.
|
||||
|
||||
!!! danger
|
||||
Unsubscribing from goals directly is only possible if they do not have any parent quests.
|
||||
More on that below.
|
||||
|
||||
## What is a Quest?
|
||||
|
||||
Quests are groups of goals, defined in a structured form, with support for assigning
|
||||
notes next to goalIDs, and other generally useful stuff.
|
||||
|
||||
Their purpose for users is to give them a quick way to assign a bunch of goals they might care about.
|
||||
|
||||
For example, a dedicated user may make their own "Kaiden Checklist", and create a set of goals
|
||||
they think are useful for kaidens to aim for. Another user interested in this can just
|
||||
subscribe to that quest, and all the goals will be managed for them.
|
||||
|
||||
### Identification
|
||||
|
||||
Quests are identified by their `questID`, which is a completely arbitrary string.
|
||||
|
||||
At the moment, users cannot create quests directly, they are hard-defined by the [Database Seeds](../infrastructure/database-seeds.md).
|
||||
|
||||
### Evaluating a User's Progress
|
||||
|
||||
Similarly to goals, user's can subscribe to quests.
|
||||
|
||||
Quest subscriptions are interesting in that the `progress` factor is always an integer -- how many goals in the quest the user has achieved, and the `outOf` factor is always an integer -- how many goals need to be achieved in the quest for the quest to be marked as achieved.
|
||||
|
||||
Similarly, the `achieved` status on quests is identical to `progress` being greater than or equal to `outOf`.
|
||||
|
||||
### Parenting Goal Subscriptions
|
||||
|
||||
When a quest is subscribed to, all the goals in the quest are also subscribed to.
|
||||
|
||||
When a user is subscribed to a quest, they *must* also be subscribed to all the goals in
|
||||
that quest. If they aren't, they've desynced with the quest, and that's an awful
|
||||
user experience.
|
||||
|
||||
As such, Tachi keeps track of all the quests that care about this goal subscription, and users are *prevented from unsubscribing from this goal* while any of their quest subscriptions parent the goal.
|
||||
|
||||
### Instant Indirect Achievements
|
||||
|
||||
Subscribing to a quest *mandates* that all of the goals are assigned. It is not rare for this to mean that you subscribe to goals that are instantly achieved.
|
||||
|
||||
This is fine. If this does happen though, `wasInstantlyAchieved` is set on the goal, and no
|
||||
webhook event is emitted.
|
||||
|
||||
### Unsubscribing
|
||||
|
||||
For all `goalID`s in the quest, we check if this goal subscription has any other parent quests.
|
||||
|
||||
If this quest was the only reason the goal was assigned (i.e. it wasn't assigned directly, or isn't part of another quest), this quest will be unsubscribed from.
|
||||
|
||||
## Questlines
|
||||
|
||||
Questline are ordered lists of quests. Their purpose is to group quests together
|
||||
visually in an ordered manner.
|
||||
|
||||
Users *can not* "subscribe" to questlines, but they can use questlines as a utility for subscribing to all the related quests.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Score ID implementation
|
||||
|
||||
Score IDs exist to dedupe scores when a user re-submits
|
||||
the same scores. This happens frequently with `file/`
|
||||
and `api/` [Import Types](../import/import-types.md),
|
||||
as they typically resubmit the same scores.
|
||||
|
||||
If a user only got a score once, we don't want to store
|
||||
it twice.
|
||||
|
||||
Sadly, we can't depend on things like timestamps to assert
|
||||
whether or whether not we've saw a score before. Many services
|
||||
alter/tamper their timestamps such that they're unreliable.
|
||||
|
||||
!!! example
|
||||
E-Amusement IIDX CSVs will change the timestamp of every single
|
||||
score when a new version comes out to the second the user created
|
||||
their new account.
|
||||
|
||||
*****
|
||||
|
||||
## Hashing
|
||||
|
||||
The score ID is created by joining the following properties:
|
||||
|
||||
- The `userID` that got this score
|
||||
- The `chartID` that this score was on
|
||||
- All [Provided Metrics](todo) for this GPT
|
||||
- Any [Optional Metrics](todo) that have been marked as `partOfScoreID`
|
||||
|
||||
and then hashing them with SHA256.
|
||||
|
||||
This is then prefixed with `T`, and returned.
|
||||
|
||||
## Clobbering
|
||||
|
||||
Since a scoreID isn't necessarily all of the possible statistics for a score, it's
|
||||
possible for users to "clobber" their scores, by importing a score without as many
|
||||
pieces of info (i.e. no judgements), then trying to import that same score with
|
||||
judgements later will not work, as the scoreID sees it as a duplicate.
|
||||
|
||||
See [Clobbering](todo).
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# Search Implementation
|
||||
|
||||
Tachi's search implementation uses MongoDB's $text index. This breaks a query into words
|
||||
and compares each of them to the provided text fields.
|
||||
|
||||
*****
|
||||
|
||||
## __textScore
|
||||
|
||||
For our code, we mutate the documents we want to return with a special field: `__textScore`.
|
||||
|
||||
This field declares how 'close' the provided query was to the $text fields in this document.
|
||||
|
||||
This is sometimes exposed in the API for sorting reasons.
|
||||
|
||||
!!! bug
|
||||
MongoDB's $text matching algorithm isn't great for fuzzy matches - It doesn't
|
||||
like song titles like 'A', as it thinks 'a' is an article, and doesn't match it properly as
|
||||
a word.
|
||||
|
||||
!!! info
|
||||
Why not regex for fuzzy matches?
|
||||
|
||||
Regex has performance issues on larger datasets and we
|
||||
want to avoid it. Most regexes cannot use indexes, and therefore invoke a COLLSCAN, which
|
||||
we want to avoid.
|
||||
|
||||
## User Searching
|
||||
|
||||
Searching users, on the other hand, has to use regex-based
|
||||
searching.
|
||||
|
||||
The `$text` method attempts to break things up based on their
|
||||
words, but that doesn't help with usernames, as they are all
|
||||
too frequently `XxX_One_Long_Str1ng_xXx`.
|
||||
|
||||
Instead, we use a case insensitive regex - similar to SQL's
|
||||
`LIKE`.
|
||||
|
||||
This means we do not have a `__textScore` property for this
|
||||
search to sort on. Instead, we just constrict returns to
|
||||
around 15, and have the user whittle their search down
|
||||
better.
|
||||
@@ -0,0 +1,135 @@
|
||||
# Songs And Charts
|
||||
|
||||
Tachi structures its song and chart data in a specific
|
||||
manner in order to avoid copying properties all over
|
||||
the place.
|
||||
|
||||
*****
|
||||
|
||||
## Songs
|
||||
|
||||
Songs look like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "5.1.1.",
|
||||
"artist": "dj nagureo",
|
||||
"id": 1,
|
||||
"firstVersion": "0",
|
||||
"alt-titles": [],
|
||||
"search-titles": [],
|
||||
"data": {
|
||||
"genre": "PIANO AMBIENT"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
They're relatively small documents, and contain some metadata.
|
||||
|
||||
Depending on the game, the `data` prop will contain
|
||||
game-specific properties (i.e. not all games have
|
||||
song genres!)
|
||||
|
||||
Songs, however, don't contain any information about
|
||||
their *charts*. The charts are what people actually play.
|
||||
|
||||
In short, songs are *just* a collection of metadata that
|
||||
parents a chart document!
|
||||
|
||||
*****
|
||||
|
||||
## Chart Documents
|
||||
|
||||
Chart Documents **MUST** belong to a song. In the below
|
||||
chart, `songID` refers to the above song document:
|
||||
|
||||
```json
|
||||
{
|
||||
"rgcID": null,
|
||||
"chartID": "c2311194e3897ddb5745b1760d2c0141f933e683",
|
||||
"difficulty": "ANOTHER",
|
||||
"songID": 1,
|
||||
"playtype": "SP",
|
||||
"levelNum": 10,
|
||||
"level": "10",
|
||||
"flags": {
|
||||
"IN BASE GAME": true,
|
||||
"OMNIMIX": true,
|
||||
"N-1": true,
|
||||
"2dxtra": false
|
||||
},
|
||||
"data": {
|
||||
"inGameID": 1000,
|
||||
"notecount": 786,
|
||||
},
|
||||
"isPrimary": true,
|
||||
"versions": [
|
||||
"27-omni",
|
||||
"26-omni",
|
||||
"27",
|
||||
"26",
|
||||
"inf",
|
||||
"16-cs",
|
||||
"12-cs",
|
||||
"10-cs",
|
||||
"8-cs",
|
||||
"7-cs",
|
||||
"bmus"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Songs can have multiple charts, but charts can only have
|
||||
one song.
|
||||
|
||||
## Primary
|
||||
|
||||
Sometimes, games like to rechart things and release them
|
||||
under the exact same song. Sometimes, they even do this
|
||||
under the *exact* same internal songID!
|
||||
|
||||
A 'Primary' chart refers to a chart that is the *current*
|
||||
variant of that chart for this game. Non-Primary charts
|
||||
are not eligible for any rating calculations, nor do they
|
||||
contribute to profile rating.
|
||||
|
||||
To check if a chart is primary or not, the `isPrimary` property
|
||||
handles that.
|
||||
|
||||
For example:
|
||||
|
||||
```json
|
||||
{
|
||||
"rgcID": null,
|
||||
"chartID": "103ff8bb004e1a8a005f808c025c3feb",
|
||||
"difficulty": "ANOTHER",
|
||||
"songID": 1,
|
||||
"playtype": "SP",
|
||||
"levelNum": 5,
|
||||
"level": "5",
|
||||
"flags": {
|
||||
"IN BASE GAME": true,
|
||||
"OMNIMIX": true,
|
||||
"N-1": true,
|
||||
"2dxtra": false
|
||||
},
|
||||
"data": {
|
||||
"inGameID": 1000,
|
||||
"notecount": 433
|
||||
},
|
||||
"isPrimary": false,
|
||||
"versions": [
|
||||
"1"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Notice that the chartID is different, and `isPrimary`
|
||||
is set to false.
|
||||
|
||||
Even though this chart is "5.1.1 (SP ANOTHER)", it isn't
|
||||
the primary chart for this songID + playtype + difficulty.
|
||||
|
||||
For the UI, Tachi will hide non-primary charts by default.
|
||||
Realistically, they only exist to support legacy scores
|
||||
without having to throw them away when a rechart occurs.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Converter Failures
|
||||
|
||||
This page contains the list of possible converter fail
|
||||
states that can be thrown. All of them can be found in
|
||||
`src/lib/score-import/common/converter-failures.ts`.
|
||||
|
||||
*****
|
||||
|
||||
## SkipScoreFailure
|
||||
|
||||
This is a special case of a failure, in that it isn't
|
||||
logged or marked as an error in the import. This call
|
||||
just means we should skip the score.
|
||||
|
||||
An example case of this would be something like the
|
||||
eamusement IIDX CSV format, which can pass 'empty' scores
|
||||
for a chart. We would want to skip over those, and they
|
||||
aren't an error.
|
||||
|
||||
When this is thrown, the score is skipped, and nothing is
|
||||
imported.
|
||||
|
||||
## SongOrChartNotFoundFailure
|
||||
|
||||
!!! bug
|
||||
KTData means 'Kamaitachi Data', but this applies to
|
||||
Bokutachi just as much, it is just a legacy name
|
||||
holdover.
|
||||
|
||||
This failure means that the score could not match with
|
||||
anything in the database, such as giving a chartID that
|
||||
Tachi does not have in its database.
|
||||
|
||||
This failure also takes the current data, importType and
|
||||
parser context. This is so we can create [Orphan Scores](./orphans.md).
|
||||
|
||||
## Invalid Score Failure
|
||||
|
||||
This score provided invalid data that we cannot process.
|
||||
|
||||
An example of this would be an IIDX score with -1 EX Score.
|
||||
|
||||
## Internal Failure
|
||||
|
||||
This is an internal failure, and thrown when something bad
|
||||
has occured on our end.
|
||||
|
||||
The most common case for throwing this is the "Song-Chart"
|
||||
desync. This occurs when a chart has no parent song (and
|
||||
a chart must have a parent song.)
|
||||
|
||||
!!! info
|
||||
This is generally used for states that should never happen,
|
||||
but we want to handle them nicely anyway.
|
||||
|
||||
## Any Exception
|
||||
|
||||
If any exception is thrown from a converter that is **NOT**
|
||||
an `instanceof` ConverterFailure, it is treated as an
|
||||
InternalFailure.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Updating Goals
|
||||
|
||||
If a user has goals set, we need to update their
|
||||
progress on those goals as a result of this import.
|
||||
|
||||
*****
|
||||
|
||||
## Getting Relevant Goals
|
||||
|
||||
In V1 of Tachi, we fetched *all* goals for a user,
|
||||
and recalculated all of them.
|
||||
|
||||
This was fine for users with some goals, but for users with
|
||||
lots of goals, it quickly became a huge performance boon.
|
||||
|
||||
To avoid that, Tachi V2 fetches only the "relevant" goals
|
||||
as a result of the import. This is calculated as follows:
|
||||
|
||||
- Single Goals are matched if their chartID is inside the set of chartIDs affected.
|
||||
|
||||
- Multi Goals are matched if their data contains any chartIDs that were affected.
|
||||
|
||||
- Folder goals are matched if the folder contains any charts inside the set of chartIDs affected.
|
||||
|
||||
## Processing Goals
|
||||
|
||||
For every goal matched, we iterate over it and evaluate
|
||||
it using `EvaluateGoalForUser`.
|
||||
|
||||
We then need to convert the returns of that function into
|
||||
the expected goal format for ImportDocuments.
|
||||
|
||||
That means we have to compare the old UserGoal's progress
|
||||
with the newly returned one.
|
||||
|
||||
If nothing has changed, we return undefined. If something
|
||||
has changed, we return a `bwrite` key that contains a
|
||||
MongoDB bulk-write operation to update the UserGoal, and the expected import format of goalID, old, new.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Import Documents and Import Timings
|
||||
|
||||
The final step of the process is to coalesce all the returns
|
||||
of the various steps above into one analysable document
|
||||
and return it.
|
||||
|
||||
*****
|
||||
|
||||
## Logging
|
||||
|
||||
If the import had over 500 scores, we log the import at Info level. This is because those are generally rare, and we
|
||||
want to know how our performance is doing at a glance.
|
||||
|
||||
If the import has over 1 score, it is logged as verbose.
|
||||
|
||||
Else, the import is logged as debug.
|
||||
|
||||
## Timings
|
||||
|
||||
An internal document - Import Timings - is created by
|
||||
storing the time each step of the import process took
|
||||
in miliseconds.
|
||||
|
||||
This is analysed regularly to check for performance
|
||||
degradation and look for potential optimisations.
|
||||
|
||||
This is also stored in the database.
|
||||
|
||||
## Returns
|
||||
|
||||
The Import Document **is** the return from ScoreImportMain.
|
||||
|
||||
This document contains all the relevant information
|
||||
about the import and what it has resulted in, and
|
||||
is also stored in the database under a random ID.
|
||||
@@ -0,0 +1,65 @@
|
||||
# Import Types
|
||||
|
||||
The score import code for Tachi uses 'Import Types' to
|
||||
determine what kind of code to run on a given import,
|
||||
and what converter to call (more on that later.)
|
||||
|
||||
*****
|
||||
|
||||
## Format
|
||||
|
||||
An import type is just a string. It is formatted as follows:
|
||||
|
||||
```
|
||||
type/name
|
||||
```
|
||||
|
||||
## Types
|
||||
|
||||
There are three types.
|
||||
|
||||
| Type | Description |
|
||||
| :: | :: |
|
||||
| `file/` | The data is coming from a file, such as one from a multipart form submission. |
|
||||
| `ir/` | The data is coming from a HTTP Request sent to us. |
|
||||
| `api/` | The data is being fetched by us from another service through HTTP Requests. |
|
||||
|
||||
## User Intent
|
||||
|
||||
Some imports are performed with user intent, such as
|
||||
the user going to the Tachi website and uploading
|
||||
a file and clicking submit.
|
||||
|
||||
Some imports are performed without user intent, i.e.
|
||||
the user sets up a score hook to automatically import
|
||||
their scores whenever they get one. The reason for the
|
||||
distinction is for hiding "automatic" imports from the
|
||||
users imports page.
|
||||
|
||||
For certain endpoints, the special header `X-User-Intent`
|
||||
is set to indicate that this was done with User Intent.
|
||||
|
||||
!!! warning
|
||||
As this is just a convention, you can abuse it, but you
|
||||
shouldn't, because it's not nice to break things
|
||||
(and you may be banned).
|
||||
|
||||
## List of Import Types
|
||||
|
||||
| ImportType | Description | Availability | User Intent |
|
||||
| :: | :: | :: | :: |
|
||||
| `file/eamusement-iidx-csv` | The E-amusement CSV format for IIDX. This type accepts both Pre-HV and Post-HV formats. | Kamaitachi | Yes |
|
||||
| `file/batch-manual` | The Tachi BATCH-MANUAL format submitted through a multipart form. | Kamaitachi & Bokutachi | Yes |
|
||||
| `file/solid-state-squad` | The IIDX XML output by Solid State Squad. | Kamaitachi | Yes |
|
||||
| `file/mer-iidx` | The IIDX JSON output from MER. | Kamaitachi | Yes |
|
||||
| `file/pli-iidx-csv` | Same as eamusement's IIDX CSV, but output from PLI. | Kamaitachi | Yes |
|
||||
| `ir/direct-manual` | The Tachi BATCH-MANUAL format but submitted in a HTTP Request body as `application/json`. | Kamaitachi & Bokutachi | Depends on Header |
|
||||
| `ir/barbatos` | Barbatos's format submitted in a HTTP Request Body. | Kamaitachi | No |
|
||||
| `ir/fervidex` | Fervidex's score format submitted in a HTTP Request Body. | Kamaitachi | No |
|
||||
| `ir/fervidex-static` | Fervidex's profile sync format submitted in a HTTP Request Body. | Kamaitachi | No |
|
||||
| `ir/usc` | An implementation of the [USCIR](https://uscir.rtfd.io)'s POST /scores endpoint. | Bokutachi | No |
|
||||
| `ir/beatoraja` | A handler for BokutachiIR's score format. | Bokutachi | No |
|
||||
| `api/flo-iidx`, `api/eag-iidx` | Both the same IIDX format, but yielded from different APIs with different players. | Kamaitachi | Depends on Calling[^1] |
|
||||
| `api/flo-sdvx`, `api/eag-sdvx`, `api/min-sdvx`| See above, but for SDVX. | Kamaitachi | Depends on Calling[^1] |
|
||||
|
||||
[^1]: If this is synced manually by the user, I.e. they have called it to be synced, then it is with intent. If it was called for them through automatic synchronisation, then it was not done with user intent.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Importing DryScores
|
||||
|
||||
This page goes over hydrating a DryScore into a fully
|
||||
fledged Tachi score, and then goes over how it is imported.
|
||||
|
||||
!!! warning
|
||||
Performance has been squeezed out of this process
|
||||
rather aggressively, as such, some of the more
|
||||
'obvious' ways to do things have been ignored
|
||||
for performance gains.
|
||||
|
||||
!!! info
|
||||
The code that handles this logic is found in
|
||||
`src/lib/score-import/framework/score-importing/score-importing.ts`.
|
||||
|
||||
*****
|
||||
|
||||
## Return Format
|
||||
|
||||
The `ImportIterableDatapoint` returns
|
||||
`ImportProcessingInfo` or `null`. The format for the former
|
||||
can be found in `tachi-common`, and **should be read** before
|
||||
following this document!
|
||||
|
||||
As a rough outline:
|
||||
```ts
|
||||
{
|
||||
// whether this import worked or not
|
||||
success: boolean;
|
||||
// ScoreImported implies success: true, and all others imply success: false.
|
||||
type: "ScoreImported" | "SongOrChartNotFound" // ... and more;
|
||||
// An error message.
|
||||
message: String | null;
|
||||
// Some errors return some data about the error, or things like scoreImported returns the score that was imported.
|
||||
content: see_implementation
|
||||
}
|
||||
```
|
||||
|
||||
## Dealing with converter returns
|
||||
|
||||
As mentioned in [Parsers and Converters](./parse-conv.md),
|
||||
converters will return a DryScore, and its matching chart
|
||||
and song on success.
|
||||
|
||||
However, failures are also an expected throw from a
|
||||
converter function. We handle these throws by logging
|
||||
them dependent on their severity, and returning
|
||||
that this score has failed to be imported properly
|
||||
in the `ImportProcessingInfo` format.
|
||||
|
||||
If the converter was successful, we now have a DryScore, song and chart to work with.
|
||||
|
||||
## Hydration
|
||||
|
||||
Our first step is to turn that DryScore into a 'real score'.
|
||||
|
||||
!!! info
|
||||
The code for this is found in `HydrateScore`.
|
||||
|
||||
Before anything, we calculated the ScoreID for this score.
|
||||
This is used to dedupe scores, and is a checksum of
|
||||
the following properties.
|
||||
|
||||
- userID
|
||||
- chartID
|
||||
- lamp
|
||||
- grade
|
||||
- score
|
||||
- percent
|
||||
|
||||
We fill out all the properties that can be calculated
|
||||
about the score, such as the aptly named `calculatedData`,
|
||||
and things like `gradeIndex` and `lampIndex`, which are
|
||||
just the index of the grade/lamp string in the set of grade/lamps.
|
||||
|
||||
Other properties, such as `timeAdded` (the time this data was inserted into the database) can be trivially attached onto the score.
|
||||
|
||||
## Insertion
|
||||
|
||||
Now that we have a real Tachi score, we can insert it into
|
||||
the database.
|
||||
|
||||
The obvious solution here is something like:
|
||||
```ts
|
||||
await db.scores.insert(scoreDocument);
|
||||
```
|
||||
|
||||
But this has performance implications for large sets of
|
||||
scores.
|
||||
|
||||
To solve these performance issues, we use a queue to insert
|
||||
scores.
|
||||
|
||||
ScoreDocuments are appended to a queue of 500.
|
||||
If the queue hits 501 scores, the queue is bulk-written to
|
||||
the database. At the end of hydrating and queueing all scores, the queue is flushed one last time.
|
||||
|
||||
This simple optimisation provides large performance benefits.
|
||||
|
||||
## Final Returns
|
||||
|
||||
Importing each datapoint gives us an `ImportProcessingInfo`
|
||||
object OR `null`. We want to return only the former,
|
||||
as `null` is for skipped scores.
|
||||
|
||||
We filter out all the nulls and flush the queue before
|
||||
returning the array of `ImportProcessingInfo` objects
|
||||
that were returned from our import process.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Score Import Main
|
||||
|
||||
This page refers to the entry point function for a score
|
||||
import.
|
||||
|
||||
This function can be found at `src/lib/score-import/framework/score-import-main.ts`.
|
||||
|
||||
## Signature
|
||||
|
||||
`ScoreImportMain` takes four arguments and an optional fifth.
|
||||
|
||||
| Argument | Type | Description |
|
||||
| :: | :: | :: |
|
||||
| `user` | [UserDocument](../../schemas/user.md) | The user that is making this import request. |
|
||||
| `userIntent` | boolean | Whether this import was performed with User Intent - See [Import Types](./import-types.md#user-intent) |
|
||||
| `importType` | ImportType | What kind of import "type" this is. For more on this, see [Import Types](./import-types.md)
|
||||
| `InputParser` | Function | The parser function to call. For more info, see [Parsing and Converting](./parse-conv.md)
|
||||
| `providedImportObjects` (Optional) | { logger, importID } | Optionally, a logger and existing importID can be passed here. This is used for scenarios where the logger and importID have already been created before importMain was called. |
|
||||
|
||||
## Import Logger
|
||||
|
||||
For ease of debugging, a logger is passed between most
|
||||
functions in the score importing process. This logger
|
||||
contains the context of the currently importing user
|
||||
and the ImportID.
|
||||
|
||||
## Steps
|
||||
|
||||
To make the process easier to parse, the importing process
|
||||
is split into multiple steps. These steps may split into
|
||||
further sub-steps, depending on how much occurs in said
|
||||
step.
|
||||
|
||||
!!! tip
|
||||
These steps are also commented inline in
|
||||
`score-import-main.ts`.
|
||||
|
||||
!!! warning
|
||||
These steps are only an outline of the process!
|
||||
|
||||
For more information on each specific step, links
|
||||
are given.
|
||||
|
||||
### Setup
|
||||
|
||||
If `providedImportObjects` is not given, then ScoreImportMain generates a new Logger and an ImportID.
|
||||
|
||||
### Parsing
|
||||
|
||||
This step calls the ParserFunction provided as an argument
|
||||
to this function. The returns from that are then used
|
||||
in the below steps.
|
||||
|
||||
- Full Page: [Parsing and Converting](./parse-conv.md)
|
||||
|
||||
### Importing
|
||||
|
||||
We iterate over the `iterable` returned from the Parser Function,
|
||||
calling the Converter Function for each element.
|
||||
|
||||
Once we have converted the element, we can fill it in
|
||||
and insert it into the database.
|
||||
|
||||
This process returns an array called the "Import Info".
|
||||
|
||||
This object holds information about the set of charts this
|
||||
import involved, and the set of scores that were imported.
|
||||
|
||||
- Full Page (Converting): [Parsing and Converting](./parse-conv.md)
|
||||
- Full Page (Importing): [Importing](./importing.md)
|
||||
|
||||
!!! info
|
||||
This step does most of the work, and is where a
|
||||
lot of our complexity lies.
|
||||
|
||||
### Parse Import Info
|
||||
|
||||
The next steps require certain information from our Import Info.
|
||||
|
||||
This step parses that data into some values we have use for, such as the set of chartIDs impacted
|
||||
by this import.
|
||||
|
||||
- Full Page: [Parsing ImportProcessingInfo](./parse-ipi.md)
|
||||
|
||||
### Sessions
|
||||
|
||||
Now that we have imported the scores and have got
|
||||
an array of scoreIDs newly created from Parse Import Info,
|
||||
we can move on to creating sessions from this import.
|
||||
|
||||
- Full Page: [Sessions](./sessions.md)
|
||||
|
||||
### Personal Bests
|
||||
|
||||
This step updates the 'personal bests' for the user. It
|
||||
iterates over every unique chartID modified in this import
|
||||
and conjoins the users best scores on the chart into one document.
|
||||
|
||||
This is to have a single document that has the users best
|
||||
lamp and score, which avoids having to join it together
|
||||
whenever we want to display a users best score and lamp
|
||||
at the same time.
|
||||
|
||||
- Full Page: [Personal Bests](./pbs.md)
|
||||
|
||||
### Game Stats
|
||||
|
||||
This step updates the user's Game Stats.
|
||||
|
||||
If this is the users first play on this game, then we create
|
||||
game stats for them.
|
||||
|
||||
The update is performed by recalculating their statistics
|
||||
with the new scores inserted.
|
||||
|
||||
!!! info
|
||||
This section is referred to as UGS sometimes internally.
|
||||
|
||||
This is short for User Game Stats.
|
||||
|
||||
- Full Page: [User Game Stats](./ugs.md)
|
||||
|
||||
### Goals
|
||||
|
||||
This step checks the charts modified in this import and
|
||||
retrieves all the relevant goals. It then checks
|
||||
all of these goals for whether their status has
|
||||
changed or not, and updates them accordingly.
|
||||
|
||||
- Full Page: [Goals](./goals.md)
|
||||
|
||||
### Quests
|
||||
|
||||
Checks the set of goals modified in the previous step,
|
||||
and gets the relevant quests. This step then re-evaluates all of those quests and
|
||||
updates accordingly.
|
||||
|
||||
- Full Page: [Quests](./quests.md)
|
||||
|
||||
### Import Document
|
||||
|
||||
Finally, we take all of the returns of the above steps
|
||||
and combine them into an import document.
|
||||
|
||||
This is inserted to the database, and is also the return
|
||||
value of this function.
|
||||
|
||||
### Import Timings
|
||||
|
||||
Every step in this process is timed in miliseconds.
|
||||
|
||||
The time each step took is saved alongside the ImportID.
|
||||
|
||||
This is used for internal debugging and degraded performance
|
||||
checks.
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# Orphan Scores
|
||||
|
||||
SongOrChartNotFoundFailures state that we don't have the data
|
||||
to understand what chart/song this score is for.
|
||||
|
||||
However, that doesn't necessarily imply we won't have that
|
||||
data in the future. Orphaned scores are a way of storing
|
||||
score-like data without a song or chart as a parent.
|
||||
|
||||
Then, when the song and chart are made available, those
|
||||
scores can be imported as if nothing had changed!
|
||||
|
||||
*****
|
||||
|
||||
## How do they work?
|
||||
|
||||
Orphaned scores store the `data` and `context` from the parser.
|
||||
|
||||
When unorphaning is attempted, the Converter Function is
|
||||
called with that data and context.
|
||||
|
||||
If it results in a SongOrChartNotFoundFailure, we do nothing.
|
||||
|
||||
If it results in a success, we have a new DryScore we can import.
|
||||
|
||||
!!! info
|
||||
If another type of failure occurs - i.e. InvalidScoreFailure,
|
||||
then we remove the orphaned score from the DB.
|
||||
|
||||
!!! note
|
||||
This is why Converter Functions are completely
|
||||
static, so we can safely call the same Converter
|
||||
Function with this data to unorphan scores.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Score Import Overview
|
||||
|
||||
Score importing in Tachi is *very* complex, and involves
|
||||
a lot of moving parts. For that reason, the entire process
|
||||
(and my rationale behind each step) is documented here.
|
||||
|
||||
The code for Score Importing can be found at
|
||||
`src/lib/score-import`.
|
||||
|
||||
The entry point function is `src/lib/score-import/framework/score-import-main.ts`, and everything else is located nearby.
|
||||
|
||||
*****
|
||||
|
||||
## Folders
|
||||
|
||||
Inside `src/lib/score-import` is two folders, `framework/`
|
||||
and `import-types`.
|
||||
|
||||
### `framework/`
|
||||
|
||||
The framework folder contains the 'moving parts' of the score importing,
|
||||
such as processing pbs and inserting scores.
|
||||
|
||||
This folder contains the "entry point" for the importing
|
||||
mechanism, which is discussed in [Score Import Main](./main.md).
|
||||
|
||||
### `import-types/`
|
||||
|
||||
The Import-Types folder contains the specific parsers
|
||||
and converter functions for a given ImportType.
|
||||
|
||||
You can read more about import types [here](./import-types.md).
|
||||
@@ -0,0 +1,323 @@
|
||||
# Parsing and Converting
|
||||
|
||||
Tachi faces an interesting problem with its score importing
|
||||
code. We could rewrite our entire score importing process
|
||||
for every possible import method, or we could try to
|
||||
factor out the common parts of this process.
|
||||
|
||||
The latter is what we've gone for, but it isn't without
|
||||
its own traps.
|
||||
|
||||
We face the first obvious problem when it comes to actually
|
||||
"getting" the scores from a source and turning them into
|
||||
our format.
|
||||
|
||||
Our source needs to be dynamically set, sometimes we might
|
||||
be drawing scores from a file, sometimes from a HTTP
|
||||
request body, sometimes from another API!
|
||||
|
||||
Then, we need to convert the data that document gave us
|
||||
into a format the rest of our code to understand.
|
||||
|
||||
To solve this problem, Tachi uses two processes,
|
||||
Parsing and Converting.
|
||||
|
||||
## Outline
|
||||
|
||||
As a rough outline, the parser converts data from
|
||||
any source into an iterable.
|
||||
|
||||
The iterable is then traversed, and the Converter
|
||||
is called for every element of the iterable.
|
||||
|
||||
The converter function turns the element into something
|
||||
the rest of the Tachi import code can work with.
|
||||
|
||||
## Parsing
|
||||
|
||||
Parsing is the job of drawing the data from the source into
|
||||
an **iterable** (The usage of iterable and **NOT** array will be explained later).
|
||||
|
||||
A parser function is intended to be created *outside* of
|
||||
score-import-main, which means it can be really whatever
|
||||
it wants to be, and draw data from whatever source it needs
|
||||
to.
|
||||
|
||||
!!! warning
|
||||
Parser functions return more than just an iterable,
|
||||
but in the interest of introducing complexity slowly
|
||||
we're going to pretend that they only return an
|
||||
iterable for now.
|
||||
|
||||
### Implementation
|
||||
|
||||
As an example, let's say we had a CSV being sent to us as a file.
|
||||
|
||||
We could write a parser for that as follows:
|
||||
|
||||
```js
|
||||
// note: awful example code, this is NOT how to parse a CSV!
|
||||
function ParserFunction(request) {
|
||||
let csv = request.file;
|
||||
|
||||
return csv.toString().split("\n");
|
||||
}
|
||||
```
|
||||
|
||||
This takes the provided file, splits it on newlines, and returns
|
||||
the array. This meets all our criteria, as it returns
|
||||
an iterable, but we still have problems.
|
||||
|
||||
Firstly, the calling signature for this involves a `request`. Score Import Main doesn't know of any HTTP requests, as it might not even be triggered by one.
|
||||
|
||||
We could solve this by passing a request object, and
|
||||
also any other context a parser could possibly need,
|
||||
but that quickly breaks down when we start needing to
|
||||
fetch from APIs.
|
||||
|
||||
Instead, we define a ParserFunction is defined to be a function that
|
||||
takes *one* argument, the Import Logger.
|
||||
|
||||
This means that we actually have to use some
|
||||
closures in order to call a parser.
|
||||
|
||||
```js
|
||||
function CreateParser(request) {
|
||||
return (logger) => {
|
||||
return request.file.toString().split("\n");
|
||||
}
|
||||
}
|
||||
|
||||
const ParserFunction = CreateParser(request);
|
||||
```
|
||||
|
||||
Using closures, we can dynamically create the parser function
|
||||
and enclose any context it needs to use inside of it.
|
||||
|
||||
This approach allows ScoreImportMain to continue knowing
|
||||
nothing about where the data is coming from, and
|
||||
avoids passing lots of initial context, bloating up the
|
||||
score import function.
|
||||
|
||||
The iterable returned by the parser should be in the
|
||||
most immediately sensible format.
|
||||
|
||||
!!! info
|
||||
Parser Functions may be asynchronous.
|
||||
|
||||
### Iterable Vs. Array
|
||||
|
||||
Above, we specifically mentioned that the Parser function
|
||||
returns an *iterable*. There's a subtle difference here
|
||||
that allows for significant performance gains when getting
|
||||
data from a paginated API.
|
||||
|
||||
Let's write an example where we retrieve data from an API.
|
||||
|
||||
```js
|
||||
function CreateParser() {
|
||||
return async (logger) => {
|
||||
let page = 1;
|
||||
|
||||
let array = [];
|
||||
while (true) {
|
||||
// pretend this returns some scores
|
||||
|
||||
let res = await fetch("example.com?page=" + page).then(r => r.json());
|
||||
|
||||
array.push(...res.scores);
|
||||
|
||||
if (!res.moreData) {
|
||||
break;
|
||||
}
|
||||
|
||||
page++;
|
||||
}
|
||||
|
||||
return array;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
It's not bad, but it has some performance issues if the
|
||||
user has a lot of scores, by creating a massive array
|
||||
in memory and *then* passing it.
|
||||
|
||||
We can improve this code significantly using generators.
|
||||
|
||||
```js
|
||||
function CreateParser() {
|
||||
return async function* (logger) {
|
||||
let page = 1;
|
||||
|
||||
while (true) {
|
||||
let res = await fetch("example.com?page=" + page).then(r => r.json());
|
||||
|
||||
for (const score of res.scores) {
|
||||
yield score;
|
||||
}
|
||||
|
||||
if (!res.moreData) {
|
||||
break;
|
||||
}
|
||||
|
||||
page++;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
With generators, this code can now yield each score from
|
||||
the data returned, instead of having to bubble it up in
|
||||
one large array.
|
||||
|
||||
### Other Returns
|
||||
|
||||
We mentioned above that parsers return more than
|
||||
just an iterable.
|
||||
|
||||
In practice, a Parser returns four things in an object. Context, the Game this import is for, the Iterable discussed above and a Class Handler.
|
||||
|
||||
#### `context`
|
||||
|
||||
Some import formats require context to be parsed properly.
|
||||
|
||||
The most obvious example is the IIDX E-Amusement CSV format, which does not declare whether the scores are from
|
||||
single or double play inside the file.
|
||||
|
||||
To properly interpret a score of, say, 1000 on 5.1.1. ANOTHER, we need to have some context about the score.
|
||||
That is, we need data that isn't part of the 'score'.
|
||||
|
||||
To solve this, parsers return a `context` object which
|
||||
contains context for the converter function.
|
||||
|
||||
For IIDX E-Amusement, the parser function could interpret
|
||||
the playtype from user input, and then return it as context.
|
||||
|
||||
#### `game`
|
||||
|
||||
Sometimes, the game an import is for is not statically
|
||||
known from the Import Type. This means that the parser
|
||||
will have to return the `game` this import is for rather
|
||||
than inferring it.
|
||||
|
||||
An example of a format doing this would be the [BATCH MANUAL](foo) format,
|
||||
which declares what game the import is for in a header.
|
||||
|
||||
#### `classHandler`
|
||||
|
||||
Sometimes, an import may result in the changing of a users'
|
||||
classes. This may also sometimes be dependent on data that
|
||||
is part of the import.
|
||||
|
||||
An example of this would be the Fervidex Static import type,
|
||||
which passes a set of the users scores, and also passes
|
||||
their Dans.
|
||||
|
||||
Since dans are part of the user's IIDX classes, we need to
|
||||
take the information from that request and update classes
|
||||
accordingly.
|
||||
|
||||
The classHandler returned from the parser should be a callable function
|
||||
that will update the user's classes dependent on this data.
|
||||
|
||||
!!! info
|
||||
If no class information is dependent on the content
|
||||
known by the parser function, then `classHandler`
|
||||
can be set to null, and it will not be called.
|
||||
|
||||
### Final Example
|
||||
|
||||
With all these other things to return, we can
|
||||
write a simple example of a parser as follows:
|
||||
|
||||
```js
|
||||
function CreateIIDXFileParser(request) {
|
||||
return (logger) => {
|
||||
let csv = request.file;
|
||||
|
||||
let playtype = request.body.playtype;
|
||||
|
||||
if (playtype !== "SP" && playtype !== "DP") {
|
||||
// todo: throw error
|
||||
}
|
||||
|
||||
return {
|
||||
iterable: csv.toString().split("\n"),
|
||||
game: "iidx",
|
||||
context: {
|
||||
playtype,
|
||||
},
|
||||
classHandler: null,
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Converters
|
||||
|
||||
let's say we have a parser for SDVX that returns:
|
||||
```js
|
||||
[{
|
||||
score: 9000000,
|
||||
lamp: "HARD CLEAR",
|
||||
timestamp: 1623760037
|
||||
},
|
||||
// <more similar elements here...>
|
||||
]
|
||||
```
|
||||
|
||||
Our converter has to take every element here and
|
||||
*convert* it into the form the rest of Tachi can
|
||||
work with.
|
||||
|
||||
Unlike Parser Functions, Converter Functions are static,
|
||||
that means that they cannot be dynamically constructed
|
||||
with closures or similar.
|
||||
|
||||
### Arguments
|
||||
Converters are called with these four arguments:
|
||||
|
||||
| Property | Type | Description |
|
||||
| :: | :: | :: |
|
||||
| `data` | ParserElement | The given element in the parser's iterable. |
|
||||
| `context` | ParserContext | The context the parser returned. |
|
||||
| `importType` | [ImportType](./import-types.md) | The import type this import is for. |
|
||||
| `logger` | Logger | The import logger. |
|
||||
|
||||
### Returns
|
||||
|
||||
Converter functions are expected to return three values
|
||||
on success.
|
||||
|
||||
- `dryScore`
|
||||
|
||||
A dry score is a 'partial' real Tachi Score format
|
||||
with certain properties left blank (because they can be
|
||||
filled elsewhere).
|
||||
|
||||
In short, the Dry Score is all the properties that can
|
||||
only be derived from the parser's passed data.
|
||||
|
||||
Other properties, like calculated data are filled out
|
||||
in a score "hydration" process (hence the name Dry Score).
|
||||
|
||||
- `chart`
|
||||
|
||||
The chart this score is for.
|
||||
|
||||
- `song`
|
||||
|
||||
The song this score is for.
|
||||
|
||||
### Throws
|
||||
|
||||
Converters are also expected to fail.
|
||||
For this, we have a very specific throw format
|
||||
that should be thrown whenever a specific error has occured.
|
||||
|
||||
These are called Failures, and a full list of them can be
|
||||
found [here](./conv-failures.md).
|
||||
|
||||
If a non-failure is thrown, it is logged as an error and the score is ignored (because crashing the entire
|
||||
import from an unexpected error is a bad idea.)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Parsing Import Processing Info
|
||||
|
||||
The ImportProcessingInfo array returned from our
|
||||
importing process is unwieldy to work with. We are interested
|
||||
in what it has to say, though.
|
||||
|
||||
This step converts that array into data we use
|
||||
in successive score import steps.
|
||||
|
||||
!!! info
|
||||
Because the logic for this is essentially just one
|
||||
loop, it does not get its own file in `score-importing/`.
|
||||
|
||||
Instead, it is defined inside `score-import-main.ts`.
|
||||
|
||||
*****
|
||||
|
||||
## Output
|
||||
|
||||
| Property | Type | Description |
|
||||
| :: | :: | :: |
|
||||
| `scoreIDs` | Array<string> | The array of scoreIDs imported as a result of this import. |
|
||||
| `errors` | Array<{ type, message }> | An array of the failed ImportProcessingInfo's Types and Error Messages.
|
||||
| `chartIDs` | Set<string> | A set of the chartIDs modified by this import. This is a set to ensure that every chartID here is unique. |
|
||||
| `scorePlaytypeMap` | Record<Playtype, Array<Score>> | All of the score documents returned filtered into Playtype buckets.
|
||||
|
||||
### What's with ScorePlaytypeMap?
|
||||
|
||||
Some imports may import scores from multiple Playtypes. We
|
||||
frequently want to only refer to the set of scores modified
|
||||
under a given playtype (For importing a users game:playtype stats, as an example)
|
||||
so this is a very useful data structure to have.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Personal Bests
|
||||
|
||||
Tachi has a lot of scenarios where it needs to reference
|
||||
a users best score alongside their best lamp.
|
||||
|
||||
This is a standard for most arcade games to conjoin the
|
||||
best aspects of all your scores, so Tachi needs to
|
||||
replicate that behaviour.
|
||||
|
||||
However, not all games agree on what aspects to join about
|
||||
scores. Although in most scenarios we will be joining
|
||||
a user's best lamp with their best score, there are some
|
||||
games (like IIDX) which have other statistics that need
|
||||
to be conjoined.
|
||||
|
||||
*****
|
||||
|
||||
## Calculating Personal Bests
|
||||
|
||||
A user is only allowed one personal best per chart, this
|
||||
allows us to know what personal bests need to be modified
|
||||
by looking at the set of chartIDs modified from this import.
|
||||
|
||||
We can iterate over that set and recalculate the users
|
||||
personal best for each chart.
|
||||
|
||||
Firstly, we select the users score on the chart with the
|
||||
highest percent, then we select the users score on the
|
||||
chart with the highest lampIndex.
|
||||
|
||||
We can then join the relevant properties of this
|
||||
into a PBScore, where the best parts of the lampPB
|
||||
are unioned with the best part of the scorePB.
|
||||
|
||||
!!! info
|
||||
A ScorePB refers to a user's best Percent/Score on a chart.
|
||||
|
||||
A PBScore refers to the aforementioned conjoined score
|
||||
document.
|
||||
|
||||
## Game Specific Things
|
||||
|
||||
Some games require specific conjoining code. The below games
|
||||
have said exceptional behaviour.
|
||||
|
||||
### SDVX, USC
|
||||
|
||||
VF5 and VF6 depends on the users best lamp and their best score.
|
||||
|
||||
That is, you cannot just select the larger volforce from
|
||||
the two scores, as HC 9m -> NC 9.1M should have the
|
||||
volforce of HC 9.1M.
|
||||
|
||||
### IIDX, BMS
|
||||
|
||||
IIDX and BMS have BP. This number is the amount of bads
|
||||
plus the amount of poors a player made. The lowest non-null
|
||||
value for this should be selected as that property for the
|
||||
PBScore.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Updating Quests
|
||||
|
||||
We update quests by looking at the set of modified
|
||||
goalIDs, and check the user's assigned quests to
|
||||
get those that are affected.
|
||||
|
||||
*****
|
||||
|
||||
## Returns
|
||||
|
||||
We evaluate every quest that contain a goal that has just had it's status change.
|
||||
For each one we create a bulkwrite operation to update the user's quest
|
||||
progress.
|
||||
|
||||
If the user has newly achieved a quest, a Webhook Event
|
||||
is emitted, which could be hooked into by our Discord Bot.
|
||||
|
||||
If the progress has changed at all, the data is
|
||||
pushed to an array which is then returned. This is
|
||||
attached onto the Import Document.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Sessions
|
||||
|
||||
This page documents how Sessions are constructed.
|
||||
For a humanised explaination of sessions, see
|
||||
[this page](../../wiki/features.md#sessions).
|
||||
|
||||
*****
|
||||
|
||||
## Creating New Sessions
|
||||
|
||||
Scores are sorted on the time they were achieved. Scores
|
||||
without timestamps are discarded from any session processing.
|
||||
|
||||
Scores are then grouped into buckets of two hours, in pseudocode:
|
||||
|
||||
```ts
|
||||
let lastTimestamp = 0;
|
||||
|
||||
for (score of scores) {
|
||||
if (score.timestamp > lastTimestamp + TWO_HOURS) {
|
||||
CloseBucket();
|
||||
CreateNewBucketWithThisScore();
|
||||
}
|
||||
else {
|
||||
AppendToBucket();
|
||||
}
|
||||
|
||||
lastTimestamp = score.timestamp;
|
||||
}
|
||||
```
|
||||
|
||||
This means that if there was more than two hours between
|
||||
any two successive scores, the session is marked as terminated
|
||||
and a new bucket of scores is created.
|
||||
|
||||
Once we have gotten all the scores we
|
||||
can create a session from them. We need to generate a random
|
||||
name (we have some stuff in `src/datasets` for this).
|
||||
|
||||
We also need to calculate statistics for this session,
|
||||
this is dependent on the game + playtype being used.
|
||||
|
||||
## Appending To Sessions
|
||||
|
||||
If a score within 2 hours of an existing session,
|
||||
then that score is appended to the session.
|
||||
|
||||
!!! bug
|
||||
Sessions do not currently conjoin if a user was to
|
||||
get a score between two sessions such that
|
||||
there was now less than two hours between the first
|
||||
sessions last score and the second sessions first score.
|
||||
|
||||
This is so difficult to properly resolve that Tachi simply
|
||||
makes no attempt to stop it.
|
||||
@@ -0,0 +1,116 @@
|
||||
# User Game Stats
|
||||
|
||||
A user has overarching statistics for a game.
|
||||
|
||||
This part of the score importing process updates those
|
||||
overarching statistics, such as their profile rating
|
||||
and their classes.
|
||||
|
||||
!!! note
|
||||
If a user has never played this game + playtype
|
||||
combination before, a new user game stats
|
||||
object is saved to the database as a result
|
||||
of this update.
|
||||
|
||||
*****
|
||||
|
||||
## Format
|
||||
|
||||
A User Game Stats object looks like this:
|
||||
```ts
|
||||
{
|
||||
userID: string;
|
||||
game: "iidx" | "bms" // ... so on
|
||||
playtype: "SP" | "DP" // ... the set of playtypes this game supports.
|
||||
// A set of continuous statistics about a user.
|
||||
ratings: {
|
||||
// A record of numbers with keys determined
|
||||
// by the game + playtype combination.
|
||||
// such as:
|
||||
VF6: number;
|
||||
// for SDVX and USC.
|
||||
},
|
||||
// A set of discrete statistics about a user.
|
||||
classes: {
|
||||
// This typically contains things like dan ranks
|
||||
// or skill divisions, such as:
|
||||
dan: 14,
|
||||
// for IIDX, or
|
||||
skillColour: 19
|
||||
// for gitadora.
|
||||
|
||||
// Note that instead of remembering each number
|
||||
// for each class, they are defined as constants
|
||||
// in common/src/constants/game.ts
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Ratings
|
||||
|
||||
Ratings are calculated for the given Game + Playtype combination.
|
||||
|
||||
Different combinations will have different statistics,
|
||||
Gitadora will have a profile skill level, whereas SDVX
|
||||
would use Profile Volforce.
|
||||
|
||||
All the rating functions for a game are declared
|
||||
in `src/lib/score-import/framework/user-game-stats/rating.ts`.
|
||||
|
||||
## Classes
|
||||
|
||||
There are two types of classes in Tachi. Static classes
|
||||
are classes that can always be derived from the data around
|
||||
them. Things like this are typically discrete buckets
|
||||
for continuous data, such as Skill Colours in GITADORA.
|
||||
|
||||
The other type of classes are external classes. These
|
||||
are things like Dans, where the information cannot be
|
||||
derived from any other data around it, and instead must
|
||||
be explicitly told to change.
|
||||
|
||||
### Class Handlers
|
||||
|
||||
You may recall that in [Parsers and Converters](./parse-conv.md) it was documented that parsers can return a
|
||||
ClassHandler. The ClassHandler is intended to handle those
|
||||
explicit changes.
|
||||
|
||||
For example, let's say we have an import type that tells us:
|
||||
```js
|
||||
{
|
||||
scores: [{score: 1000, songID: 1, diff: "spa"}],
|
||||
sp_class: "kaiden"
|
||||
}
|
||||
```
|
||||
|
||||
We could return a classHandler from our parser function
|
||||
that encloses this updated information, such as:
|
||||
```ts
|
||||
function CreateClassHandler(spClass: number): ClassHandler {
|
||||
return (../../* classHandlers have some args but we dont need them for this example */) => {
|
||||
return {
|
||||
dan: spClass
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This returned classHandler is called with 5 arguments
|
||||
and is expected to return a partial record of the users
|
||||
classes. This is then merged with the static classes
|
||||
and returned as the users classes.
|
||||
|
||||
### Static Class Handlers
|
||||
|
||||
Static class handlers are built in, and always called
|
||||
when a user's game stats is updated.
|
||||
|
||||
## Deltas
|
||||
|
||||
Once a user's classes have been calculated, they are diffed
|
||||
against the current user's stats. If they have been
|
||||
increased, then a redis message is sent out, and things
|
||||
like the Tachi discord bot may choose to echo this
|
||||
in the discord.
|
||||
|
||||
The deltas are returned to be part of the ImportDocument.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Codebase Overview
|
||||
|
||||
This part of the documentation is for the [Tachi-Server](https://github.com/TNG-dev/Tachi/tree/staging/server) codebase.
|
||||
|
||||
## Codebase Documentation vs. Code Documentation
|
||||
|
||||
This is documentation for the **Codebase**. **NOT** documentation for the code.
|
||||
|
||||
The distinction is because we aren't writing a library here - there's no need to document function
|
||||
signatures or what function calls are meant to do. That can all be done inline because no other
|
||||
projects depend on our function calls!
|
||||
|
||||
This documentation is more meta-level. Why things are in certain folders, what certain enums
|
||||
correspond to, how `thing` works, etc.
|
||||
|
||||
## Repos and Licenses
|
||||
|
||||
Tachi is a monorepo, and is made up of many projects. These are:
|
||||
|
||||
- `client/`, Which is a React frontend for Tachi.
|
||||
|
||||
The client and the server are fairly decoupled. Someone could trivially create their own frontend client for Tachi.
|
||||
|
||||
- `server/`, Which is an Express-Typescript backend for Tachi.
|
||||
|
||||
This contains all of our API calls, and interfaces with our database, and powers the actual score import engine.
|
||||
|
||||
- `database-seeds/`, Which is a git-tracked set of data to be synced with Tachi.
|
||||
|
||||
**This is the source of truth for the songs, charts, and more on the site!**
|
||||
By submitting PRs to this, you can fix bugs on the website, add new charts, and more.
|
||||
|
||||
- `bot/`, Which is a discord bot frontend for Tachi.
|
||||
|
||||
- `common/`, Which contains common types, utils and functions shared between all other packages.
|
||||
|
||||
This is also published to NPM when it hits production.
|
||||
|
||||
- `docs/`, Which contains Tachi documentation.
|
||||
|
||||
- `sieglinde/`, Which contains our BMS/PMS analysis functions.
|
||||
|
||||
Of these, `server/` and `client/` are licensed under the AGPL3. The `database-seeds/` are licensed under the unlicense, and everything else is MIT.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Tachi API Clients
|
||||
|
||||
Tachi lets users create API Keys, such that programs can interact with their account on their behalf. These API Keys have permissions and other things. You can read about that [here](../../api/auth.md).
|
||||
|
||||
However, it's common for another programmer to want certain permissions for their client. For example, if I was making a score import hook for Tachi, I would need the `score_submit` permission.
|
||||
|
||||
We could ask users to manually create an API Key, copy it into a config file, and hope that they get the permissions right, or we could set up a nice flow.
|
||||
|
||||
## OAuth2 Flow
|
||||
|
||||
Tachi fully supports OAuth2. When you create a Tachi API Client, you can set a `redirectUri`, which is where your OAuth2 flow will go. You can read about it [here](./oauth2.md).
|
||||
|
||||
## Client File Flow
|
||||
|
||||
Alternatively, if you're just looking for a nice UI for users to create an API Key with the right permissions, check out the [Client File Flow](./file-flow.md).
|
||||
@@ -0,0 +1,25 @@
|
||||
# Branching Model
|
||||
|
||||
`Tachi` maintains two long-running branches, and uses a remarkably simple model for merging.
|
||||
|
||||
*****
|
||||
|
||||
## Branches
|
||||
|
||||
### `release/v2.x`
|
||||
|
||||
This is the current release version of `Tachi`, and is automatically deployed into production.
|
||||
|
||||
!!! note
|
||||
Our CI automatically selects the largest value of `x` to use as the production
|
||||
branch.
|
||||
|
||||
### `staging`
|
||||
|
||||
This is the development branch, and is where pull requests
|
||||
are merged to. This will be automatically deployed to the Tachi staging servers
|
||||
for further testing.
|
||||
|
||||
## How should I PR?
|
||||
|
||||
You should submit your PRs for `staging`. If this change should be backported into production, note that in your PR.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Database Seeds
|
||||
|
||||
Tachi tracks the contents of its songs and charts in something called the [Database Seeds](https://github.com/TNG-dev/Tachi/tree/staging/database-seeds).
|
||||
|
||||
The databases in question aren't (normally) altered by the server code. We essentially overload git and its CI tools to version control parts of our database.
|
||||
|
||||
## What's in the seeds?
|
||||
|
||||
The seeds contain all the [SongDocument](../../schemas/song.md)s and [ChartDocument](../../schemas/chart.md)s for all of the games supported by Tachi.
|
||||
|
||||
They also include all Folder Documents, Table Documents and BMS Course Documents.
|
||||
|
||||
## Synchronisation
|
||||
|
||||
When pushes are made to `staging` (the main branch), our running staging servers will automatically update to that new bit of data.
|
||||
|
||||
When pushes are made to `release/2.X`, our running *production* servers will automatically update in the same way.
|
||||
|
||||
## Why bother?
|
||||
|
||||
Making all of this data public and easily accessible is one of the best ways to help out other people making rhythm game tools.
|
||||
|
||||
The database seeds are an invaluable resource for other programmers who don't want to scrape data themselves.
|
||||
|
||||
It's also very useful for Tachi. Having charts on a public git repo allows anyone to trivially PR things they know to be wrong.
|
||||
This level of openness to contribution is great, and has resulted in a lot of good work being done by the community.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Client File Flow
|
||||
|
||||
While we have an [OAuth2 Flow](./oauth2.md), that requires another webserver.
|
||||
What if you just want an API key to throw inside a config file? This is a
|
||||
common use case.
|
||||
|
||||
For this, we have the Client File Flow. This flow is entirely done on our
|
||||
site, and results in the user downloading a file, or copying a string.
|
||||
|
||||
## Outline
|
||||
|
||||
!!! note
|
||||
This documentation uses `bokutachi.xyz` as the example site. You should
|
||||
replace this with the instance of Tachi you're pointing against, if it
|
||||
is different.
|
||||
|
||||
- You navigate the user to `https://bokutachi.xyz/client-file-flow/YOUR_CLIENT_ID`.
|
||||
- They are asked if they want to create an API Key for your client.
|
||||
- If they select yes, an API Key is created for your client, and depending on your client parameters, they get it.
|
||||
|
||||
|
||||
## Download Format
|
||||
|
||||
When you create a Tachi API Client, you can select the `File Template` parameter. This will change the format of the key given to the user.
|
||||
|
||||
For example, Let's say you wanted the user to download a `.json` file with
|
||||
your token.
|
||||
|
||||
You could set a template of something like:
|
||||
|
||||
```json
|
||||
{
|
||||
"tachi-api-token": "%%TACHI_KEY%%",
|
||||
"someOtherField": "foo"
|
||||
}
|
||||
```
|
||||
|
||||
If the `File Template` is not set, it is just output normally, without any templating.
|
||||
|
||||
The first instance of `%%TACHI_KEY%%` will be replaced with the generated API key.
|
||||
|
||||
The other file parameter you control is the `File Name`. If this is set, the user will be presented with a button that will download the above content.
|
||||
|
||||
If it is not set, the contents of the template are shown in browser, and the user will have to copy-paste the API Key.
|
||||
@@ -0,0 +1,185 @@
|
||||
# Logging
|
||||
|
||||
As mentioned in the [Toolchain](./toolchain), we use
|
||||
WinstonJS for logging. We use a slightly modified setup
|
||||
of winston, but the same basic logging principles apply.
|
||||
|
||||
!!! note
|
||||
If you like my defaults for logging, they can be quickly
|
||||
invoked with the [Mei](https://github.com/zkldi/mei) wrapper.
|
||||
|
||||
*****
|
||||
|
||||
## Log Levels
|
||||
|
||||
Tachi uses the following log levels, listed in order of
|
||||
severity.
|
||||
|
||||
### Crit
|
||||
|
||||
```ts
|
||||
logger.crit("foo");
|
||||
```
|
||||
|
||||
`crit` or Critical is the most severe log level in tachi.
|
||||
|
||||
Calling this means **the process must now completely exit**.
|
||||
|
||||
This fail state is triggered in things like not being able
|
||||
to connect to a Mongo or Redis instance, where the application
|
||||
absolutely cannot function anymore and should quit immediately.
|
||||
|
||||
### Severe
|
||||
|
||||
```ts
|
||||
logger.severe("foo");
|
||||
```
|
||||
|
||||
`severe` is the second most severe log level in tachi.
|
||||
|
||||
Calling this means an error has occured, and that error implies
|
||||
it will apply to multiple parts of the codebase.
|
||||
|
||||
This log level is used with things like a Song-Chart
|
||||
desync -- A chart must have a song as a parent, but if it
|
||||
doesn't, that's a severe-level error.
|
||||
|
||||
Think of this like an error with wider-reaching implications.
|
||||
|
||||
### Error
|
||||
|
||||
```ts
|
||||
logger.error("foo");
|
||||
```
|
||||
|
||||
`error` is the third most severe log level in tachi.
|
||||
|
||||
Calling this means an error has occured, but the impact of
|
||||
it is limited to the function or general area it was called in.
|
||||
|
||||
Errors may be recovered from or ignored by the code, but generally that should be reserved for `warn`.
|
||||
|
||||
### Warn
|
||||
|
||||
```ts
|
||||
logger.warn("foo");
|
||||
```
|
||||
|
||||
`warn` indicates that something has gone wrong, but is safely
|
||||
recoverable from, such as a mathematical function being given NaN unexpectedly, and just returning 0.
|
||||
|
||||
`warn` calls **MUST** be recoverable from, and must not
|
||||
imply severe damage to the global state of the application.
|
||||
|
||||
### Info
|
||||
|
||||
```ts
|
||||
logger.info("foo");
|
||||
```
|
||||
|
||||
!!! info
|
||||
This is the default [LOG_LEVEL](./config) for tachi.
|
||||
|
||||
This means that all the levels below this are
|
||||
not displayed to the console or stored.
|
||||
|
||||
`info` indicates that something notable has happened. This
|
||||
should not be used for any errors - instead, it should be
|
||||
used for notable events, such as a new user signing up.
|
||||
|
||||
### Verbose
|
||||
|
||||
```ts
|
||||
logger.verbose("foo");
|
||||
```
|
||||
|
||||
`verbose` indicates that something has happened. This is
|
||||
used for debugging, and is typically only enabled in dev
|
||||
to find out what has gone wrong.
|
||||
|
||||
### Debug
|
||||
|
||||
```ts
|
||||
logger.debug("foo");
|
||||
```
|
||||
|
||||
`debug` is the least notable logging level. This is used
|
||||
exclusively for debugging, and logs typically uninteresting
|
||||
things like Captcha Requests being sent, or a single score was imported.
|
||||
|
||||
## Logger Usage
|
||||
|
||||
In `src/lib/logger/logger.ts` we define a wrapper around winston
|
||||
for our logging needs. We use two functions for this.
|
||||
|
||||
The first function is `CreateLogCtx`. This spawns an
|
||||
instance of the logger with "context".
|
||||
|
||||
This context allows us to keep track of common information
|
||||
for the logger.
|
||||
|
||||
For file-level logging, you should do:
|
||||
|
||||
```ts
|
||||
import CreateLogCtx from "~src/lib/logger/logger";
|
||||
const logger = CreateLogCtx(__filename);
|
||||
|
||||
logger.info("foo");
|
||||
// [src/file.ts] INFO: foo
|
||||
```
|
||||
|
||||
!!! info
|
||||
CreateLogCtx is short for Create Log Context.
|
||||
|
||||
This will spawn an instance of the logger with the context
|
||||
of the current filename.
|
||||
|
||||
For more specific logging, you should create a logger
|
||||
and pass it around as an argument to a function.
|
||||
|
||||
This is used in `src/lib/score-import`, as we create a
|
||||
logger with the user's name and import type as "context".
|
||||
|
||||
```ts
|
||||
const logger = CreateLogCtx(`${username} ${userID}`);
|
||||
|
||||
logger.info("foo");
|
||||
// [zkldi 1] INFO: foo
|
||||
|
||||
SomeOtherFunction(argument1, argument2, logger);
|
||||
```
|
||||
|
||||
!!! info
|
||||
Logger should be the last argument for a function that
|
||||
takes a logger.
|
||||
|
||||
The only exception to this is if it uses our `fetch` API,
|
||||
which **MUST** always be the last call.
|
||||
|
||||
If we have an existing logger and want to append context,
|
||||
we can do that with the `AppendLogCtx` function.
|
||||
|
||||
```ts
|
||||
const logger = CreateLogCtx("foo");
|
||||
|
||||
logger.info("hello!");
|
||||
// [foo] INFO: hello!
|
||||
|
||||
const logger2 = AppendLogCtx("bar", logger);
|
||||
|
||||
logger2.info("hello!");
|
||||
// [foo | bar] INFO: hello!
|
||||
```
|
||||
|
||||
## Log Files
|
||||
|
||||
Logs are saved to `${pwd}/logs` in two files. The first
|
||||
file logs everything above `LOG_LEVEL`. The second file
|
||||
logs `error` and above calls.
|
||||
|
||||
!!! bug
|
||||
Unintentional behaviour occurs here if LOG_LEVEL is
|
||||
above `error` - the main file will not log errors,
|
||||
but the error file will log errors regardless.
|
||||
|
||||
This may be fixed at some point, but is fairly low priority.
|
||||
@@ -0,0 +1,59 @@
|
||||
# OAuth2 Flow
|
||||
|
||||
`tachi-server` has a functional implementation of OAuth2, which lets people create clients to request APIKeys from users.
|
||||
|
||||
This is the preferred way of handling authorisation between web applications, as it can be done without the user ever really having to deal with their API keys!
|
||||
|
||||
!!! note
|
||||
The below steps assume some familiarity with OAuth2. If you are not familiar, I find [this](https://www.digitalocean.com/community/tutorials/an-introduction-to-oauth-2) to be the best explaination.
|
||||
|
||||
We use an *almost* standard OAuth2 flow, but with the added react-app caveat of POSTing for an intermediate token. If that makes sense to you, you don't need to read this page!
|
||||
|
||||
## Process
|
||||
|
||||
In this scenario, we have two users, user A, who is making a service that integrates with Tachi, and user B, who wants to link integrate their service with their tachi profile.
|
||||
|
||||
!!! info
|
||||
In this example we will use `bokutachi.xyz` as the Tachi site name.
|
||||
|
||||
- An OAuth2 client is created by user A.
|
||||
|
||||
This client will have the following properties.
|
||||
|
||||
```json
|
||||
{
|
||||
"clientID": "ABCDEF", // this is a random string in practice.
|
||||
"clientSecret": "GHIJKL", // this is another random string.
|
||||
"name": "Epic Games",
|
||||
"author": 1,
|
||||
"redirectUri": "https://epicgames.example.com/tachi-auth-callback",
|
||||
"requestedPermissions": ["customise_score"],
|
||||
"apiKeyFormat": null, // These are for the Client File Flow.
|
||||
"apiKeyFilename": null, // More on that later.
|
||||
}
|
||||
```
|
||||
|
||||
- User B wants to link their account to this service, and must click on an auth link on Tachi.
|
||||
|
||||
In the `tachi-client`, this link is `https://bokutachi.xyz/oauth/request-auth?clientID={clientID}`
|
||||
|
||||
EpicGames would show this link to the user, and they would click it.
|
||||
|
||||
While on Tachi, they are presented with the option to accept linking with `clientID`, or decline it.
|
||||
|
||||
- If they accept, `tachi-client` will make a POST request to `https://bokutachi.xyz/api/v1/oauth/create-code`, which will create an intermediate authorisation code.
|
||||
|
||||
The user and this authorisation code are then taken to the `redirectUri` defined in the client. In our case, this means they are taken to
|
||||
`https://epicgames.example.com/tachi-auth-callback?code=SOME_INTERMEDIATE_TOKEN`
|
||||
|
||||
This token **IS NOT** an API Key, but rather an intermediate value that needs to then be converted up.
|
||||
|
||||
EpicGames would now have to take this token and make a POST request to `https://bokutachi.xyz/api/v1/oauth/token`, with their client secret and the intermediate token.
|
||||
|
||||
This POST request will then return the API Key EpicGames wants! The user can then be redirected by EpicGames to wherever they want.
|
||||
|
||||
## The application I want to integrate isn't a web app!
|
||||
|
||||
That's fine. Infact, it's very common for us to integrate with applications
|
||||
that just want an API token inside a JSON file. For that, we have the
|
||||
[Client File Flow](./file-flow.md)
|
||||
@@ -0,0 +1,439 @@
|
||||
# Configuration Info
|
||||
|
||||
The codebase uses a file called `conf.json5` to handle
|
||||
various configurable options. It also reads some things from the process environment.
|
||||
|
||||
## What is JSON5?
|
||||
|
||||
JSON5 is an extension of JSON which is better suited
|
||||
for configuration files.
|
||||
|
||||
You can read more about it [here](https://json5.org/), but
|
||||
the main benefits for us are as follows:
|
||||
|
||||
- Comments
|
||||
- No Quoting properties
|
||||
- Trailing Commas
|
||||
|
||||
## Example Config File
|
||||
|
||||
```js
|
||||
{
|
||||
MONGO_DATABASE_NAME: "testingdb",
|
||||
CAPTCHA_SECRET_KEY: "something_secret",
|
||||
SESSION_SECRET: "something_secret",
|
||||
FLO_API_URL: "https://flo.example.com",
|
||||
EAG_API_URL: "https://eag.example.com",
|
||||
MIN_API_URL: "https://min.example.com",
|
||||
FLO_OAUTH2_INFO: {
|
||||
CLIENT_ID: "DUMMY_CLIENT_ID",
|
||||
CLIENT_SECRET: "DUMMY_CLIENT_SECRET",
|
||||
REDIRECT_URI: "https://example.com",
|
||||
},
|
||||
EAG_OAUTH2_INFO: {
|
||||
CLIENT_ID: "DUMMY_CLIENT_ID",
|
||||
CLIENT_SECRET: "DUMMY_CLIENT_SECRET",
|
||||
REDIRECT_URI: "https://example.com",
|
||||
},
|
||||
ARC_AUTH_TOKEN: "unused",
|
||||
ENABLE_SERVER_HTTPS: false,
|
||||
OUR_URL: "https://example.com",
|
||||
INVITE_CODE_CONFIG: {
|
||||
BATCH_SIZE: 2,
|
||||
INVITE_CAP: 100,
|
||||
BETA_USER_BONUS: 5,
|
||||
},
|
||||
CDN_CONFIG: {
|
||||
WEB_LOCATION: "/cdn",
|
||||
SAVE_LOCATION: {
|
||||
TYPE: "LOCAL_FILESYSTEM",
|
||||
LOCATION: "./test-cdn",
|
||||
SERVE_OWN_CDN: true,
|
||||
},
|
||||
},
|
||||
TACHI_CONFIG: {
|
||||
TYPE: "omni",
|
||||
NAME: "Tachi Example Config",
|
||||
GAMES: [
|
||||
"iidx",
|
||||
"museca",
|
||||
"maimai",
|
||||
"sdvx",
|
||||
"ddr",
|
||||
"bms",
|
||||
"chunithm",
|
||||
"usc",
|
||||
],
|
||||
IMPORT_TYPES: [
|
||||
"file/eamusement-iidx-csv",
|
||||
"file/batch-manual",
|
||||
"file/solid-state-squad",
|
||||
"file/mer-iidx",
|
||||
"file/pli-iidx-csv",
|
||||
"ir/direct-manual",
|
||||
"ir/barbatos",
|
||||
"ir/fervidex",
|
||||
"ir/fervidex-static",
|
||||
"ir/beatoraja",
|
||||
"ir/usc",
|
||||
"ir/kshook-sv3c",
|
||||
"api/eag-iidx",
|
||||
"api/eag-sdvx",
|
||||
"api/flo-iidx",
|
||||
"api/flo-sdvx",
|
||||
"api/min-sdvx",
|
||||
],
|
||||
},
|
||||
LOGGER_CONFIG: {
|
||||
FILE: false,
|
||||
CONSOLE: true,
|
||||
LOG_LEVEL: "info",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
!!! warning
|
||||
**DO NOT BLINDLY COPY THIS CONFIGURATION FILE!**
|
||||
|
||||
Seriously, The `SESSION_SECRET` token MUST not be
|
||||
public.
|
||||
|
||||
## JSON5 Properties
|
||||
|
||||
All properties are required unless called optional or they have a default.
|
||||
|
||||
### MONGO_DATABASE_NAME
|
||||
|
||||
- Type: String
|
||||
|
||||
What collection to use for your database.
|
||||
|
||||
### CAPTCHA_SECRET_KEY
|
||||
|
||||
- Type: String
|
||||
|
||||
Google gives us a Captcha Secret Key in order for Captcha
|
||||
to work on our site.
|
||||
|
||||
### SESSION_SECRET
|
||||
|
||||
- Type: String
|
||||
|
||||
This key is used to encrypt Session Cookies.
|
||||
|
||||
!!! warning
|
||||
If this key is figured out, anyone can log in as
|
||||
anyone.
|
||||
|
||||
Make sure this is an appropriately long string
|
||||
generated from a *cryptographically secure* source.
|
||||
That is, do not just mash your keyboard.
|
||||
|
||||
You can generate secure random strings with something
|
||||
like [KeePass](https://keepass.info/).
|
||||
|
||||
### FLO_API_URL, EAG_API_URL, MIN_API_URL
|
||||
|
||||
- Type: String
|
||||
|
||||
The URL for the `FLO, EAG or MIN` services. This is used for integration
|
||||
with the Kamaitachi version of Tachi.
|
||||
|
||||
### FLO/MIN/EAG_OAUTH2_INFO
|
||||
|
||||
- Type: OAuth2Info (Optional)
|
||||
|
||||
If present, these define our OAuth2 Client data for interacting with these services. These are like this like this:
|
||||
```js
|
||||
{
|
||||
FLO_OAUTH2_INFO: {
|
||||
CLIENT_ID: "OUR_CLIENT_ID",
|
||||
CLIENT_SECRET: "OUR_CLIENT_SECRET",
|
||||
REDIRECT_URI: "https://tachi.example.com"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
!!! warning
|
||||
The server will throw a fatal error if you have OAUTH2_INFO set for one service, but not an API_URL.
|
||||
|
||||
Maybe a better solution would be to have the API_URL inside the OAUTH2_INFO. Ah well.
|
||||
|
||||
### ENABLE_SERVER_HTTPS
|
||||
|
||||
- Type: Boolean (optional)
|
||||
|
||||
Whether to use HTTPS. If this is set to true, the files `cert/key.pem` and `cert/cert.pem` will be
|
||||
used as the privateKey and the certificate, respectively.
|
||||
|
||||
!!! note
|
||||
In production, we use an Nginx reverse proxy which handles this.
|
||||
A warning will be emitted if you are using HTTPS mode, as it's not generally meant to be used
|
||||
at this level.
|
||||
|
||||
### CLIENT_DEV_SERVER
|
||||
|
||||
- Type: String
|
||||
- Default: Null
|
||||
|
||||
If present, and a string, this points to the local dev server for a react app. Having this
|
||||
option set results in CORS being enabled for *that* specific URL. This is useful for local
|
||||
development, but should not be used in production.
|
||||
|
||||
|
||||
### RATE_LIMIT
|
||||
|
||||
- Type: Positive Integer
|
||||
- Default: 500
|
||||
|
||||
Determines how many requests an APIKey OR IP can make every minute.
|
||||
|
||||
The default is set to the very generous 500, as it's possible for users to accidentally hit 100 requests/min by refreshing very fast.
|
||||
|
||||
### OAUTH_CLIENT_CAP
|
||||
|
||||
- Type: Positive Integer
|
||||
- Default: 15
|
||||
|
||||
The amount of OAuth2Clients one user can create at any one time. Defaults to 15.
|
||||
|
||||
### OPTIONS_ALWAYS_SUCCEEDS
|
||||
|
||||
- Type: Boolean
|
||||
- Default: false
|
||||
|
||||
If true, all `OPTIONS` requests to the server will return `200`, no matter what. This is a hack used for development CORS.
|
||||
|
||||
### USE_EXTERNAL_SCORE_IMPORT_WORKER
|
||||
|
||||
- Type: Boolean
|
||||
- Default: false
|
||||
|
||||
If true, an external worker process will be used to handle score imports.
|
||||
|
||||
!!! warning
|
||||
You have to run this worker yourself. `pnpm build && pnpm start-score-worker` will
|
||||
run the external worker process.
|
||||
|
||||
### EXTERNAL_SCORE_IMPORT_WORKER_CONCURRENCY
|
||||
|
||||
- Type: Integer
|
||||
- Default: 10
|
||||
|
||||
How many score imports one worker should be allowed to work on at a time. This improves parallelisation of score imports.
|
||||
|
||||
### BEATORAJA_QUEUE_SIZE
|
||||
|
||||
- Type: Integer
|
||||
- Default: 3
|
||||
|
||||
How many unique players have to have played a chart on the beatoraja IR for it to be de-orphaned.
|
||||
|
||||
!!! note
|
||||
Note that LR2 scores or database imports do not count towards this total.
|
||||
|
||||
!!! warning
|
||||
The lowest legal value for this field is 2.
|
||||
|
||||
### OUR_URL
|
||||
|
||||
- Type: String
|
||||
|
||||
Where *this* server is hosted. This is used to
|
||||
provide callback URLs inside emails. You may stub
|
||||
it out if emails are unsupported.
|
||||
|
||||
|
||||
### EMAIL_CONFIG
|
||||
|
||||
- Type: EMAIL_CONFIG | undefined.
|
||||
|
||||
Configures how emails will be sent by Tachi.
|
||||
If not present, email calls will become no-ops, and
|
||||
certain features (such as resetting passwords)
|
||||
will be disabled.
|
||||
|
||||
`FROM` determines the email `From` header, and optionally
|
||||
`SENDMAIL_BIN` can override the location of the `sendmail`
|
||||
binary. Defaults to `/usr/bin/sendmail`, but some distros
|
||||
may have it in `sbin`.
|
||||
|
||||
`TRANSPORT_OPS` Passes a set of options to the email transport. For more
|
||||
information, see the nodemailer documentation for SMTPTransport.Options.
|
||||
|
||||
```ts
|
||||
interface EMAIL_CONFIG {
|
||||
FROM: string;
|
||||
SENDMAIL_BIN?: string
|
||||
TRANSPORT_OPS: any;
|
||||
}
|
||||
```
|
||||
|
||||
### INVITE_CODE_CONFIG
|
||||
|
||||
- Type: INVITE_CODE_CONFIG (Optional)
|
||||
|
||||
Configures how invites are created by Tachi.
|
||||
If not present, the site will not require invite codes at all.
|
||||
|
||||
`BATCH_SIZE` determines how many invites to create every month,
|
||||
`INVITE_CAP` determines how many invites a user can have -- ever.
|
||||
`BETA_USER_BONUS` determines how many additional invites users of Kamaitachi 1 have out of the box.
|
||||
|
||||
```ts
|
||||
interface INVITE_CODE_CONFIG: {
|
||||
BATCH_SIZE: integer;
|
||||
INVITE_CAP: integer;
|
||||
BETA_USER_BONUS: integer;
|
||||
};
|
||||
```
|
||||
|
||||
### TACHI_CONFIG
|
||||
|
||||
- Type: TACHI_CONFIG
|
||||
|
||||
Configures what the Tachi Server instance supports, and what it's generally doing.
|
||||
|
||||
```ts
|
||||
interface TACHI_CONFIG: {
|
||||
NAME: string;
|
||||
TYPE: "ktchi" | "btchi" | "omni";
|
||||
GAMES: Game[];
|
||||
IMPORT_TYPES: ImportTypes[];
|
||||
}
|
||||
```
|
||||
|
||||
#### NAME
|
||||
|
||||
The name of the server. This is reported at `/api/v1/status`.
|
||||
|
||||
#### TYPE
|
||||
|
||||
What type of tachi-server this is. `ktchi` will enable Kamaitachi Only routes, `btchi` will enable
|
||||
Bokutachi only routes, and `omni` will enable both.
|
||||
|
||||
#### GAMES
|
||||
|
||||
What games are supported by this server. For more information, see [Games](../../wiki/games.md).
|
||||
|
||||
#### IMPORT_TYPES
|
||||
|
||||
What importTypes are legal for this server. For more information, see [Import Types](../import/import-types.md).
|
||||
|
||||
### LOGGER_CONFIG
|
||||
|
||||
- Type: LOGGER_CONFIG (Optional)
|
||||
|
||||
Configures how logs are sent around in Tachi.
|
||||
|
||||
```ts
|
||||
interface LOGGER_CONFIG: {
|
||||
LOG_LEVEL: "debug" | "verbose" | "info" | "warn" | "error" | "severe" | "crit";
|
||||
CONSOLE: boolean;
|
||||
FILE: boolean;
|
||||
SEQ_API_KEY: string | undefined;
|
||||
DISCORD?: {
|
||||
WEBHOOK_URL: string;
|
||||
WHO_TO_TAG: string[];
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
#### LOG_LEVEL
|
||||
|
||||
What log level to use out of the box. If no LOGGER_CONFIG is provided, defaults to "info".
|
||||
|
||||
#### CONSOLE
|
||||
|
||||
Whether to log to the console or not. If no LOGGER_CONFIG is provided, defaults to true.
|
||||
|
||||
#### FILE
|
||||
|
||||
Whether to log to a log file or not. If no LOGGER_CONFIG is provided, defaults to false.
|
||||
|
||||
#### SEQ_API_KEY
|
||||
|
||||
(Optional)
|
||||
|
||||
If present, this is an API Key for logging to a Seq server, which is set in the process environment.
|
||||
|
||||
If no LOGGER_CONFIG is provided, this is not set.
|
||||
|
||||
#### DISCORD
|
||||
|
||||
(Optional)
|
||||
|
||||
If present, this configures a discord `WEBHOOK_URL` to log info or higher messages to.
|
||||
`WHO_TO_TAG` is an array of userIDs to tag in the case of a `severe` or `fatal` error.
|
||||
|
||||
If no LOGGER_CONFIG is provided, this is not set.
|
||||
|
||||
### CDN_CONFIG
|
||||
|
||||
- Type: CDN_CONFIG
|
||||
|
||||
Configures the CDN for the Tachi Server. For local development it's recommended you use a local filesystem share,
|
||||
for production usage it's recommended you're behind a CDN.
|
||||
|
||||
```ts
|
||||
interface CDN_CONFIG: {
|
||||
WEB_LOCATION: string;
|
||||
SAVE_LOCATION:
|
||||
| { TYPE: "LOCAL_FILESYSTEM"; LOCATION: string; SERVE_OWN_CDN?: boolean }
|
||||
| {
|
||||
TYPE: "S3_BUCKET";
|
||||
ENDPOINT: string;
|
||||
ACCESS_KEY_ID: string;
|
||||
SECRET_ACCESS_KEY: string;
|
||||
BUCKET: string;
|
||||
KEY_PREFIX?: string;
|
||||
REGION?: string;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
#### WEB_LOCATION
|
||||
|
||||
Configures a URL to redirect users to when returning CDN contents. This could be something like `cdn.bokutachi.xyz`.
|
||||
|
||||
#### SAVE_LOCATION
|
||||
|
||||
Configures where files are actually saved to. If TYPE is "LOCAL_FILESYSTEM", it will save to `SAVE_LOCATION.LOCATION` on the servers drive.
|
||||
|
||||
If TYPE is "S3_BUCKET", it will save files to an S3-API compatible bucket, like S3 itself or Backblaze.
|
||||
|
||||
!!! note
|
||||
LOCAL_FILESYSTEM's `SERVE_OWN_CDN` option is useful for local development, it will mount your cdn as an express endpoint under `/cdn`.
|
||||
|
||||
This saves you having to set up your own NGINX box for serving local files.
|
||||
|
||||
## Process Environment
|
||||
|
||||
The process environment also contains necessary things for functional Tachi Server running.
|
||||
|
||||
!!! info
|
||||
These variables are put into the process environment instead of the conf.json5 file because
|
||||
they're easier to change between docker instances. This Helps us scale and deploy.
|
||||
|
||||
### PORT
|
||||
|
||||
The PORT environment variable specifies what port our express server should listen to.
|
||||
If not set, this will log a warning and default to 8080.
|
||||
|
||||
### MONGO_URL
|
||||
|
||||
Where our mongoDB instance is. This would be `127.0.0.1:27017` if hosting on the same box.
|
||||
If not set, this will terminate the process with a critical error.
|
||||
|
||||
### SEQ_URL
|
||||
|
||||
Where a Seq instance is. If no `LOGGER_CONFIG.SEQ_API_KEY` was defined, this is fine. If it was,
|
||||
this will log a warning, and nothing will be sent to Seq.
|
||||
|
||||
### NODE_ENV
|
||||
|
||||
Expected to be either "dev", "production", "staging" or "test". If not set, this will terminate the process.
|
||||
|
||||
### REPLICA_IDENTITY
|
||||
|
||||
Optional. If present, this declares the identity of this server as a replica.
|
||||
@@ -0,0 +1,138 @@
|
||||
# File/Folder Organisation
|
||||
|
||||
`tachi-server` has a specific setup of files and folders
|
||||
to ensure that code is at where it's most sensible.
|
||||
|
||||
!!! note
|
||||
This documentation is a rough guide for where
|
||||
to place files if you are writing a new file,
|
||||
or where to look for certain functionality.
|
||||
|
||||
It is not a comprehensive tutorial for every file
|
||||
in the repo, as that would be a pain to keep updated.
|
||||
|
||||
*****
|
||||
|
||||
## Top Level
|
||||
|
||||
All of these are at the root level of the project.
|
||||
|
||||
### `/src`
|
||||
|
||||
All of the server TypeScript code goes here.
|
||||
|
||||
### `/js`
|
||||
|
||||
!!! info
|
||||
This folder is gitignored.
|
||||
|
||||
When compiled, `tsc` will output the JS code here.
|
||||
|
||||
### `/scripts`
|
||||
|
||||
Various scripts for interacting with `tachi-server`, such
|
||||
as single-use scripts for importing some data, or
|
||||
frequently used scripts such as updating BMS tables.
|
||||
|
||||
!!! danger
|
||||
The scripts in here are not regularly maintained,
|
||||
**especially** the ones inside `single-use`. You should
|
||||
**ABSOLUTELY NOT** run those if you do not know what
|
||||
they do.
|
||||
|
||||
Seriously, you could destroy your server.
|
||||
|
||||
## TypeScript Source Code
|
||||
|
||||
All of these are inside `/src`.
|
||||
|
||||
### `/datasets`
|
||||
|
||||
Some of `tachi-server`'s code interacts with datasets that
|
||||
aren't worth putting into MongoDB, such as splash text.
|
||||
|
||||
This is mainly for things where we want to randomly select
|
||||
from the list, and not perform any serious lookups - which
|
||||
is why it's a good fit for splash text/automatic session names.
|
||||
|
||||
!!! info
|
||||
Selecting a random element from an entire collection
|
||||
in MongoDB is relatively expensive, and would quadruple
|
||||
the time an import takes.
|
||||
|
||||
!!! warning
|
||||
TypeScript does not support copying over non-code files.
|
||||
|
||||
You can use `cp` in post to move files around, or
|
||||
place the data in memory, either is fine.
|
||||
|
||||
### `/external`
|
||||
|
||||
Code relating to the "external" applications for `tachi-server`,
|
||||
such as MongoDB and Redis.
|
||||
|
||||
### `/lib`
|
||||
|
||||
Sets of code for `tachi-server` functionality. This is the
|
||||
main important part of the codebase for handling things
|
||||
like score imports, logging, and more.
|
||||
|
||||
### `/server`
|
||||
|
||||
This contains the express application that `tachi-server`
|
||||
uses in order to be a server.
|
||||
|
||||
This contains our API, IR implementations and a way of
|
||||
serving our PWA.
|
||||
|
||||
### `/test-utils`
|
||||
|
||||
Tachi's tests need mocks and some specialised code in order
|
||||
to work well. This folder contains all of those things.
|
||||
|
||||
!!! warning
|
||||
**NOTHING** from this folder should be ran in production.
|
||||
|
||||
### `/utils`
|
||||
|
||||
Small utilities for interacting with Tachi, such as
|
||||
functions that retrieve a user given certain params.
|
||||
|
||||
This also contains utilities for handling song/chart
|
||||
database lookups - such as looking up on BMS hash.
|
||||
|
||||
## Express Server
|
||||
|
||||
All of the below folders are under `/src/server`.
|
||||
|
||||
As mentioned above, Tachi stores the routing for our
|
||||
APIs and IR implementations here.
|
||||
|
||||
### `/middleware`
|
||||
|
||||
This contains the middleware we use for the server,
|
||||
such as authentication middleware and such.
|
||||
|
||||
### `/router`
|
||||
|
||||
This contains the actual 'routes' for our server.
|
||||
|
||||
The folders here **MUST** be 1:1 with the endpoints
|
||||
on the server. For example, the implementation of
|
||||
|
||||
```
|
||||
https://bokutachi.xyz/api/v1/foo/bar
|
||||
```
|
||||
|
||||
**MUST** be found at `src/server/router/api/v1/foo/bar/router.ts`
|
||||
|
||||
### Router Files
|
||||
|
||||
The only files allowed to declare endpoints are `router.ts` files.
|
||||
|
||||
This allows us to seperate functionality from API structure
|
||||
in cases where an API call needs to do a lot of things.
|
||||
|
||||
## Test Files
|
||||
|
||||
This documentation has been moved to [its own page](./testing.md)!
|
||||
@@ -0,0 +1,10 @@
|
||||
# Style
|
||||
|
||||
For style, we use a custom tool called [Cadence](https://github.com/CadenceJS/Cadence).
|
||||
|
||||
This is effectively a huge ESLint config with everything I care about enabled.
|
||||
|
||||
It's worth noting that this is an *extremely strict* linter. You are **expected** to have "Format on Save" turned on, in order to use this linter properly.
|
||||
|
||||
!!! info
|
||||
You can run ESLint in the repo any time with `pnpm lint`.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Tests
|
||||
|
||||
Tachi makes extensive use of tests to ensure code
|
||||
is of high quality when its released.
|
||||
|
||||
*****
|
||||
|
||||
## Running Tests
|
||||
|
||||
Tests are ran with [node-tap](https://node-tap.org).
|
||||
|
||||
You can run the server test suite with the following script:
|
||||
|
||||
```
|
||||
pnpm test
|
||||
```
|
||||
|
||||
This will execute every test, and also perform coverage
|
||||
analysis.
|
||||
|
||||
!!! bug
|
||||
This may or may not impact you depending on how
|
||||
dependencies install, but `tap` requires `ts-node`
|
||||
and `typescript` to also be installed.
|
||||
|
||||
Otherwise, you will get a
|
||||
`cannot use import outside of module`
|
||||
error, which indicates that our typescript hasn't
|
||||
been transpiled.
|
||||
|
||||
## Writing Tests
|
||||
|
||||
Tests should be located in the same folder as the file they're testing, and tests should only ever test
|
||||
exports from one file at a time.
|
||||
|
||||
Tests should call things like `ResetDBState` and `CloseAllConnections` as lifecycle hooks to ensure
|
||||
the file exits.
|
||||
|
||||
Tests should use the extension `.test.ts`, and keep
|
||||
the same filename as the file they are testing.
|
||||
|
||||
## Coverage
|
||||
|
||||
Coverage should be kept above 80%. New code should be tested.
|
||||
|
||||
Pull Requests will be rejected if they contain significant
|
||||
changes that are untested!
|
||||
|
||||
## Single Process TAP
|
||||
|
||||
Since we don't mock our MongoDB install out, we run a real instance of Mongo for our tests, and run against it.
|
||||
|
||||
TAP runs every test file in isolation. This is a good idea normally, as it means state is never shared across files, and nothing bad is generally going to happen.
|
||||
|
||||
*However*, it takes around 2 seconds to connect to the MongoDB server when a new test file is started. These 2 seconds add up over the course of a ~1500 test file suite, and result in tests taking over 15-20 minutes.
|
||||
|
||||
To get around this, we use a Single Process TAP hack, which rolls all of our test files up into one test file.
|
||||
|
||||
The hack part here is **not** an understatement, and it is genuinely rather dirty. **BUT**, it runs in ~40-50 seconds, works the same for coverage, and can just generally be ignored as an abstraction layer.
|
||||
|
||||
As a downside, this makes error messages *way* more cryptic, but for a 20x performance gain, it can be lived with.
|
||||
|
||||
!!! warning
|
||||
If you are getting failures like "Child test left in suite", one of your test files is *crashing* the entire process, leaving all other tests just dangling.
|
||||
Reference in New Issue
Block a user