From aa117cdd0ccd871a45a09de55de9fe0eb3909f02 Mon Sep 17 00:00:00 2001
From: bicarus <202771338+bicarus-dev@users.noreply.github.com>
Date: Sat, 22 Aug 2026 21:59:14 -0700
Subject: [PATCH] Updated Video Stream (markdown)

---
 Video-Stream.md | 64 +++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 54 insertions(+), 10 deletions(-)

diff --git a/Video-Stream.md b/Video-Stream.md
index 383b57c..8d31178 100644
--- a/Video-Stream.md
+++ b/Video-Stream.md
@@ -1,19 +1,60 @@
-Guidance for building an app against the `-apistream` video stream: which format to use, what the parameters mean, and what the client is responsible for. The endpoint list and query parameters are in the main README.
+Guidance for building an app against the `-apistream` video stream: how to discover it, which format to use, what the parameters mean, and what the client is responsible for. The endpoint list and query parameters are also in the main README.
 
 > [!NOTE]
 > This feature is in active development and things may change.
 
-## Reference Implementation 
+## Reference Implementation
 
 See https://github.com/spice2x/substream which is a web app that performs streaming & uses WebSocket to send back touch input.
 
 ## Connecting
 
-Open a plain HTTP GET and read the body until you close it or the server does. There is no handshake and no authentication.
+Connect to the JSON API first and call `capture.get_streams()`. Do not guess the stream port or paths: builds may expose different formats, and no stream server listens when `-apistream` is disabled.
 
-The URL is: `http://host:apiport+2/stream.mjpg` where host is the host IP and `apiport` is the value passed to `-api` (and then add two).
+Request:
 
-A stream always begins at a keyframe, so there is nothing to seek and no startup state to recover.
+```json
+{
+    "id": 1,
+    "module": "capture",
+    "function": "get_streams",
+    "params": []
+}
+```
+
+Example response:
+
+```json
+{
+    "id": 1,
+    "errors": [],
+    "data": [{
+        "port": 1339,
+        "formats": [
+            { "name": "mjpeg", "path": "/stream.mjpg" },
+            { "name": "h264", "path": "/stream.h264" }
+        ],
+        "screens": [
+            { "screen": 0, "width": 1920, "height": 1080, "busy": false },
+            { "screen": 1, "width": 1280, "height": 720, "busy": true }
+        ]
+    }]
+}
+```
+
+`data` is empty when no stream server is available. Treat an API failure or an empty response as no stream: do not fall back to deriving a port from the API port.
+
+Use the same host as the API connection, the returned `port`, and the `path` for the desired format. For example:
+
+        http://host:1339/stream.h264?screen=0&fps=30&q=70
+
+The `screens` array only contains screens whose dimensions are known. A screen may be absent while the game is starting or loading, then appear on a later call. Refresh `get_streams()` before retrying and when another viewer connects or leaves. If the user explicitly selected a screen that temporarily disappears, keep that selection and wait for it to return rather than silently switching screens.
+
+`busy` means a stream currently holds that screen. It can be your own stream if you query again after connecting. It is advisory: another client can claim the screen after the API response but before your HTTP connection arrives. A `503 Service Unavailable` is still a normal race to handle with backoff. Include busy screens in selection UIs so a user can choose one and wait for it to become free.
+
+Once discovered, open a plain HTTP GET and read the body until you close it or the server does. The stream port itself has no authentication or additional handshake.
+
+An H.264 stream always begins at a keyframe, so there is nothing to seek and no startup state to recover. Every MJPEG frame is independent.
 
 `screen`, `fps` and `q` are fixed for the life of a connection. Changing any of them means opening a new one.
 
@@ -30,13 +71,15 @@ The server ends the connection if the game changes resolution. Treat a closed co
 | Container | multipart/x-mixed-replace | none |
 | Bandwidth | about 2x to 20x higher | low |
 | Plays in a normal player | yes | yes |
-| Works in a browser | yes, in an `<img>` tag | with some work |
-| Client work | none | drives a decoder itself |
-| Latency | near instant | at least 50ms |
+| Works in a browser | yes, in an `<img>` tag | yes, through WebCodecs or MediaSource |
+| Client work | none | drives a decoder or wraps the stream for MediaSource |
+| Latency | near instant | decoder-dependent |
 
 `stream.h264` is the default choice for an app. H.264 costs roughly twenty times less bandwidth than MJPEG and phones decode it in hardware, which matters a great deal for battery life. It is a bare elementary stream, so the app feeds the bytes to MediaCodec or VideoToolbox itself and nothing sits between the socket and the decoder.
 
-`stream.mjpg` needs no container support at all and every frame stands alone, so it is the fallback when nothing else works, and the only option inside a browser. It is expensive on bandwidth and on client CPU.
+`stream.mjpg` needs no container support at all and every frame stands alone, so it is the fallback when nothing else works. It is expensive on bandwidth and on client CPU.
+
+In a secure browser context, WebCodecs can consume the H.264 access units directly. Where WebCodecs is unavailable, MediaSource can still play H.264 after the client wraps it in fragmented MP4. The MP4 track header needs the correct display dimensions before the first sample is appended; use the `width` and `height` reported for the selected screen rather than hardcoding or guessing them. The [substream](https://github.com/spice2x/substream) reference client implements both paths.
 
 ## Quality
 
@@ -99,9 +142,10 @@ Decode on the platform side and render into a `Texture`. A platform channel hand
 
 - One stream per screen. Anything beyond either limit gets `503 Service Unavailable` and is closed immediately, so back off rather than retrying in a tight loop.
 - A client that vanishes without closing its socket keeps holding its screen until the send times out, which takes a few seconds. A reconnect arriving sooner than that is refused, so retry with backoff rather than assuming the screen is gone for good.
+- The screen list is dynamic. A registered screen is omitted until its size has been measured, so an empty or incomplete list during startup is not an error. Re-query with backoff.
 - Each stream runs its own encoder on the machine hosting the game, and every captured frame costs that machine a present cycle. Streaming two screens at once roughly halves the rate each one can reach.
 - A client that stops reading is dropped after five seconds. The server never buffers a backlog for a slow reader; it skips to the newest frame instead, so frames are lost rather than delayed.
 - View only. Touch and other input still go through the JSON API, so a companion app needs both.
 - No authentication on the stream port. Anyone who can reach it can watch the screen.
-- WinXP builds have no video stream. Neither encoder is compiled in, so every endpoint returns 404, and the JSON API's JPEG screen capture is unavailable for the same reason.
+- WinXP builds have no video stream. Neither encoder is compiled in, so nothing listens on the stream port even with `-apistream`, `capture.get_streams()` returns no data, and the JSON API's JPEG screen capture is unavailable for the same reason.
 - Streaming has performance impact on the game. Limit to 30 FPS.