Created Video Stream (markdown)

Video-Stream.md

@@ -0,0 +1,98 @@
+# Video Stream
+
+Guidance for building an app against the `-apistream` video stream: which format to use, what the parameters mean, and what the client is responsible for. The endpoint list and query parameters are in the main README.
+
+## Connecting
+
+Open a plain HTTP GET and read the body until you close it or the server does. There is no handshake and no authentication.
+
+A stream always begins at a keyframe, so there is nothing to seek and no startup state to recover.
+
+`screen`, `fps` and `q` are fixed for the life of a connection. Changing any of them means opening a new one.
+
+A screen carries one stream at a time. A second connection to a screen that is already streaming is refused with `503 Service Unavailable`, and nothing is ever preempted, so an existing viewer cannot be interrupted by someone else connecting.
+
+`fps` is a ceiling rather than a promise. Nothing competes with you for frames on your own screen, but you receive the lesser of your requested rate and whatever the game is actually redrawing, so a request for 30 on a screen that only updates 21 times a second delivers 21. Derive timing from arrival, not from the value you asked for.
+
+The server ends the connection if the game changes resolution. Treat a closed connection as routine and reconnect rather than surfacing an error.
+
+## Choosing a format
+
+| | `stream.mjpg` | `stream.h264` |
+| --- | --- | --- |
+| Container | multipart/x-mixed-replace | none |
+| Bandwidth | ~20x higher | low |
+| Plays in a normal player | yes | some do, MPC and mpv both decode it |
+| Works in a browser | yes, in an `<img>` tag | no |
+| Client work | none | drives a decoder itself |
+
+`stream.h264` is the default choice for an app. H.264 costs roughly twenty times less bandwidth than MJPEG and phones decode it in hardware, which matters a great deal for battery life. It is a bare elementary stream, so the app feeds the bytes to MediaCodec or VideoToolbox itself and nothing sits between the socket and the decoder.
+
+`stream.mjpg` needs no container support at all and every frame stands alone, so it is the fallback when nothing else works, and the only option inside a browser. It is expensive on bandwidth and on client CPU.
+
+## Quality
+
+`q` means different things per format, despite the shared name and default:
+
+- `stream.mjpg` passes it straight through as the standard 1-100 JPEG quality.
+- `stream.h264` maps it onto the H.264 rate factor as `CRF = 40 - q / 4`.
+
+So for H.264:
+
+| `q` | CRF | |
+| --- | --- | --- |
+| 1 | 39.75 | badly blocky |
+| 40 | 30 | visibly lossy |
+| 70 | 22.5 | default, close to x264's own default of 23 |
+| 100 | 15 | high quality |
+
+CRF is constant quality, not constant bitrate. `q` sets how much loss the encoder tolerates; the resulting bitrate floats with how much the screen is moving.
+
+## Latency
+
+The server puts a frame on the socket about 5 ms after capturing it, so nearly all of the delay a user sees comes from the client.
+
+The usual cause is a player that buffers ahead and then runs its own clock; one that decides during startup that it is half a second behind will stay half a second behind indefinitely. Whatever you use, find the option that renders on decode instead of scheduling against a presentation clock. In mpv:
+
+    mpv --demuxer-lavf-format=h264 --no-correct-pts --container-fps-override=30 --untimed --profile=low-latency http://host:1339/stream.h264
+
+The equivalents are `releaseOutputBuffer(index, true)` on MediaCodec and `kCMSampleAttachmentKey_DisplayImmediately` on VideoToolbox. An annex-b stream carries no timestamps of its own, so nothing downstream has a clock to fall behind on in the first place.
+
+## Testing
+
+`stream.h264` has no container and no timestamps. Some players cope on their own - MPC decodes it as served - while others need the format and frame rate supplied. Either way this is the cheapest way to check the stream is sane:
+
+    mpv --demuxer-lavf-format=h264 --no-correct-pts --container-fps-override=30 --untimed --profile=low-latency http://host:1339/stream.h264
+    ffplay -f h264 -fflags nobuffer -flags low_delay http://host:1339/stream.h264
+
+Set `--container-fps-override` to the rate you are actually receiving, which is the `fps` parameter only while the game keeps up with it. Without any value ffmpeg assumes 25. Getting it wrong mostly affects mpv's reported position and its buffering estimates, since `--untimed` renders each frame as it decodes rather than scheduling against the timeline.
+
+## Client notes
+
+Both platforms take the same bytes from `stream.h264`; nothing is negotiated and the server does not care what the client is.
+
+### Android
+
+MediaCodec accepts annex-b directly, with the parameter sets either in band or as `csd-0` and `csd-1`. Call `releaseOutputBuffer(index, true)` to render on decode rather than scheduling against a clock.
+
+### iOS
+
+VideoToolbox has no annex-b entry point. The client has to split the stream on start codes, build a `CMVideoFormatDescription` from the SPS and PPS with `CMVideoFormatDescriptionCreateFromH264ParameterSets`, and replace each four byte start code with a four byte big endian length before wrapping the NALs in a `CMBlockBuffer`. Set `kCMSampleAttachmentKey_DisplayImmediately` on each sample buffer and leave the display layer's `controlTimebase` unset; without it you have rebuilt the presentation clock problem.
+
+`AVSampleBufferDisplayLayer` is much less work than a full `VTDecompressionSession` and is the better starting point for a test harness. The same API exists on macOS, so the decode path can be proven there before dealing with a device.
+
+A client cannot reach a plain HTTP address on the local network without an `NSAppTransportSecurity` exception, and iOS 14 and later also require `NSLocalNetworkUsageDescription` before it will connect to a LAN address at all. Without the latter the connection fails in a way that looks like the server is unreachable.
+
+### Flutter
+
+Decode on the platform side and render into a `Texture`. A platform channel hands the socket data to MediaCodec or VideoToolbox, and the decoded frames go back through a `SurfaceTexture` on Android or a `CVPixelBuffer` on iOS. Nothing then sits between the socket and the display, and both platforms keep their render-on-decode behaviour.
+
+## Limitations
+
+- One stream per screen, and four connections in total, so with four screens that is one each. Anything beyond either limit gets `503 Service Unavailable` and is closed immediately, so back off rather than retrying in a tight loop.
+- A client that vanishes without closing its socket keeps holding its screen until the send times out, which takes a few seconds. A reconnect arriving sooner than that is refused, so retry with backoff rather than assuming the screen is gone for good.
+- Each stream runs its own encoder on the machine hosting the game, and every captured frame costs that machine a present cycle. Streaming two screens at once roughly halves the rate each one can reach.
+- A client that stops reading is dropped after five seconds. The server never buffers a backlog for a slow reader; it skips to the newest frame instead, so frames are lost rather than delayed.
+- View only. Touch and other input still go through the JSON API, so a companion app needs both.
+- No authentication on the stream port. Anyone who can reach it can watch the screen.
+- WinXP builds have no video stream. Neither encoder is compiled in, so every endpoint returns 404, and the JSON API's JPEG screen capture is unavailable for the same reason.