diff --git a/Video-Stream.md b/Video-Stream.md new file mode 100644 index 0000000..e04ff6c --- /dev/null +++ b/Video-Stream.md @@ -0,0 +1,98 @@ +# Video Stream + +Guidance for building an app against the `-apistream` video stream: which format to use, what the parameters mean, and what the client is responsible for. The endpoint list and query parameters are in the main README. + +## Connecting + +Open a plain HTTP GET and read the body until you close it or the server does. There is no handshake and no authentication. + +A stream always begins at a keyframe, so there is nothing to seek and no startup state to recover. + +`screen`, `fps` and `q` are fixed for the life of a connection. Changing any of them means opening a new one. + +A screen carries one stream at a time. A second connection to a screen that is already streaming is refused with `503 Service Unavailable`, and nothing is ever preempted, so an existing viewer cannot be interrupted by someone else connecting. + +`fps` is a ceiling rather than a promise. Nothing competes with you for frames on your own screen, but you receive the lesser of your requested rate and whatever the game is actually redrawing, so a request for 30 on a screen that only updates 21 times a second delivers 21. Derive timing from arrival, not from the value you asked for. + +The server ends the connection if the game changes resolution. Treat a closed connection as routine and reconnect rather than surfacing an error. + +## Choosing a format + +| | `stream.mjpg` | `stream.h264` | +| --- | --- | --- | +| Container | multipart/x-mixed-replace | none | +| Bandwidth | ~20x higher | low | +| Plays in a normal player | yes | some do, MPC and mpv both decode it | +| Works in a browser | yes, in an `` tag | no | +| Client work | none | drives a decoder itself | + +`stream.h264` is the default choice for an app. H.264 costs roughly twenty times less bandwidth than MJPEG and phones decode it in hardware, which matters a great deal for battery life. It is a bare elementary stream, so the app feeds the bytes to MediaCodec or VideoToolbox itself and nothing sits between the socket and the decoder. + +`stream.mjpg` needs no container support at all and every frame stands alone, so it is the fallback when nothing else works, and the only option inside a browser. It is expensive on bandwidth and on client CPU. + +## Quality + +`q` means different things per format, despite the shared name and default: + +- `stream.mjpg` passes it straight through as the standard 1-100 JPEG quality. +- `stream.h264` maps it onto the H.264 rate factor as `CRF = 40 - q / 4`. + +So for H.264: + +| `q` | CRF | | +| --- | --- | --- | +| 1 | 39.75 | badly blocky | +| 40 | 30 | visibly lossy | +| 70 | 22.5 | default, close to x264's own default of 23 | +| 100 | 15 | high quality | + +CRF is constant quality, not constant bitrate. `q` sets how much loss the encoder tolerates; the resulting bitrate floats with how much the screen is moving. + +## Latency + +The server puts a frame on the socket about 5 ms after capturing it, so nearly all of the delay a user sees comes from the client. + +The usual cause is a player that buffers ahead and then runs its own clock; one that decides during startup that it is half a second behind will stay half a second behind indefinitely. Whatever you use, find the option that renders on decode instead of scheduling against a presentation clock. In mpv: + + mpv --demuxer-lavf-format=h264 --no-correct-pts --container-fps-override=30 --untimed --profile=low-latency http://host:1339/stream.h264 + +The equivalents are `releaseOutputBuffer(index, true)` on MediaCodec and `kCMSampleAttachmentKey_DisplayImmediately` on VideoToolbox. An annex-b stream carries no timestamps of its own, so nothing downstream has a clock to fall behind on in the first place. + +## Testing + +`stream.h264` has no container and no timestamps. Some players cope on their own - MPC decodes it as served - while others need the format and frame rate supplied. Either way this is the cheapest way to check the stream is sane: + + mpv --demuxer-lavf-format=h264 --no-correct-pts --container-fps-override=30 --untimed --profile=low-latency http://host:1339/stream.h264 + ffplay -f h264 -fflags nobuffer -flags low_delay http://host:1339/stream.h264 + +Set `--container-fps-override` to the rate you are actually receiving, which is the `fps` parameter only while the game keeps up with it. Without any value ffmpeg assumes 25. Getting it wrong mostly affects mpv's reported position and its buffering estimates, since `--untimed` renders each frame as it decodes rather than scheduling against the timeline. + +## Client notes + +Both platforms take the same bytes from `stream.h264`; nothing is negotiated and the server does not care what the client is. + +### Android + +MediaCodec accepts annex-b directly, with the parameter sets either in band or as `csd-0` and `csd-1`. Call `releaseOutputBuffer(index, true)` to render on decode rather than scheduling against a clock. + +### iOS + +VideoToolbox has no annex-b entry point. The client has to split the stream on start codes, build a `CMVideoFormatDescription` from the SPS and PPS with `CMVideoFormatDescriptionCreateFromH264ParameterSets`, and replace each four byte start code with a four byte big endian length before wrapping the NALs in a `CMBlockBuffer`. Set `kCMSampleAttachmentKey_DisplayImmediately` on each sample buffer and leave the display layer's `controlTimebase` unset; without it you have rebuilt the presentation clock problem. + +`AVSampleBufferDisplayLayer` is much less work than a full `VTDecompressionSession` and is the better starting point for a test harness. The same API exists on macOS, so the decode path can be proven there before dealing with a device. + +A client cannot reach a plain HTTP address on the local network without an `NSAppTransportSecurity` exception, and iOS 14 and later also require `NSLocalNetworkUsageDescription` before it will connect to a LAN address at all. Without the latter the connection fails in a way that looks like the server is unreachable. + +### Flutter + +Decode on the platform side and render into a `Texture`. A platform channel hands the socket data to MediaCodec or VideoToolbox, and the decoded frames go back through a `SurfaceTexture` on Android or a `CVPixelBuffer` on iOS. Nothing then sits between the socket and the display, and both platforms keep their render-on-decode behaviour. + +## Limitations + +- One stream per screen, and four connections in total, so with four screens that is one each. Anything beyond either limit gets `503 Service Unavailable` and is closed immediately, so back off rather than retrying in a tight loop. +- A client that vanishes without closing its socket keeps holding its screen until the send times out, which takes a few seconds. A reconnect arriving sooner than that is refused, so retry with backoff rather than assuming the screen is gone for good. +- Each stream runs its own encoder on the machine hosting the game, and every captured frame costs that machine a present cycle. Streaming two screens at once roughly halves the rate each one can reach. +- A client that stops reading is dropped after five seconds. The server never buffers a backlog for a slow reader; it skips to the newest frame instead, so frames are lost rather than delayed. +- View only. Touch and other input still go through the JSON API, so a companion app needs both. +- No authentication on the stream port. Anyone who can reach it can watch the screen. +- WinXP builds have no video stream. Neither encoder is compiled in, so every endpoint returns 404, and the JSON API's JPEG screen capture is unavailable for the same reason.