<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[mylathblog]]></title><description><![CDATA[mylathblog]]></description><link>https://mylathblog.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 14:08:45 GMT</lastBuildDate><atom:link href="https://mylathblog.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Why the shared screen stops on a slow link]]></title><description><![CDATA[The server was switching the screen off and on. Two fixes, each good for one case and bad for another.]]></description><link>https://mylathblog.hashnode.dev/why-the-shared-screen-stops-on-a-slow-link</link><guid isPermaLink="true">https://mylathblog.hashnode.dev/why-the-shared-screen-stops-on-a-slow-link</guid><category><![CDATA[WebRTC]]></category><category><![CDATA[mediasoup]]></category><category><![CDATA[debugging]]></category><dc:creator><![CDATA[ALLEN GEORGE]]></dc:creator><pubDate>Mon, 16 Nov 2026 16:16:43 GMT</pubDate><content:encoded><![CDATA[<p>This is about one bug. When a viewer has a slow link, the shared screen in our calls becomes useless: less than 1 frame per second, where it should be about 10. I spent a full day on it. I found the reason. I did not get a fix that is safe to switch on. Both things are written here.</p>
<h2>The problem</h2>
<p>Our test: one person shares a 1080p screen full of text. The viewer's link is limited to 1200 kbit/s. That is not a very bad link. A camera works fine on it (25 fps at 360p).</p>
<p>The screen gives 0.1 to 1.3 fps. The viewer sees a frozen page.</p>
<p>We send the screen in two sizes, 540p and 1080p. The small one needs about 300 to 600 kbit/s. So it should fit in 1200. But it does not work.</p>
<h2>First guess: make the small size smaller</h2>
<p>My first idea was simple. Text keyframes are big. Maybe the 540p keyframe is too big for the slow link. So send a smaller small size (360p), or send three sizes.</p>
<p>I ran each way four times.</p>
<table>
<thead>
<tr>
<th>Viewer's link</th>
<th>As today</th>
<th>Smaller small size</th>
<th>Three sizes</th>
</tr>
</thead>
<tbody><tr>
<td>1200 kbit/s</td>
<td>0.1 to 1.3 fps</td>
<td>1.0 to 3.2</td>
<td>1.9 to 3.8</td>
</tr>
<tr>
<td>2500 kbit/s</td>
<td>0.2 to 3.5</td>
<td>2.9 to 5.7</td>
<td>4.2 to 5.8</td>
</tr>
</tbody></table>
<p>A little better. Still not usable. So size was not the main problem.</p>
<h2>Second guess: the browser is not making enough frames</h2>
<p>In one run the viewer was on the 360p size for the whole minute, lost almost no packets, and still got only 3 fps. So I thought the browser makes the small size at a low frame rate when it is also busy with the big sizes.</p>
<p>Easy to check. I made the viewer's window small on a perfect link, so the server sends it the small size. Result: 9 fps. The browser is fine. Wrong guess.</p>
<h2>What is really happening</h2>
<p>I switched on the server's bandwidth logs. On the 1200 kbit/s link, the server believed the link could carry less than 300 kbit/s for about three quarters of the time. In the same seconds the link was actually delivering 1 Mbit/s.</p>
<p>Here is the chain.</p>
<ol>
<li>The server's estimator (it is Google's congestion control, the same one in Chrome) reacts to any small delay by cutting its estimate to 85% of what it was sending.</li>
<li>For a camera this is okay. A camera has lower frame rates to fall to. The lowest one needs only about 50 kbit/s. Something always fits.</li>
<li>A screen has nothing small to fall to. Its smallest size needs 300 kbit/s or more. 85% of that fits nothing. So the server sends <strong>no video at all</strong>.</li>
<li>After some seconds the estimate grows back. The server starts the screen again. A restart needs a keyframe. A keyframe of a text page is a big burst. The burst causes delay. Go to step 1.</li>
</ol>
<p>So the screen is not slow. It is being switched off and on, again and again.</p>
<h2>Fix one: do not switch off a picture that is already showing</h2>
<p>If nothing fits, keep sending the smallest size anyway.</p>
<table>
<thead>
<tr>
<th>Screen as today</th>
<th>Before</th>
<th>With fix one</th>
</tr>
</thead>
<tbody><tr>
<td>1200 kbit/s</td>
<td>0.1 to 1.3 fps</td>
<td>5.6 to 7.8 fps</td>
</tr>
<tr>
<td>2500 kbit/s</td>
<td>0.2 to 3.5 fps</td>
<td>4.7 to 6.1 fps</td>
</tr>
<tr>
<td>Cameras</td>
<td></td>
<td>no change</td>
</tr>
</tbody></table>
<p>Good. I was ready to ship it. Then I ran the remaining test cases, and one became worse.</p>
<table>
<thead>
<tr>
<th>Link drops to 600 kbit/s for 25 seconds, then becomes normal</th>
<th>Before</th>
<th>With fix one</th>
</tr>
</thead>
<tbody><tr>
<td>Frame rate over the 170 seconds</td>
<td>7.2 to 7.9 fps</td>
<td>3.6 and 4.0 fps</td>
</tr>
<tr>
<td>Time on full 1080p</td>
<td>about 155 s</td>
<td>about 40 s</td>
</tr>
</tbody></table>
<p>After the link became good again, the screen stayed on the small size for two minutes.</p>
<p>Why? The server finds out that a link has become better by sending short test bursts, called probes. But it only probes when it is sending much less than its estimate. The old behaviour (switch the video off) puts it in exactly that state, by accident, so it probes and recovers fast. With fix one the small size is always flowing, so the server never probes. And without a probe, the estimate is not allowed to grow beyond 1.5 times what is actually being delivered. From a 400 kbit/s small size it can never reach the 2 Mbit/s that 1080p needs.</p>
<p>So the bad behaviour was hiding a good side effect.</p>
<h2>Fix two: probe when the viewer wants more</h2>
<p>Make the server probe every 5 seconds whenever a viewer is getting less than half of what it wants.</p>
<table>
<thead>
<tr>
<th></th>
<th>Fix one alone</th>
<th>Fix one + fix two</th>
</tr>
</thead>
<tbody><tr>
<td>Dip for 25 s, then normal</td>
<td>3.6, 4.0 fps</td>
<td>8.1, 8.2 fps (better than the original)</td>
</tr>
<tr>
<td>Steady 1200 kbit/s</td>
<td>5.6 to 7.8 fps</td>
<td>3.7 to 5.3 fps</td>
</tr>
<tr>
<td>Audio packets lost on steady 1200 kbit/s</td>
<td>0.4 to 2%</td>
<td>5 to 11%</td>
</tr>
</tbody></table>
<p>The dip is cured. But on a link that stays slow, a probe every 5 seconds is just extra load. Audio packets started getting lost.</p>
<p>I tried one more rule: do not probe if the estimate was cut in the last 10 seconds. Then one dip out of two was not cured, and cameras on a 1200 kbit/s link went back to 720p at 20 fps, which we had fixed only one day before.</p>
<h2>What I shipped</h2>
<p>Nothing that changes behaviour. Both fixes are in the code, switched off. They can be switched on with an environment variable for testing.</p>
<p>I think this is the correct decision. Each fix helps one type of user and hurts another type. A brief dip is at least as common as a steady slow link. I should not trade one for the other without the owner seeing the numbers.</p>
<p>One more reason came at the very end. I built the exact code that is committed and ran it once more. With both fixes off it behaves like production, as it should. With fix one on, three runs gave 7 fps and the fourth gave 0 fps for the full minute. I do not know why yet. A fix with a bad run like that inside it should not be switched on.</p>
<p>What we have now that we did not have yesterday: we know exactly why the screen fails, and it is written down with numbers.</p>
<h2>What I will try next</h2>
<p>Both fixes are missing the same thing: a cheap way to learn that the link has become better.</p>
<ul>
<li>Probe at a smaller size. Today's probe is twice the estimate. It only needs to be as big as the next size the viewer wants.</li>
<li>Decide when to probe from the delay signal directly, not from the estimate.</li>
</ul>
<h2>Mistakes I made</h2>
<ul>
<li><strong>I guessed twice before I looked at the logs.</strong> Both guesses were reasonable and both were wrong. The logs showed the answer in ten minutes. I should have started there.</li>
<li><strong>I almost shipped after testing only the case I was fixing.</strong> The dip case was in the test suite all along. Run the full list before saying "it works".</li>
<li><strong>Three test runs silently failed</strong> because I killed the previous batch in a rough way and the next run could not start. For a while I was reading empty results as "no data yet". Now I check that a run actually produced rows.</li>
<li><strong>One finding from two days back was wrong.</strong> I had written "the server probes all the time". It does not. It probes only when it is sending well below its estimate. This matters for everything above.</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[What StreamScope is, and why it runs on a forked mediasoup]]></title><description><![CDATA[Rooms where people, drones, cameras, simulators and AI programs meet, and what an SFU is good for beyond video calls.]]></description><link>https://mylathblog.hashnode.dev/what-streamscope-is-and-why-it-runs-on-a-forked-mediasoup</link><guid isPermaLink="true">https://mylathblog.hashnode.dev/what-streamscope-is-and-why-it-runs-on-a-forked-mediasoup</guid><category><![CDATA[WebRTC]]></category><category><![CDATA[mediasoup]]></category><category><![CDATA[streaming]]></category><category><![CDATA[architecture]]></category><category><![CDATA[Rust]]></category><dc:creator><![CDATA[ALLEN GEORGE]]></dc:creator><pubDate>Sun, 15 Nov 2026 16:59:55 GMT</pubDate><content:encoded><![CDATA[<p><em>Engineering journal. Written for developers who know WebRTC and SFUs in general.</em></p>
<p>This is the entry to read first. The others in this journal are about single bugs and single tests; this one is about what the project is, why its media server is a fork of mediasoup, and what an SFU is good for once you stop thinking of it as "the thing behind a video call".</p>
<h2>What it is</h2>
<p>StreamScope started as a Zoom-like web app: log in, make a room, see each other. It still does that. But the rooms have turned into something else: a place where <strong>live streams of any kind meet the people and programs that need to watch them</strong>.</p>
<p>Things that sit in a room today:</p>
<ul>
<li>People, from a browser or a native desktop app, with camera, microphone and screen.</li>
<li><strong>Drones, ground vehicles and fixed cameras.</strong> A small agent on a companion computer (and, still in testing, on the radio controller itself) joins the room as a "node". It publishes the camera, and next to it the vehicle's telemetry, on the same connection.</li>
<li><strong>Cameras that only speak older protocols.</strong> RTSP cameras are pulled; encoders that push RTMP (a drone controller with one text field for a URL) or WHIP are taken in.</li>
<li><strong>Simulators.</strong> A physics simulation with a rendered camera runs on a cloud GPU and joins as a drone would, so the whole chain can be flown and tested with no aircraft.</li>
<li><strong>Programs that add something to a stream.</strong> A speech engine makes captions, translations and a dubbed voice from a member's audio. A vision model draws detections on a camera, or emits them as data.</li>
</ul>
<p>And things viewers do there besides watch: see the vehicle on a map from its telemetry, send it commands (point the camera, fly it from a cockpit view), read captions in another language, or listen to a speaker dubbed into their own.</p>
<p>So a "call" is one case. A pilot, two observers, a drone and an object detector in one room is the same machinery.</p>
<h2>Where it started: a managed SFU</h2>
<p>The first version used a managed SFU with a global anycast network, with a small Rust service for signalling. For plain calls that is hard to beat. You get a media server near every user without running one.</p>
<p>It stopped fitting as soon as the streams stopped being just calls:</p>
<ul>
<li><strong>Nothing can sit in the media path.</strong> Captions need the server to hand a member's audio to a program and take text back. A managed SFU forwards between browsers and that is all.</li>
<li><strong>No data alongside the media.</strong> Telemetry had to travel by a separate service and be matched up with the video afterwards. Two paths, two failure modes.</li>
<li><strong>No metering you control.</strong> Billing a customer for what a drone pushed in and what each viewer was sent needs per-stream counters from the thing that forwards the packets.</li>
<li><strong>Ingest was a bolt-on.</strong> Pulling an RTSP camera meant a separate worker re-publishing into the SFU as if it were a browser.</li>
</ul>
<h2>Why mediasoup</h2>
<p>The requirement was a media server we could change, embedded in a Rust process we already had. mediasoup fits that unusually well:</p>
<ul>
<li><strong>It is a library, not a product.</strong> There is a C++ worker that moves packets and a Rust crate that drives it. No signalling protocol, no rooms, no opinions. Our server links it and decides everything above the packet level.</li>
<li><strong>It already has side doors.</strong> <code>PlainTransport</code> sends and receives bare RTP, and <code>DirectTransport</code> delivers data to the host process. That is exactly what an external program needs in order to receive a stream and publish a result.</li>
<li><strong>Data channels are first-class.</strong> A <code>DataProducer</code> on the same transport as the video carries telemetry, so a vehicle's position arrives by the same path as its picture.</li>
<li><strong>Simulcast and per-consumer control.</strong> Each viewer can be sent a different size, paused, or re-prioritised by the server, which matters when one viewer has sixteen tiles and another has one.</li>
<li><strong>Real client libraries.</strong> The browser and the native apps use mediasoup's own client libraries, so we did not have to maintain SDP by hand for them.</li>
</ul>
<p>The cost of leaving a global network for one machine showed up at once. Measured side by side, camera-to-screen delay was about 23 ms on the managed SFU and about 95 ms on ours. Neither number is the SFU software. Ours was one server in another country.</p>
<p>Two changes closed it. The server moved to a data centre near the users, which cut the round trip from about 50 ms to about 20. And we stopped the receiver from holding video back to match the audio buffer, which the managed service had never done either. After both: <strong>21 to 25 ms camera to screen, against 17 to 19</strong>. Close enough that the features decide it.</p>
<h2>What the fork adds</h2>
<p>We vendored mediasoup at a pinned version and edited it, so these are concepts in the library and not conventions on top of it.</p>
<ul>
<li><strong>Source.</strong> One sender's ingest: its video, audio and telemetry, on one transport.</li>
<li><strong>Layer.</strong> One consumable stream of a source. Raw layers come from the sender. <strong>Derived layers</strong> (a transcript, a translation, a dubbed voice, processed video, detections) are made by a processor.</li>
<li><strong>Processor.</strong> An external program that declares which layers it can make. The SFU itself runs no models. Any speech engine or vision model plugs in the same way.</li>
<li><strong>Demand.</strong> A derived layer costs nothing until someone subscribes to it. The first subscriber causes the server to tap the input and tell the processor to start; the last one leaving stops it.</li>
<li><strong>Metering in the worker.</strong> The worker reports what each sender pushed in and what each viewer was actually sent, labelled by source, layer and party. Only the thing forwarding packets knows those numbers truthfully.</li>
<li><strong>Privacy per room.</strong> A room is either end-to-end encrypted (MLS), in which case the server cannot read it and no derived layers exist, or it is not, and processors may attach.</li>
</ul>
<p>Most of this needed no change to the packet path. Demand, for example, is built from consumers and pause, which mediasoup already had. The worker patches are narrow: the meter, and fixes such as the bandwidth-probe rule described in another entry.</p>
<h2>An SFU for more than calls</h2>
<p>If you are building something that is not a meeting, these are the things I would take from it.</p>
<p><strong>A room is a router, not a meeting.</strong> Members do not have to be people and do not have to send and receive symmetrically. A drone sends and never receives video. A detector receives one stream and sends one back. A simulator is a drone that does not exist.</p>
<p><strong>Put the data on the media connection.</strong> Telemetry as a data channel beside the video means one connection to establish, one to monitor, and timestamps from one clock. Commands back to the vehicle ride the same room. For telemetry the channel is unordered and never retransmits: a position that is late is worthless, and the next one is coming.</p>
<p><strong>Treat old protocols as ingest, not as peers.</strong> RTSP, RTMP and WHIP sources become ordinary sources at the door. Inside, everything is the same kind of stream, so a viewer cannot tell a browser from a twenty-year-old IP camera.</p>
<p><strong>Not every sender behaves like a browser.</strong> A hardware encoder may ignore keyframe requests and send one bitrate whatever the link does. The server cannot thin a single-layer stream, so a narrow viewer link stops it dead where a simulcast camera would have degraded. That is a design input, not a bug to fix later.</p>
<p><strong>Make expensive work demand-driven, and cap it.</strong> A transcript per member sounds harmless until there are fifty members and fifty speech sessions. Start work on the first subscriber, stop it on the last, and give the server a hard limit that no client can talk it out of.</p>
<p><strong>Delay is a product decision per stream.</strong> A pilot's video wants the lowest delay there is. A dubbed voice cannot exist until the sentence has been heard, translated and spoken, so that listener's picture is deliberately held back by several seconds to stay in step with the voice. Same room, same server, two very different targets.</p>
<h2>What it costs</h2>
<p>Running your own SFU means owning what a managed one hides:</p>
<ul>
<li><strong>Geography.</strong> One server is near some users and far from the rest. Rooms across several servers are built here and not yet switched on.</li>
<li><strong>Capacity planning.</strong> A worker is one thread. Measured on our hardware, a worker's edge is about 20 members who all publish, and rooms now spread over workers.</li>
<li><strong>The estimator's quirks are yours.</strong> A library tuned for browsers probing twice a call behaves differently in a server that probes all the time. We found that the hard way.</li>
<li><strong>Shared CPUs.</strong> Real-time media on a virtual machine whose host takes 30 to 40% of the processor at busy moments is audible. That one is not software.</li>
</ul>
<p>None of that argues against the choice. It is the price of being able to put a program in the media path, and for this project that ability is the product.</p>
]]></content:encoded></item></channel></rss>