The first time someone asks you to “add video to the app,” it sounds like an afternoon. Drop in a <video> tag, point src at a file, done. That version works right up until someone watches on a train, or on a TV, or on a connection that falls to nothing for four seconds, and then you find out what the job actually was.
I spent a stretch building and maintaining the web video player for a streaming product. The <video> tag turned out to be maybe five percent of it. The rest was everything that has to happen so that tag has something to play, on a network you don’t control, on devices you’ve never held. Here’s what that other ninety-five percent is actually like.
Media Source Extensions Make the Player a Download Scheduler
For anything past a single static file, you stop setting src and start using Media Source Extensions. Instead of handing the browser a video, you hand it a MediaSource, and then you feed bytes into SourceBuffer objects yourself, from JavaScript, while it plays.
That means the player is doing a surprising amount of work before a single frame shows up. It downloads a manifest that lists what’s available, works out which chunks of video exist at which qualities, requests them in the right order, appends them to a buffer, and keeps that buffer full enough to play smoothly without running so far ahead that it wastes the viewer’s bandwidth. It’s a little control loop reacting to the network in real time.
The mental shift that took me a while: you’re not playing a file. You’re running a download scheduler that happens to output video. Once that clicked, most of the rest of the system made sense, because almost every hard problem in the player is really a scheduling or estimation problem wearing a media costume.
The Network Is the Actual Product
You don’t control the viewer’s connection, so the player has to. This is adaptive bitrate streaming, and it’s the heart of the whole thing.
You encode the same video several times at different qualities, an encoding ladder from a tiny low-bitrate rendition up to the full-fat one, and you chop each into segments a few seconds long. The player measures how fast segments are arriving, picks a rung on the ladder that the connection can sustain, and switches rungs at segment boundaries as conditions change. Because the segments travel over plain HTTP, they also cache like any other static asset on a CDN, which is its own deep topic I got into in A Practical Caching Playbook.
The concept is simple. The judgment is not. Switch quality too eagerly and it flickers up and down, which viewers notice and dislike more than a steady, slightly-lower quality. Switch too late and the buffer drains and you stall, which is the one thing they truly hate. Every choice trades startup time against quality against rebuffering, and you cannot win all three at once. Worth knowing too: these HTTP-based streams aren’t really streams, they’re sequenced downloads, which is why their latency sits comfortably in the tens of seconds rather than anything close to real time.
Subtitles and Frame-Rate Drift
Subtitles look like the trivial part. They are not, and the reason is timing.
Browsers position everything in millisecond timestamps, but subtitle cues are often authored against frame numbers, so you convert using the frame rate. At 25 frames per second, frame 22 lands at 1000 / 25 * 22 = 880, so the cue shows up at 00:00:23.880. Fine, until you get the frame rate wrong. The standard is 25 fps across most of the world and 29.97 in the US and Japan, and if you assume the wrong one, every caption drifts. A little at the start, where nobody notices, then more, and because the error accumulates, by the end of a feature-length film the subtitles can be seconds out of sync with the dialogue.
No tutorial warns you about this. You learn it from a bug report that says “subtitles are fine at the start and broken by the end,” which sounds impossible until you understand it’s drift, not a fixed offset. That’s video in one story: the concepts are clean, and reality is a pile of accumulating edge cases you only meet by shipping.
One Player, Every Device
Then there’s the matrix. Apple’s platforms want HLS. Plenty of other devices speak MPEG-DASH. The manifest format differs, the container can differ, sometimes the codec does too, and both ends have to support the same protocol or nothing plays at all. A stream that’s flawless in one browser is a black rectangle in another.
Add DRM for licensed content and it gets worse, because now there’s a license server and a key exchange in the path before playback can even start, and the DRM systems themselves vary by platform. So the real shape of a “play this video” task is protocol times codec times DRM times device, and every cell in that grid has its own particular way of refusing to work. This is the point where video stops being a feature you complete and becomes a roadmap you maintain.
What I’d Tell Someone Starting Now
Don’t hand-roll the MediaSource and adaptive-bitrate machinery from scratch unless video genuinely is your product. The mature players, hls.js, dash.js, Shaka, Video.js, have absorbed years of exactly the edge cases above, including the subtitle-drift class of bug. Rolling your own mostly means rediscovering each of those the hard way, in production, from bug reports. The tech-lead call here is almost always integrate and extend, not build, and the engineers who insist on building usually haven’t yet met the device matrix.
And measure the things viewers actually feel. Startup time, rebuffer ratio, and how often the quality oscillates tell you far more than the peak resolution you can technically deliver. A viewer feels a four-second stall immediately and forgives 720p instantly. Optimizing the number that looks good in a deck while the rebuffer ratio quietly climbs is how you ship a player that demos well and frustrates everyone.
What the <video> Tag Actually Covers
The <video> tag is honest about exactly one thing: playing a file you already have, over a network that never hiccups, on a device you’re holding. Everything past that is a small distributed system pretending to be a media element, reacting in real time to a network and a fleet of devices you can’t see.
So when someone hands you “just add video,” the useful first question isn’t which library to use. It’s how many devices you’re targeting, how bad a connection you have to survive, whose content you’re carrying, and how you’ll know it’s stalling for someone you’ll never hear from. Answer those honestly and the afternoon task turns out to be a quarter. Better to find that out before you commit to the afternoon.