Context Aware Encoding: Building a Better Mousetrap

At Brightcove, we’re all here because we are passionate about video. We like watching it, we enjoy helping hundreds of organizations create it, and some of us are so driven as to really tinker and tweak, sometimes leading to re-inventing the core elements of video. It's this last piece that has massive impact on video technologies in general. It can be very exciting, but also super-nerdy and hard to explain sometimes.

Almost two years ago, video heads everywhere marveled at the research Netflix had just completed and applied to their entire library. Their Per-Title Encode Optimization approach, leveraged an analysis of each title to determine the best way to encode based on the complexity of the action in-frame. In simple terms, not all video is the same: sports is complex because it has lots of scene-to-scene motion, not just the players, but camera moves; episodic dramas may have much less scene-to-scene motion; and talking heads in news and current affairs programming generally have the least motion.

This motion relates to encoding complexity: the more you have, the more complex the video is to encode. Netflix’s approach was to look at each piece of content and determine how to encode it based on its inherent complexity, hence “per-title encode optimization.” For an organization like Netflix, which delivers more streams than anyone else on the planet, every bit of savings, every slight stream optimization can have massive return on investment.

As impressive as this innovation was, there are always technologists in this space who look at challenges from a different angle. This is how we’ve gone from postage stamp sized video to today’s HDR 4K, DVR enabled live experiences and beyond. It seems that no matter how impressive the next innovation is, there is another streaming technologist somewhere, thinking about how to build the proverbial better mousetrap.

In this case, Brightcove’s video research team, lead by Dr. Yuriy Reznik, came up with one of our newest video innovations at Brightcove called Context Aware Encoding (CAE). This takes the concept of content-based encoding optimization and augments it with additional information about network conditions and the device distribution amongst the audience. Think of Context Aware Encoding as having a compression expert in a box. For every title processed, and every frame therein, CAE is looking at the source asset and also making calculations about the target device and the network through which the stream will be delivered. Using this approach, CAE can optimize the encoding process so that multiple attributes of the bitrate ladder are adjusted, not just the bitrate, saving on renditions that aren’t needed. On average, this approach has shown savings on the order of one third across most content types, and as much as 50% where in-frame activity is fairly simple.

Of particular note recently, Jan Ozer, one of the video industry’s best known video experts put Brightcove’s Context Aware Encoding through its paces in a broad and subjective test.

In Jan’s words:

“Why was CAE so successful? Because with this low motion, synthetic video, it allowed Brightcove to deliver a higher resolution video to lower bitrate viewers than a traditional ladder. The result illustrates the key advantages of per-title encoding; happier viewers, lower bandwidth consumption, and lower storage and encoding costs from dropping from a seven rung ladder to four.”

This is great initial feedback for Context Aware Encoding and we’re only getting started.

As we work with more publishers, networks, and broadcasters and process more content, the algorithm will learn over time. Ultimately, this will mean that we are continuing to drive the costs out of bringing rich, compelling video experiences to audiences everywhere and transforming the media experience.