Video Codecs: What They Are & the Best Formats for Streaming

Video Codecs: What They Are and the Best Formats for Streaming in 2024

Video codecs are to streaming what analytics tools are to big data: they convert RAW video into a compact format that can be easily shared with viewers. Just as analytics tools summarize large datasets, video codecs condense streams into their most essential components. It’s impossible to distribute video online without codecs — and even DVDs of yore use these compression tools.

As a portmanteau of “coder” and “decoder,” codecs accomplish two tasks. First, they encode videos using algorithms that shrink the data into a streamable size. Once the stream reaches viewers, the same codecs decompress the content for playback.

Remember when the iPod changed the music industry back in the early 2000s? This was all thanks to a codec. Specifically, Apple’s cutting-edge music player used the MP3 codec to pack giant music libraries into a handheld device.

Today, video developers use audio codecs like AAC and video codecs like H.264 to efficiently stream live and on-demand content across the world. These come into play when the content is encoded using software or hardware, transcoded by a platform like Mux, and once more when viewers hit play on their smartphones or connected TVs.

In this article, we explain what a video codec is, how these two-part compression tools work, the best video codecs for streaming, and considerations when comparing options. Check out the video below or navigate to the section of your choice using our table of contents.

What is a video codec?

Video codecs are algorithms that digitize and shrink down media files for transmission and storage. Literally ‘coder-decoder,’ they determine how the video is encoded and decoded — often discarding unnecessary information along the way. Common video codecs for streaming include H.264, H.265, and AV1.

The purpose of a codec is twofold:

• Convert RAW content into a digital format that can be streamed by viewers.
• Reduce the file size for efficient online distribution and cost-effective storage.

Video codecs vary in terms of playback compatibility, ideal use case, and costs. H.264, for instance, is the most widely supported codec — making it your best bet for reaching users across every device. Meanwhile, H.265 is a better option for 4K streaming due to its support for high-resolution video.

How do codecs work?

Video codecs employ different compression methods to fit the highest-quality video into the smallest file size possible. The encoding process includes three steps:

1. Identify patterns that can be used to reduce data.
2. Drop all information that will go unnoticed by viewers.
3. Quickly compress the remaining data.

Tactics like eliminating spatial and temporal redundancies, applying transformation techniques, and reducing image resolution all come into play.

What is video compression?

Video compression is the process of reducing video file size. You can think of codecs as the tools and video compression as the aim. The purpose of compression is to minimize the bandwidth required to deliver videos online.

Just as a ZIP file takes up less room on your computer than a bunch PNGs, a compressed video can be more easily shared across streaming platforms and cached on content delivery networks (CDNs). Methods for doing this include lossy, lossless, temporal, and spatial compression.

Lossy vs. lossless compression

Codecs come in two flavors: lossless and lossy. With lossless codecs, the entire file is reproduced upon playback. Lossy codecs, on the other hand, create a facsimile of the original file — and thus ensure much greater compression and more manageable file sizes.

Both lossy and lossless codecs are used to share and store digital files. Trusty ZIP files are an example of lossless data compression, whereas JPEGs are an example of lossy image compression. When it comes to streaming, though, lossy is the only way to go. Video files are simply too bulky to transmit without discarding some of the data.

The level of compression used is also defined by content distributors. Encoding settings that define the resolution, framerate, and keyframe interval all impact video quality. It’s often a matter of choosing what’s more important: superior video quality or efficient distribution.

It’s also worth noting that the information that video codecs discard isn’t always detectable by the human eye. That’s because lossy codecs identify patterns of repetition and only record information that’s unique or dynamic.

Temporal compression explained

Whenever content is streamed, video frames are broken into clusters called groups of pictures, or GOPs. The start of each GOP uses a keyframe that depicts the entire image of a video. Because stagnant backgrounds and graphical overlays don’t always change from frame to frame, the subsequent delta frames only include new information (like a newscaster’s hand gestures). In this way, video codecs discard unnecessary data by only recording differences between frames within a GOP.

Also called interframe compression, this is the reason why talking head videos can be streamed at a much higher quality than live sports matches using the same bitrate.

Spatial compression explained

With spatial compression, video codecs eliminate redundant information within each frame by duplicate pixels that appear in the same image. This is also called intraframe compression because it’s applied each individual frame that appears in a stream. When done right, it achieves significant bandwidth reduction without noticeable quality degradation.

Together, temporal and spatial compression remove the bulk of spare data in a stream — which is why all the technologies in our video codec list below employ these techniques.

Video codecs vs. video containers

It’s important to note that video codecs are not the same thing as video containers. Codecs are the software or hardware algorithms used to compress and decompress data; whereas video containers are the files that wrap all of this data into a single package.

Also called multimedia containers or file formats, these hold the audio codecs, video codecs, metadata, subtitles, and other components of a stream. Video containers also define the organization of this data — providing a framework for storing and playing the compressed video and audio.

Popular video file formats include:

• MP4 (MPEG-4 Part 14)
• fMP4 (fragmented MP4)
• TS segment (MPEG-2 Transport Stream)
• WebM (Matroska)
• MOV
• WMV

When streaming live content over the internet, TS segments and MP4s are the most common containers used. A single media container often holds multiple media streams, as well as data about synchronization, seeking, and more. This is called muxing — and its how Mux Video got its name.

The best video codecs for streaming in 2024

We looked at four points of comparison to compile our list of the best video codecs for streaming: playback compatibility, quality, use case, and royalties. Here’s how the algorithms stack up.

• H264/AVC
• H.265/HEVC
• H.266/VVC
• AV1
• VP8

H.264/AVC

According to Bitmovin’s Video Developer Report, H.264 is the most popular video codec out there — used by a whopping 85% of developers. And there’s good reason for this. H.264 is unparalleled in terms of device and browser reach. As such, you can trust that viewers will have no issues accessing H.264-encoded streams, regardless of whether they’re consuming the content on their phones or smart TVs.

Also called AVC (for Advanced Video Coding), H.264 was developed by the folks at the Motion Picture Experts Group (MPEG) and the International Telecomunications Unit (ITU). It has significant penetration across the video landscape, finding its way into streaming, cable broadcasting, Blu-ray disks, and DVDs.

H.264 codec is often combined with the AAC audio codec — which is how Mux Video outputs streams. In addition to being so well supported, it’s cheap to encode and super efficient for low-latency applications.

Pros

• Widely supported
• Affordable
• Efficient
• Royalties are clearly defined
• Great for low-latency streaming

Cons

• A legacy technology that shipped in 2003
• Not a great fit for 4K or HDR content

Container formats:

• MP4
• TS
• 3GP
• MOV
• F4

H.265/HEVC

H.265 is the successor to H.264, and a close second in terms of popularity. It was also developed jointly by MPEG and ITU, with an aim to add support for 8K resolution and improve compression efficience.

Also called High Efficiency Video Coding (HEVC), the codec delivered on these goals. It can encode 4K video in the same quality as H.264 at half the data rate. This requires more computational power and costs, but that’s the price you pay when encoding high-resolution videos.

Unfortunately, though, patents and royalty issues stalled adoption. Because content distributors were unclear on the fees and licensing structure, playback support also lagged until recently. Google recently added support for H.265 in Chrome, and many hardware encoders and decoders are also hopping on the H.265 train.

Pros

• Encodes 4K and HDR video
• Universally supported on Smart TVs
• Greater compression than H.264

Cons

• Complexity around patent licensing
• Limited browser support

Container formats

• MP4
• TS
• 3GP
• MKV

H.266/VVC

As one of the newest video codecs out there, H.266 shows potential to change the video compression game. The Fraunhofer Heinrich Hertz Institute designed H.264 to handle ultra-high-definition 8K video and immersive 360° experiences. The codec also delivers much greater compression than H.264 and H.265, thereby cutting down the costs of streaming high-resolution videos.

But because H.266 is just a baby, playback support across devices is in its early stages. You also need specialized software and hardware to encode H.266 video.

Pros

• Great for 8K and 360° experiences
• Unparalleled compression efficiency
• Supported by LG’s 8K TVs

Cons

• Limited playback and encoding support
• Some confusion around royalties

Container format

• MP4

AV1

As an open-source coded, AV1 is free to use. It was developed by the Alliance for Open Media, which is a group of media experts formed by Amazon, Netflix, Google, Microsoft, Cisco, and Mozilla. AV1 is roughly equivalent to H.266 in terms of encoding quality, but it’s quite computationally draining to use. This also equates to steep encoding costs.

Chip manufactures like Nvidia are beginning to add AV1 support to GPUs — with Apple recently adding support to the iPhone 15 Pro. This is big news that will drive adoption.

Pros

• 30% more efficient than H.265
• Excellent for 4K video
• From Apple to YouTube, vendors are rolling out support
• Open-source (which means no vendor lock in)
• Royalty-free

Cons

• Lacks support on Smart TVs
• Computationally demanding to encode

Container format

• MP4
• 3GP
• MKV

VP9

VP9 is a royalty-free, open-source video codec developed by Google. It delivers about the same quality as H.265 and is widely used for WebRTC streaming. While VP9-encoded video can be played back by most end-user devices, very few hardware-based encoders support it — making it a poor choice for live origination and transcoding.

Pros

• Supported by most browsers and end-user devices
• Open-source (which means no vendor lock in)
• Royalty-free

Cons

• Lacks hardware-based encoder support
• Computationally demanding to encode

Container format

• WEBM
• MKV

Considerations when comparing codecs

As seen in the video codec list above, you’ll want to consider the following criteria when comparing options:

• Playback compatibility. The first question and developer should ask is what’s the best codec for playback? All the compression efficiency in the world does little when your end user aren’t able to access the stream. That’s why Mux outputs video in H.264, the most widely supported option out there.
• Use case. Different codecs lend better to different use cases. So you’ll want to determine whether you’re looking to stream low-latency content, encode 8K video, or simply get your content to a video API like Mux as efficiently as possible.
• Quality. The goal of video compression is to eliminate as much data as possible without compromising quality. As such, you may be asking, what’s the best video codec for quality? According to video compression expert Jan Ozer, H.264 delivers the highest quality, which is one more reason why it’s at the top of our list.
• Costs. Video codecs impact costs in a number of ways. Royalties, the size of the files you’re streaming, the computational power required, and the need for specialty live streaming encoders should all be considered. If you’re looking to avoid licensing fees, AV1 is a great option. But because most end-user devices can’t decode it, you’ll have to pay transcoding costs to actually reach your viewers.

How to transcode any codec with Mux

Just because you choose a specific codec doesn’t mean you’re stuck with it. Most content distributors use a video platform like Mux to transcode their streams and optimize the content for delivery.

Mux Video ingests a broad range of codecs, including H.265, VP9, and even Apple ProRes. The Mux API then repackages the content — quickly and reliably — and seamlessly scales to audiences of any size.

Get started for free today or request a live demo.

Back to Articles