Khronos Finalizes Vulkan Video Extensions for Accelerated H.264 and H.265 Encode

December 19, 2023 by Lynne Iribarren, Khronos Member of the Vulkan Video TSG and Ahmed Abdelkhalek, AMD, Vulkan Video TSG Chair vulkan

In April 2021, the Vulkan® Working Group at Khronos® released a set of provisional extensions, collectively referred to as ‘Vulkan Video’ which provide seamless encoding and decoding of video streams using a variety of video coding standards. The December 2022 release of Vulkan 1.3.238 saw the finalization of the extensions to decode H.264 and H.265, and today, with the release of Vulkan 1.3.274, Khronos has finalized their counterpart: the extensions to enable encoding of H.264 and H.265 video streams. Leveraging the Vulkan framework, they provide a standardized, seamless, low-overhead, and highly controllable way to produce H.264 and H.265 video via hardware accelerators, with applications ranging from real-time, low-latency streaming to offline server-scale transcoding.

Incorporating industry feedback, the extensions saw many improvements since their introduction, from a bidirectional interface (overrides) to help with coding and exposing advanced hardware capabilities, to rate control configuration parameters and an interface to aid with quality vs. performance trade-offs. This feedback also prompted the release of the first video maintenance extension. In addition, given the high industry demand for AV1 codec support, an AV1 decode extension release is imminent, with an AV1 encode extension development also underway. Figure 1 depicts Vulkan Video extensions along with their status and relations.

The encode extensions grant low-level control over much of the encoding process, while still keeping the efficiency and performance of hardware encoding acceleration. Implementers have the freedom to tweak details such as quantization index, per-slice bit allocation, arithmetic coder, deblocking, and more. Given this flexibility and complexity, a balanced programming interface for rate control gives users a choice between more automated operation and low-level tweaking of frame parameters.

So, What Changed?

This section briefly summarizes the changes included in this release relative to prior descriptions of the Vulkan Video extensions.

Changes to Video Encode

Encoder Rate Control

Often the most important aspect of encoder configuration for applications, the encoder rate control API was given special attention in Vulkan Video. From exposing parameters for standard rate control modes (e.g. CBR/VBR), to allowing applications to provide hints about other intended stream encoding parameters (e.g. picture/reference patterns), to providing the ability to configure per-layer rate control parameters (e.g. for streams with multiple temporal layers), the rate control API offers a rich set of features for various use cases and lays a solid foundation for future extensions. Encoder rate control configuration is performed using the vkCmdControlVideoCodingKHR command.

Encoder Quality Levels

Video encoder implementations often fine tune the use of various encoding tools and rate control parameters depending on the desired quality versus performance/latency trade-offs of different use cases. Now implementations report the number of quality levels supported for a given video profile and usage. A new API vkGetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR may be used to retrieve implementation recommendations for various encoding parameters and configurations (e.g. rate control).

Implementation Overrides

Due to the complex nature of video encoding, and the ever-changing nature of hardware encoders and their capabilities, an interface, known as overrides, permits bidirectional communication that guarantees that the output video stream will be compliant. In addition, applications may opt-in for optimization overrides to allow implementations more flexibility to optimize for the specified usage and hints. Full disclosure about the occurrence of overrides for video session parameters or frame parameters is also reported for developers interested in more detailed analysis of such overrides.

Retrieval of encoded video session parameters bitstream segments

To facilitate implementation overrides for bitstream compliance and optimizations, applications are expected to retrieve the encoded video session parameter bitstream segments (e.g. H.264 SPS/PPS) from the implementation using the new API call vkGetEncodedVideoSessionParametersKHR against the given VkVideoSessionParametersKHR object.

Encoder Feedback Query

To allow future extension of encoder feedback statistics in a manner similar to pipeline statistics, the new VK_QUERY_TYPE_VIDEO_ENCODE_FEEDBACK_KHR is now used to retrieve the video bitstream offset and size.

Figure 2 depicts the Vulkan Video encoding process, which remains largely unchanged compared to previous descriptions.

Changes to Video Decode & Encode

VK_KHR_video_maintenance1

Along with the video encoding extensions, Khronos is releasing a maintenance extension incorporating community and industry feedback, which improves flexibility for both decoding and encoding. This extension permits decoding implementations to create images usable with video decoding without the need to explicitly specify the video profiles they will be used with. The same applies for encoding, where an attached per-image video profile limits usability with large and complex transcoding frameworks.

In addition to flexibility improvements, a new, simpler interface for specifying video queries inline with video decode and encode operation commands has been added, known as inline queries.

Requiring pSetupReferenceSlotKHR for non-reference pictures

When the Vulkan Video decode extensions were finalized applications were required to provide a reconstructed picture resource and DPB slot (via VkVideoDecodeInfoKHR::pSetupReferenceSlot) only if the picture being decoded will become a reference. However, no shipping implementation actually supported specifying NULLfor pSetupReferenceSlot, and further some implementations discovered cases that require the use of the reconstructed picture resource and/or DPB slot for transient storage during decoding a non-reference picture. A similar situation applies to encoding non-reference pictures. As a result, the vulkan video extensions were updated to require providing pSetupReferenceSlotKHR for non-reference pictures.

I Want All The Details!

The following proposal documents provide a much more detailed and thorough review of all finalized Vulkan Video extensions, including the rationale behind the chosen design approach, various issues encountered during the development of Vulkan Video and how they were resolved, as well as code samples to aid in application development:

Developers interested in using Vulkan Video are strongly encouraged to read the above proposal documents - they really give you a good head start!

All details about the Vulkan Video API are available in the final extensions specification links:

Call for Action, Feedback & Support!

The finalization and release of these Vulkan Video extensions marks a significant milestone in the Vulkan ecosystem roadmap, adding fully accelerated H.264 and H.265 encode to this widely available cross-platform GPU API. We encourage developers to utilize these extensions to bring new levels of performance and functionality to their video applications on Windows and Linux.

We welcome you to join Vulkanised 2024 (February 5-7 in MountainView CA.), which will include a presentation & live demo about Vulkan Video and much more!

NVIDIA, Intel & AMD are the first IHVs implementing support for these extensions:

NVIDIA: Windows and Linux BETA drivers available now
Intel: Coming soon
AMD: Coming soon

Additionally, the open-source drivers RADV and ANV are adding support for the finalized Vulkan Video encode extensions, in addition to already supporting the Vulkan Video decode extensions.

Strong ecosystem adoption continues for Vulkan Video, and there are several implementers of the spec:

The reference app part of the Vulkan Video samples repository with encoding support is coming soon in 1Q24.
FFmpeg, in a branch currently under development, soon to be merged into the main repository.
Gstreamer, in a branch currently under development and a description of the journey can be found here.

Those interested in testing the code should follow their guidelines and are welcome to submit feedback.

An upcoming release of Vulkan SDK will include updated Vulkan headers and Validation Layer support for the newly released video extensions. In the meantime, you can find the Vulkan headers here.

We look forward to your use and feedback on Vulkan Video! Please share your experience and thoughts through the Vulkan Video Encode Release GitHub Issue. This issue will also be updated to provide links to Vulkan Video-related resources as they become available.

We also encourage your participation in extending Vulkan Video to support more codecs and features. See khronos.org/members for information about how to join Khronos and participate in the definition of any of our standards.

Thank you for your interest and support of Vulkan Video. We hope you find it effective for your use cases and applications, and we look forward to supporting your needs with more codecs and features!