Skip to main content

ffmpeg-slow-processing-problem

How to solve FFmpeg video editing recording delay, video length reduction, CPU 100% problem

While editing (recording) video using FFmpeg on a real-time video recording server, a recording delay issue occurred. Videos recorded at the same time were shorter on some computers. Let's take a look at the situation in which the issue occurred, the cause of it, and the process of solving it.

Problem situation​

  • There are several servers for recording real-time video.
    • Spring Boot & Kotlin based server in operation
  • Each server performs the function of recording input video files using FFmpeg
  • The specifications of each server are slightly different for each server (do not use the same hardware or the same instance)
Info

FFmpeg is an open source project developed under the leadership of Michael Niedermayer with the goal of decoding and encoding all video, music, and photo formats. In detail, it is a program that plays the following roles.

  1. Open Source: FFmpeg is an open source video and audio processing tool, free for anyone to use and modify.
  2. Multiplatform: FFmpeg runs on Windows, macOS, Linux and many other operating systems.
  3. Support for various formats: FFmpeg supports hundreds of formats. It enables encoding, decoding, conversion and streaming of video, audio, images and other multimedia data.
  4. Video and audio encoding/decoding: FFmpeg supports almost all major video and audio codecs. This means that users can encode and decode video and audio files in a variety of formats.
  5. Streaming: FFmpeg provides the ability to stream video and audio in real time. This can be used to build media servers or serve content to online streaming platforms.
  6. Filters and Effects: FFmpeg provides a variety of video and audio filters to help you edit and enhance your content. For example, you can resize, crop, rotate, color correct, and remove noise.
  7. Command Line Interface: FFmpeg is used through a command line interface. It allows you to perform a variety of multimedia tasks using simple commands and is ideal for automation and batch processing.
  8. Library form: FFmpeg is also available in library form, allowing other applications to use its multimedia features. It can be used in video editing software, media players, streaming servers, etc.

problem​

  • In certain servers, video encoding appears shorter than the recorded length.
    • Check various causes to check the problem and check server specifications and system resource usage such as CPU, Graphics, and Memory
  • Confirm that the system resource usage rate is different between the server in the problem environment and the server in normal operation.
    • Normal: CPU usage does not change significantly, and graphics card usage increases from 0% to 30-40%.
    • Abnormal: CPU is continuously being used at 100%, and graphics card usage does not increase.
  • When comparing each environment, there are differences as follows
    • Normal: Environment equipped with Nvidia's external Graphic Card. Existing developments can also be checked in this environment
    • Abnormal: Server equipped with only Intel built-in graphics
  • In a system that processes video 100% with the CPU without using the graphics card for a certain period of time, it is assumed that the video is shortened because processing cannot be completed within the specified time. **
    • In the code, the recording function is set to operate for a certain period of time.

Resolution​

  • I looked for a way to use Intel's built-in graphics.
    • I didn't know that Intel's built-in graphics had very good performance.
    • However, I thought it would work much more efficiently if I divided the work rather than using 100% of the CPU.

ffmpeg -codecs


![image](/img/2024/2024-04-03-13-22-53.png)

* Using the above command, you can check the available codecs of the currently installed ffmpeg as shown above.
* When recording, each server used the H.264 codec, and models equipped with an Nvidia Graphic Card used the `h264_nvenc` encoder.
* Models equipped with only Intel integrated graphics cannot use `h264_nvenc`. I used `libx264` or `libopenh264` encoder, which were SW-based (using CPU) methods.
* At this time, when `h264_qsv` was used, **CPU usage did not increase by even 10%, and the usage rate of previously unused built-in graphics increased by about 30%**. **Recording time has not been reduced and is now the normal length.**
  * However, because the encoder was different, the quality of the two videos with different graphics cards seemed somewhat different.
* I gained knowledge that normal SW operation can be expected only when the HW characteristics of each server are well utilized, and that even in cases where the HW is different, it is necessary to check in advance.

:::info
H.264 is a video standard codec announced in 2003. It is a more advanced video coding than the existing MPEG-4 Part 2, so it is also known as AVC (Advanced Video Coding). Not only does it have a good compression rate, but it also has good picture quality even at a relatively low bit rate, so it is used in many services.
:::