How can I embed user data (like a json fragment) into video frames using ffmpeg (libav*)?

Question

fmpeg

I have a task to be able to add user data to each frame. This is necessary to transfer various meta-data (for example, car numbers, rectangles indicating the position of detected objects on a video frame). I checked many different instructions, specs, interfaces, however, none of them work properly...

What I have:
- ffmpeg libraries built from source, latest version
- Encoding video via VAAPI using `av_hwframe_get_buffer`, `av_hwframe_transfer_data` , `avcodec_send_frame` and `avcodec_receive_packet`.
- I use h264, h265 and MJPEG codecs (the application has the ability to choose one of three codecs)
- I tried to plant metadata using the metadata field in AVFrame, it did not help, there are no signs of wedge-in data in the recorded file (MPEG-TS format), I checked it with a HEX editor
- I tried using SideData in AVPacket, nothing too ...

Maybe someone will tell you How can you implement the idea correctly? Even if it will be necessary to form a raw h264 / h265 package containing the specified metadata, are there working examples somewhere?

Answer 1

MP4 container can contain subtitle tracks. Maybe write them down?
For example, DJI does this in their drones: telemetry is written once per second in MP4 video. You can turn on English subtitles in the player and see shutter speed, ISO, GPS coordinates, etc.

screen