Recommended by Haowen:
Author: glumes
FFmpeg supports calling platform hardware for decoding after version 3.1, that is, MediaCodec on Android can be called through the C code of FFmpeg.
There are corresponding instructions on the official website, and the address is as follows:
trac.ffmpeg.org/wiki/HWAcce...
As can be seen from the figure, MediaCodec is supported not only on Android, but also VideoToolbox on iOS, and Direct3D 11 on Windows.
Note: Android MediaCodec currently only supports decoding, not coding.
However, in order to verify whether it is feasible, make a simple demonstration, and finally a complete code will be given.
The first is the compilation of FFmpeg. There are many switch options for its compilation. Make sure that mediacodec related options are turned on, as follows:
--enable-mediacodec --enable-decoder=h264_mediacodec --enable-decoder=hevc_mediacodec --enable-decoder=mpeg4_mediacodec --enable-hwaccel=h264_mediacodec
It can be seen that mediacodec supports three optional coding formats: h264, hevc and mpeg4. If it is not within the range, consider soft solution.
I won't elaborate on how to compile. I'll write a special article to introduce it later.
After compiling the corresponding so, you can print the list of formats supported by AVCodec to see if there is mediacodec.
The specific codes are as follows:
char info[40000] = {0};
AVCodec *c_temp = av_codec_next(NULL);
while (c_temp != NULL) {
if (c_temp->decode != NULL) {
sprintf(info, "%s[Dec]", info);
} else {
sprintf(info, "%s[Enc]", info);
}
switch (c_temp->type) {
case AVMEDIA_TYPE_VIDEO:
sprintf(info, "%s[Video]", info);
break;
case AVMEDIA_TYPE_AUDIO:
sprintf(info, "%s[Audio]", info);
break;
default:
sprintf(info, "%s[Other]", info);
break;
}
sprintf(info, "%s %10s\n", info, c_temp->name);
c_temp = c_temp->next;
}
Traverse through the next pointer of AVCodec, and then print the results. See the following content to show that the compilation is successful.
The supported formats already have h264_mediacodec and mpeg4_mediacodec.
The next step is decoding. The API call for FFmpeg decoding is repeated in public articles before official account.
-
First, use avformat_ open_ The input method opens the file to get the AVFormatContext.
-
Then through avformat_find_stream_info find the video stream information of the file.
-
To get file related information and video stream information, it is mainly to get coding format information, and then find the corresponding decoder. You can also use avcodec_ find_ decoder_ by_ The name method directly finds the specific decoder.
-
With the decoder, you can create the decoding context AVCodecContext and pass it through avcodec_open2 method to open the decoder
-
Then through av_read_frame reads the contents of the file for further decoding.
-
Next is the familiar avcodec_send_packet sent to decoder, avcodec_receive_frame retrieves the decoded data from the decoder.
Focus on the differences between calling hardware decoding and ordinary decoding:
The first step is to load the JNI in so_ Set the JavaVM to FFmpeg in the onload method.
jint JNI_OnLoad(JavaVM *vm, void *res) {
av_jni_set_java_vm(vm, 0);
return JNI_VERSION_1_4;
}
Without this step, you can't call Java methods by reflection.
Next, judge whether the hardware decoding type is supported. The above is determined by AVCodec. In fact, FFmpeg gives the definition of hardware type in the AVHWDeviceType enumeration variable.
enum AVHWDeviceType {
AV_HWDEVICE_TYPE_NONE,
AV_HWDEVICE_TYPE_VDPAU,
AV_HWDEVICE_TYPE_CUDA,
AV_HWDEVICE_TYPE_VAAPI,
AV_HWDEVICE_TYPE_DXVA2,
AV_HWDEVICE_TYPE_QSV,
AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
AV_HWDEVICE_TYPE_D3D11VA,
AV_HWDEVICE_TYPE_DRM,
AV_HWDEVICE_TYPE_OPENCL,
AV_HWDEVICE_TYPE_MEDIACODEC,
AV_HWDEVICE_TYPE_VULKAN,
};
Via AV_ hwdevice_ get_ type_ The name method can convert these enumerated values into corresponding strings, such as AV_ HWDEVICE_ TYPE_ The string corresponding to mediacodec is mediacodec, which is also available in the source code:
static const char *const hw_type_names[] = {
[AV_HWDEVICE_TYPE_CUDA] = "cuda",
[AV_HWDEVICE_TYPE_DRM] = "drm",
[AV_HWDEVICE_TYPE_DXVA2] = "dxva2",
[AV_HWDEVICE_TYPE_D3D11VA] = "d3d11va",
[AV_HWDEVICE_TYPE_OPENCL] = "opencl",
[AV_HWDEVICE_TYPE_QSV] = "qsv",
[AV_HWDEVICE_TYPE_VAAPI] = "vaapi",
[AV_HWDEVICE_TYPE_VDPAU] = "vdpau",
[AV_HWDEVICE_TYPE_VIDEOTOOLBOX] = "videotoolbox",
[AV_HWDEVICE_TYPE_MEDIACODEC] = "mediacodec",
[AV_HWDEVICE_TYPE_VULKAN] = "vulkan",
};
Like traversing AVCodec, you should also traverse whether FFmpeg supports mediacodec.
type = av_hwdevice_find_type_by_name(mediacodec);
if (type == AV_HWDEVICE_TYPE_NONE) {
LOGE("Device type %s is not supported.\n", mediacodec);
LOGE("Available device types:");
while((type = av_hwdevice_iterate_types(type)) != AV_HWDEVICE_TYPE_NONE)
LOGE(" %s", av_hwdevice_get_type_name(type));
LOGE("\n");
return -1;
}
Make sure that mediacodec is supported, then decoding can be used. As mentioned earlier, the purpose of obtaining file information is to open the decoder, but for example, for H.264 file encoding format, how to select mediacodec in addition to soft decoding for H.264 decoder?
For convenience, direct avcodec_find_decoder_by_name just find the decoder of mediacodec.
if (!(decoder = avcodec_find_decoder_by_name("h264_mediacodec"))) {
LOGE("avcodec_find_decoder_by_name failed.\n");
return -1;
}
After finding the decoder, you should also get some configuration information of the decoder, such as what the decoded format looks like? mediacodec decoding is NV21.
for (i = 0;; i++) {
// Decoder configuration
const AVCodecHWConfig *config = avcodec_get_hw_config(decoder, i);
if (!config) {
LOGE("Decoder %s does not support device type %s.\n",
decoder->name, av_hwdevice_get_type_name(type));
return -1;
}
if (config->methods & AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX &&
config->device_type == type) {
// Hard solution format
hw_pix_fmt = config->pix_fmt;
break;
}
}
At present, mediacodec decoding only has buffer mode, and there is no direct texture decoding.
The next step is to add some hardware decoding contexts to the decoding context AVCodecContext.
static int hw_decoder_init(AVCodecContext *ctx, const enum AVHWDeviceType type)
{
int err = 0;
if ((err = av_hwdevice_ctx_create(&hw_device_ctx, type,
NULL, NULL, 0)) < 0) {
LOGE("Failed to create specified HW device.\n");
return err;
}
// Hard decoded context
ctx->hw_device_ctx = av_buffer_ref(hw_device_ctx);
return err;
}
After completing this series of operations, the normal decoding is completed, and the decoded AVFrame content is obtained.
If the AVFrame format is the same as the configuration format of hardware decoding, use AV_ hwframe_ transfer_ The data method converts it into the normal YUV format.
if (frame->format == hw_pix_fmt) {
/* retrieve data from GPU to CPU */
if ((ret = av_hwframe_transfer_data(sw_frame, frame, 0)) < 0) {
LOGE("Error transferring the data to system memory\n");
goto fail;
}
tmp_frame = sw_frame;
} else
tmp_frame = frame;
After completing these operations, the decoding has been successful, and the actual operation is OK.