ffplay -- Decoding thread analysis

The decoding thread of ffplay is independent of the reading thread, and each type of stream (AVStream) has its own decoding thread

The PacketQueue is used to store avpackets within the respective playback time taken from read \ u thread.
FrameQueue is used to store each decoded AVFrame.
Clock is used to synchronize audio and video.

The decoding thread is responsible for decoding the PacketQueue data into the AVFrame and storing it into the FrameQueue.

For different streams, the decoding process is the same and the same. Let's take a look at video ﹐ thead first.

For the filter part, no analysis will be done first, and the simplified code is as follows (for the convenience of reading, the following code also omits the declaration of some local variables):

static int video_thread(void *arg)
{
    AVRational tb = is->video_st->time_base;
    AVRational frame_rate = av_guess_frame_rate(is->ic, is->video_st, NULL);for (;;) {
        ret = get_video_frame(is, frame); //Decode to get a frame of video
        if (ret < 0)//End of decoding
            goto the_end;
        if (!ret)//Not decoded to get the picture
            continue;
​
        duration = (frame_rate.num && frame_rate.den ? av_q2d((AVRational){frame_rate.den, frame_rate.num}) : 0);//Estimating frame length with frame rate
        pts = (frame->pts == AV_NOPTS_VALUE) ? NAN : frame->pts * av_q2d(tb);//Convert pts to seconds
        ret = queue_picture(is, frame, pts, duration, frame->pkt_pos, is->viddec.pkt_serial);//Store the decoded frame in FrameQueue
        av_frame_unref(frame);if (ret < 0)
            goto the_end;
    }
 the_end:av_frame_free(&frame);
    return 0;
}

The overall process of the thread is very clear:

Call get video frame to decode a frame of image
"Calculation" duration and pts
Call queue? Picture and put it in FrameQueue

Get video frame is simplified as follows:

static int get_video_frame(VideoState *is, AVFrame *frame)
{
    int got_picture;
    // Decoding video frames
    if ((got_picture = decoder_decode_frame(&is->viddec, frame, NULL)) < 0)
        return -1;
    // Judge whether decoding is successful
    if (got_picture) {
        double dpts = NAN;

        if (frame->pts != AV_NOPTS_VALUE)
            dpts = av_q2d(is->video_st->time_base) * frame->pts;

        frame->sample_aspect_ratio = av_guess_sample_aspect_ratio(is->ic, is->video_st, frame);
        // Determine whether the frame needs to be discarded
        //Frame drop is the switch variable that controls whether frames are lost. If it is 1, it is always judged whether frames are lost;
        //If it is 0, the frame will never be lost;
        //If it is - 1 (the default), it will determine whether to lose the frame when the master clock is not video.
        if (framedrop>0 || (framedrop && get_master_sync_type(is) != AV_SYNC_VIDEO_MASTER)) {
            if (frame->pts != AV_NOPTS_VALUE) {
                double diff = dpts - get_master_clock(is);
                if (!isnan(diff) && fabs(diff) < AV_NOSYNC_THRESHOLD &&
                    diff - is->frame_last_filter_delay < 0 &&
                    is->viddec.pkt_serial == is->vidclk.serial &&
                    is->videoq.nb_packets) {
                    is->frame_drops_early++;
                    av_frame_unref(frame);
                    got_picture = 0;
                }
            }
        }
    }

    return got_picture;
}

It is mainly called to decode.

The simplified part is mainly a process for frame loss:

The main condition for frame loss is that diff - is - > frame > last Filter > delay < 0. Frame > last Filter > delay is related to the filter, which can be ignored first, that is, frame loss when diff < 0 - frame loss when PTS < get master > clock (is).

The process of real decoding -- decoder ﹣ decode ﹣ frame,

This function also includes the decoding of audio and subtitle. Similarly, first look at the simplified trunk Code:

static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) {
    for (;;) {
        //1. When the stream is continuous, call avcodec ﹣ receive ﹣ frame to obtain the decoded frame
        if (d->queue->serial == d->pkt_serial) {
            do {
                ret = avcodec_receive_frame(d->avctx, frame);
                if (ret == AVERROR_EOF) {
                    return 0;
                }
                if (ret >= 0)
                    return 1;
            } while (ret != AVERROR(EAGAIN));
        }//2. Take a packet and filter "out of date" packets
        do {
            if (d->queue->nb_packets == 0)
                SDL_CondSignal(d->empty_queue_cond);
            if (packet_queue_get(d->queue, &pkt, 1, &d->pkt_serial) < 0)
                return -1;
        } while (d->queue->serial != d->pkt_serial);//3. Send the packet to the decoder
        avcodec_send_packet(d->avctx, &pkt);
    }
}

The trunk code of the decoder ﹣ decode ﹣ frame is a cycle. Only when you get a frame of decoded data, or the decoding error, or the end of the file, can you return it.

The cycle can be divided into three steps:

1. When the stream is continuous, the avcodec ﹣ receive ﹣ frame is called continuously to obtain the decoded frame. D - > queue is video
PacketQueue(videoq), D - > pkt_serial is the serial number of the latest Packet. After judgment, D - > queue - > serial
==D - > pkt ﹣ serial to ensure that the flow is continuous, call avcodec ﹣ receive ﹣ Frame in a circular way, and return if there is a Frame. (even if a new Packet has not been sent, this is to be compatible with the situation that one Packet can solve multiple frames)
2. Take a packet and filter "out of date" packets. The main blocking call is packet queue get (refer to the analysis of packet queue: https://zhuanlan.zhihu.com/p/43295650 ) In addition, when PacketQueue is empty, an empty queue cond condition signal will be sent to inform the reader thread to continue reading data. (empty ﹣ queue ﹣ cond is continue ﹣ read ﹣ thread. You can refer to the analysis of read thread to see when the read thread will wait for the condition quantity: https://zhuanlan.zhihu.com/p/43672062)
3. Send the packet to the decoder.

Avcodec? Decode? Video2 has been marked as obsolete. @deprecated Use avcodec_send_packet()
It is recommended to use these two functions for decoding.

In the omitted code, there is a concept of "packet" pending, which is used to resend in case of send failure:

if (avcodec_send_packet(d->avctx, &pkt) == AVERROR(EAGAIN)) {
    av_log(d->avctx, AV_LOG_ERROR, "Receive_frame and send_packet both returned EAGAIN, which is an API violation.\n");
    d->packet_pending = 1;
    av_packet_move_ref(&d->pkt, &pkt);
}

If avcodec send packet returns EAGAIN, the current pkt is stored in D - > pkt, and then the flag packet pending is set to 1.

When the Packet is fetched, the read of Packet pending:

do {
    if (d->queue->nb_packets == 0)
        SDL_CondSignal(d->empty_queue_cond);
    //If the pkt to be retransmitted, the pkt to be retransmitted will be taken first, otherwise a pkt will be taken from the queue
    if (d->packet_pending) {
        av_packet_move_ref(&pkt, &d->pkt);
        d->packet_pending = 0;
    } else {
        if (packet_queue_get(d->queue, &pkt, 1, &d->pkt_serial) < 0)
            return -1;
    }
} while (d->queue->serial != d->pkt_serial);

In the omitted code, there is also a processing for flush﹐pkt:

if (pkt.data == flush_pkt.data) {
avcodec_flush_buffers(d->avctx);
d->finished = 0;
d->next_pts = d->start_pts;
d->next_pts_tb = d->start_pts_tb;
}
After knowing the code of PacketQueue, we know that after sending a flush pkt to PacketQueue, the serial value of PacketQueue will be increased by 1, and the serial value of the incoming flush pkt and PacketQueue will be the same. So if there is an "out of date" Packet, the first pkt retrieved after filtering will be flush pkt.

According to the api requirements, you need to call avcodec? Flush? Buffers at this time.

Above, we have analyzed the key get video frame function in video thread. According to the analyzed code, we can get a frame of data after the correct decoding. Next, put this frame into FrameQueue:

duration = (frame_rate.num && frame_rate.den ? av_q2d((AVRational){frame_rate.den, frame_rate.num}) : 0);
pts = (frame->pts == AV_NOPTS_VALUE) ? NAN : frame->pts * av_q2d(tb);
ret = queue_picture(is, frame, pts, duration, frame->pkt_pos, is->viddec.pkt_serial);
av_frame_unref(frame);

The main call is queue? Picture:

static int queue_picture(VideoState *is, AVFrame *src_frame, double pts, double duration, int64_t pos, int serial)
{
    Frame *vp;if (!(vp = frame_queue_peek_writable(&is->pictq)))  Get the node to be written from the queue vp
        return -1;
​
    vp->sar = src_frame->sample_aspect_ratio;
    vp->uploaded = 0;
​
    vp->width = src_frame->width;
    vp->height = src_frame->height;
    vp->format = src_frame->format;
​
    vp->pts = pts;
    vp->duration = duration;
    vp->pos = pos;
    vp->serial = serial;set_default_window_size(vp->width, vp->height, vp->sar);av_frame_move_ref(vp->frame, src_frame);  	Will be decoded src_frame Copy to vp->frame   Post copy src_frame invalid
    frame_queue_push(&is->pictq);   			To update windex and size++
    return 0;
}

So far, the video decoding process, except for the fitler part, has basically been analyzed.

The decoding process of audio, regardless of the filter part, is almost the same as that of video. We will not repeat the analysis.

The decoding process of subtitle is slightly different, focusing on different places.

static int subtitle_thread(void *arg)
{
    VideoState *is = arg;
    Frame *sp;
    int got_subtitle;
    double pts;for (;;) {
        //Note that here is frame ﹐ queue ﹐ peek ﹐ writable and then decoder ﹐ decode ﹐ frame
        if (!(sp = frame_queue_peek_writable(&is->subpq)))
            return 0;if ((got_subtitle = decoder_decode_frame(&is->subdec, NULL, &sp->sub)) < 0)
            break;
​
        pts = 0;if (got_subtitle && sp->sub.format == 0) {
            if (sp->sub.pts != AV_NOPTS_VALUE)
                pts = sp->sub.pts / (double)AV_TIME_BASE;
            sp->pts = pts;
            sp->serial = is->subdec.pkt_serial;
            sp->width = is->subdec.avctx->width;
            sp->height = is->subdec.avctx->height;
            sp->uploaded = 0;/* now we can update the picture count */
            frame_queue_push(&is->subpq);
        } else if (got_subtitle) {
            avsubtitle_free(&sp->sub);
        }
    }
    return 0;
}

The main process is as follows:

Frame? Queue? Peek? Writable takes a writable frame node first

Decoder? Decode? Frame

We will find that the main flow sequence in subtitle "thread" is the opposite to video "thread". Video "thread" is decoded first, then the writable frame node is taken, and then written.

Here, my guess is: because of the slow consumption of subtitle FrameQueue (for example, a common sentence corresponds to a frame), FrameQueue is often in a state of being full. If you decode first, and then call frame ﹣ queue ﹣ peek ﹣ writable, the probability will be blocked. If seek occurs, the decoded frame will be wasted. It's better to wait for the lock before decoding than to wait for the lock before decoding. Of course, it's just a guess. The specific reason is that we should thoroughly understand and renew it.

When decoding subtitles, avcodec? Decode? Subtitle2 is also used. The corresponding decoder? Decode? Frame is simplified as:

static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) {
    int ret = AVERROR(EAGAIN);
    for (;;) {
        //1. Judge the return value
        if (d->queue->serial == d->pkt_serial) {
            do {
                if (ret == AVERROR_EOF) {
                    return 0;
                }
                if (ret >= 0)
                    return 1;
            } while (ret != AVERROR(EAGAIN));
        }//2. take pkt
        packet_queue_get(d->queue, &pkt, 1, &d->pkt_serial);//3. decoding
        ret = avcodec_decode_subtitle2(d->avctx, sub, &got_frame, &pkt);
        if (ret < 0) {
            ret = AVERROR(EAGAIN);
        } else {
            if (got_frame && !pkt.data) {//If it is null and set to pending, continue next time
               d->packet_pending = 1;
               av_packet_move_ref(&d->pkt, &pkt);
            }
            ret = got_frame ? 0 : (pkt.data ? AVERROR(EAGAIN) : AVERROR_EOF);
        }
    }
}

Because it shares a function with audio/video, and uses avcodec  decode  subtitle2 (compared with send/receive, which decodes only one function), the code order looks strange:

Judge return value

Take pkt

Decode

Step 1 is more suitable after step 3, but because it is in the circulatory body, it can also operate correctly.

For avcodec ﹣ decode ﹣ subtitle2, there is a processing of null ﹣ packet, which is usually issued after reading the file.

static int packet_queue_put_nullpacket(PacketQueue *q, int stream_index) { AVPacket pkt1, *pkt = &pkt1; av_init_packet(pkt); pkt->data = NULL; pkt->size = 0; pkt->stream_index = stream_index; return packet_queue_put(q, pkt); }

When decoding subtitles, null packet is used to extract the remaining decoding data from the decoder. If the data can be retrieved (got frame = = 1), the null packet will be temporarily stored in D - > pkt, and the packet will be set in pending, and fetching will continue next time. Until avcodec? Decode? Subtitle2 returns got? Frame = = 0

The logic of this part is explained in the note of avcodec "decode" subtitle2:

/* Some decoders (those marked with AV_CODEC_CAP_DELAY) have a delay between input

  • and output. This means that for some packets they will not immediately
  • produce decoded output and need to be flushed at the end of decoding to get
  • all the decoded data. Flushing is done by calling this function with packets
  • with avpkt->data set to NULL and avpkt->size set to 0 until it stops
  • returning subtitles. It is safe to flush even those decoders that are not
  • marked with AV_CODEC_CAP_DELAY, then no subtitles will be returned.
    */
    At this point, the decoding part of ffplay is analyzed.
Published 100 original articles, won praise 1, visited 4559
Private letter follow

Posted on Fri, 13 Mar 2020 10:41:58 -0400 by scotthoff