Write at the beginning: This article extends from Simulcast This article, this article has only written a general idea, and the implementation details are described in detail here; The following functions are implemented in the M76 version of the source code.

Implementation of simulcast in H264EncoderImpl

In the native source code of WebRTC on the mobile terminal, H264 can be implemented in two ways: H264 supported by system based hardware and H264_ encoder_ openH264 software coding implemented in impl.cc (video_coding / codecs / H264 /);
In addition to the Simulcast function described in this paper (this scheme has been tested and verified in practice), the implementation of software H264 is based on FFmpeg and openH264: Code can refer to ;

Class and encoder initialization

int32_t H264EncoderImpl::InitEncode(const VideoCodec* inst,
                                    int32_t number_of_cores,
                                    size_t max_payload_size) {
 // ReportInit();
 // Omit some parameter check codes

// The following two lines of code obtain the number of simulcasts
  int number_of_streams = SimulcastUtility::NumberOfSimulcastStreams(*inst);
  bool doing_simulcast = (number_of_streams > 1);
// Verify whether the multicast parameters are legal
  if (doing_simulcast &&
      !SimulcastUtility::ValidSimulcastParameters(*inst, number_of_streams)) {
    return WEBRTC_VIDEO_CODEC_ERR_SIMULCAST_PARAMETERS_NOT_SUPPORTED;
  }
 // Save some data to variables in configurations_
  num_temporal_layers_ = codec_.H264()->numberOfTemporalLayers;

  for (int i = 0, idx = number_of_streams - 1; i < number_of_streams;
       ++i, --idx) {
    ISVCEncoder* openh264_encoder;
    // Create encoder
    if (WelsCreateSVCEncoder(&openh264_encoder) != 0) {
      // Failed to create encoder.
     // Omit code
      return WEBRTC_VIDEO_CODEC_ERROR;
    }
    // Store encoder in array
    encoders_[i] = openh264_encoder;
    // When the number of simucast layers is greater than 1, a buffer is required to create a reduced image
    // The set start bit rate and maximum bit rate are saved in configurations_
   
    //Create encoder parameters according to the configuration of each multicast layer. From configurations_ Middle take
    SEncParamExt encoder_params = CreateEncoderParams(i);
    // Initialize encoder
    if (openh264_encoder->InitializeExt(&encoder_params) != 0) {
     // Omit code
      return WEBRTC_VIDEO_CODEC_ERROR;
    }
    // Omit code
    ...
  return WEBRTC_VIDEO_CODEC_OK;
}

From the H264EncoderImpl::InitEncode function in the above source code, we can see that the object really responsible for coding is ISVCEncoder* openh264_encoder: in InitEncode function, if there are multiple simulcast layers, create and initialize the encoder circularly, and prepare the initialization of parameters such as scaling buffer of simulcast layer;
The encoding is in the Encode function:

Simucast coding

The coding implementation is complex and the amount of code is large. Here we only look at the core processing code:

int32_t H264EncoderImpl::Encode(
    const VideoFrame& input_frame,
    const std::vector<VideoFrameType>* frame_types) {
    //Code omission
    ...
  // Traverse the stored encoder and encode corresponding to each video layer
  for (size_t i = 0; i < encoders_.size(); ++i) {
    //Prepare coded input frame
    pictures_[i] = {0};
   //Code omission
    ...
    // Downscale images on second and ongoing layers.
    if (i == 0) { //Code omission
      ...
    }else{ //Reduce the image on the second layer and the ongoing layer.
      //Code omission
      ...
    }
    //If the secondary layer does not need to be sent, ignore it
    if (!configurations_[i].sending) {continue; }
    //Encoding keyframes
    if (send_key_frame) {
      //Code omission
      ...=
	    int enc_ret = encoders_[i]->EncodeFrame(&pictures_[i], &info);
	    if (enc_ret != 0) { return WEBRTC_VIDEO_CODEC_ERROR;
    }
    // The encoded image is divided into segments and updated at the same time
    // |encoded_image_|.
    RTPFragmentationHeader frag_header;
    RtpFragmentize(&encoded_images_[i], *frame_buffer, &info, &frag_header);

    // In this case, the encoder can skip frames to save bandwidth
    // |encoded_images_[i]._length| == 0.
    if (encoded_images_[i].size() > 0) {
      //Code omission
      ...=
      encoded_image_callback_->OnEncodedImage(encoded_images_[i],
                                              &codec_specific, &frag_header);
    }
  }
  return WEBRTC_VIDEO_CODEC_OK;
}

The above is the brief code for encoding multiple encoders to encode multi-channel video. For the specific code, please refer to the source code implementation;

In Android/iOS and other mobile terminals, Simulcast implemented by h264encoder impl often has deficiencies in CPU computing power and battery life due to OpenH264 software coding

In addition, WebRTC iOS compilation of FFmpeg and OpenH264 is still a huge pit. Is there a Simulcast implemented by hardware coding

The above is the words in the original text. iOS is trying to use rtc_use_h264 is set to true, but there will be other problems. In Google talk group WebRTC IOS h264 support! , the description is consistent with the problems encountered in the actual compilation; Roughly as shown in the figure:

In webrtc, there is also an adapter implementation simulcastencoder adapter

Simulcastencoder adapter implementation

Implementation of hardware simulcast based on adapter class

There is a similar implementation in the code of chromium blink. For details, see Chromium source code

Create corresponding factory

Create a factory class by referring to the blink implementation,. h file:

#include <memory>


#include "rtc_base/system/rtc_export.h"
#include "api/video_codecs/video_encoder_factory.h"


namespace webrtc {

// Creates a new factory that can create the built-in types of video encoders.
// The factory has simulcast support for VP8.
RTC_EXPORT std::unique_ptr<VideoEncoderFactory>
CreateVideoEncoderAdapterFactory(std::unique_ptr<VideoEncoderFactory> hardware_encoder_factory);

}  // namespace webrtc

. m file implementation:

#include "video/video_codec_factory_adapter.h"
//#include "base/memory/scoped_refptr.h"
//#include "base/memory/ptr_util.h"

#include "build/build_config.h"
//#include "media/base/media_switches.h"
//#include "media/engine/internal_decoder_factory.h"
#include "api/video_codecs/video_encoder_software_fallback_wrapper.h"
#include "media/engine/simulcast_encoder_adapter.h"
#include <vector>

#include "absl/memory/memory.h"
#include "absl/strings/match.h"
#include "api/video_codecs/sdp_video_format.h"
#include "api/video_codecs/video_encoder.h"
#include "media/base/codec.h"
#include "media/base/media_constants.h"
#include "media/engine/encoder_simulcast_proxy.h"
#include "media/engine/internal_encoder_factory.h"
#include "rtc_base/checks.h"


namespace webrtc {
//template<typename T, typename... Ts>
//std::unique_ptr<T> make_unique(Ts&&... params)
//{
//    return std::unique_ptr<T>(new T(std::forward<Ts>(params)...));
//}
namespace {

template <typename Factory>
bool IsFormatSupported(const Factory* factory,
                       const webrtc::SdpVideoFormat& format) {
  return factory && format.IsCodecInList(factory->GetSupportedFormats());
}

// Merge |formats1| and |formats2|, but avoid adding duplicate formats.
std::vector<webrtc::SdpVideoFormat> MergeFormats(
    std::vector<webrtc::SdpVideoFormat> formats1,
    const std::vector<webrtc::SdpVideoFormat>& formats2) {
  for (const webrtc::SdpVideoFormat& format : formats2) {
    // Don't add same format twice.
    if (!format.IsCodecInList(formats1))
      formats1.push_back(format);
  }
  return formats1;
}
//bool IsFormatSupported(
//                       const Factory* factory,
//                       const webrtc::SdpVideoFormat& format) {
//  return factory && IsFormatSupported(factory->GetSupportedFormats(), format);
//}
//
 Merge |formats1| and |formats2|, but avoid adding duplicate formats.
//std::vector<webrtc::SdpVideoFormat> MergeFormats(
//    std::vector<webrtc::SdpVideoFormat> formats1,
//    const std::vector<webrtc::SdpVideoFormat>& formats2) {
//  for (const webrtc::SdpVideoFormat& format : formats2) {
//    // Don't add same format twice.
//    if (!IsFormatSupported(formats1, format))
//      formats1.push_back(format);
//  }
//  return formats1;
//}

// This class combines a hardware factory with the internal factory and adds
// internal SW codecs, simulcast, and SW fallback wrappers.
class EncoderAdapter : public webrtc::VideoEncoderFactory {
 public:
   EncoderAdapter(
      std::unique_ptr<webrtc::VideoEncoderFactory> hardware_encoder_factory)
      : hardware_encoder_factory_(std::move(hardware_encoder_factory)) {}

  webrtc::VideoEncoderFactory::CodecInfo QueryVideoEncoder(
      const webrtc::SdpVideoFormat& format) const override {
    const webrtc::VideoEncoderFactory* factory =
          IsFormatSupported(hardware_encoder_factory_.get(), format)
            ? hardware_encoder_factory_.get()
            : &software_encoder_factory_;
    return factory->QueryVideoEncoder(format);
  }

    std::unique_ptr<webrtc::VideoEncoder> CreateVideoEncoder(
      const webrtc::SdpVideoFormat& format) override {
    const bool supported_in_software =
        IsFormatSupported(&software_encoder_factory_, format);
    const bool supported_in_hardware =
        IsFormatSupported(hardware_encoder_factory_.get(), format);

    if (!supported_in_software && !supported_in_hardware)
      return nullptr;

    if (absl::EqualsIgnoreCase(format.name.c_str(),
                                         cricket::kVp9CodecName)) {
      // For VP9 and AV1 we don't use simulcast.
      // return software_encoder_factory_.CreateVideoEncoder(format);
      return hardware_encoder_factory_->CreateVideoEncoder(format);
    }

    if (!supported_in_hardware || !hardware_encoder_factory_.get()) {
      return absl::make_unique<webrtc::SimulcastEncoderAdapter>(
          &software_encoder_factory_, format);
    } else if (!supported_in_software) {
    //STD:: make was originally used here_ Unique, the api is in c++14, so it is replaced by the implementation in absl
      return absl::make_unique<webrtc::SimulcastEncoderAdapter>(
          hardware_encoder_factory_.get(), format);
    }

    return absl::make_unique<webrtc::SimulcastEncoderAdapter>(&software_encoder_factory_, format);
  }

  std::vector<webrtc::SdpVideoFormat> GetSupportedFormats() const override {
    std::vector<webrtc::SdpVideoFormat> software_formats =
        software_encoder_factory_.GetSupportedFormats();
    return hardware_encoder_factory_
               ? MergeFormats(software_formats,
                              hardware_encoder_factory_->GetSupportedFormats())
               : software_formats;
  }

 private:
  webrtc::InternalEncoderFactory software_encoder_factory_;
  const std::unique_ptr<webrtc::VideoEncoderFactory> hardware_encoder_factory_;
};
} // namespace
std::unique_ptr<VideoEncoderFactory> CreateVideoEncoderAdapterFactory(
  std::unique_ptr<webrtc::VideoEncoderFactory> hardware_encoder_factory) {
  return absl::make_unique<EncoderAdapter>(std::move(hardware_encoder_factory));
}

}  // namespace webrtc

Note: the simulcastencoder adapter is a construction method with three parameters in the latest code;

usage method

//  ios usage
        auto video_encoder_factory = std::move(native_encoder_factory);
        return [self initWithNativeAudioEncoderFactory:webrtc::CreateBuiltinAudioEncoderFactory()
                             nativeAudioDecoderFactory:webrtc::CreateBuiltinAudioDecoderFactory()
                             nativeVideoEncoderFactory:webrtc::CreateVideoEncoderAdapterFactory(std::move(video_encoder_factory))
                             nativeVideoDecoderFactory:webrtc::CreateBuiltinVideoDecoderFactory()
                                     audioDeviceModule:[self audioDeviceModule]
                                 audioProcessingModule:nullptr];
// android usage
  cricket::MediaEngineDependencies media_dependencies;
  media_dependencies.task_queue_factory = dependencies.task_queue_factory.get();
  media_dependencies.adm = std::move(audio_device_module);
  media_dependencies.audio_encoder_factory = std::move(audio_encoder_factory);
  media_dependencies.audio_decoder_factory = std::move(audio_decoder_factory);
  media_dependencies.audio_processing = std::move(audio_processor);
  auto video_encoder_factory =
      absl::WrapUnique(CreateVideoEncoderFactory(jni, jencoder_factory));
  media_dependencies.video_encoder_factory = 
        CreateVideoEncoderAdapterFactory(std::move(video_encoder_factory));
  media_dependencies.video_decoder_factory =
      absl::WrapUnique(CreateVideoDecoderFactory(jni, jdecoder_factory));
  dependencies.media_engine =
      cricket::CreateMediaEngine(std::move(media_dependencies));

  rtc::scoped_refptr<PeerConnectionFactoryInterface> factory =
      CreateModularPeerConnectionFactory(std::move(dependencies));