preface:

WebRTC is an open Web standard, which is used to support two-way real-time communication of voice, video and general data between browsers. Driven by the big companies led by Google, WebRTC technologies have gradually matured and standardized, and become a Web-based real-time audio and video communication solution supported by various mainstream browsers.
WebRTC itself is a P2P like technology applied to the client. SRS4.0 introduces WebRTC processing capability mainly to build the SFU capability of the server (what is SFU? Readers can search by themselves). Borrow one here Network diagram To illustrate the working principle of SFU:

As shown in the above figure, there are usually multiple WebRTC clients for video conference scenarios. At this time, by deploying the SFU server, a streaming connection for local audio and video data is established between each WebRTC client and the SFU server. At the same time, multiple streaming connections can be established between the WebRTC client and the SFU server as needed. The advantage of this is that it makes use of the powerful client access capability of SFU server, and will not consume too much CPU computing power on the server because of audio and video streaming.

Therefore, the main purpose of introducing WebRTC capability into SRS4.0 is:
1) Support the browser to pull the stream from the SRS server without plug-in and play it directly.
2) The total delay (minimum millisecond delay) of the bass video data.
3) It supports two-way audio and video capabilities and live broadcast of wheat connected scenes.

Objectives:

WebRTC includes many knowledge points, from the generation and exchange of SDP messages, the establishment of connection in ICE mode, DTLS handshake / SRTP encryption and decryption, RTP/RTCP data encapsulation and transmission, to various Qos processing to improve the audio and video user experience in the face of network jitter and insufficient bandwidth. Each knowledge point involves a lot of content. This chapter will start with the establishment of WebRTC push-pull stream connection, By analyzing the overall flow of audio and video data between key classes and key functions, we first understand the code logic of SRS4.0 WebRTC server module as a whole.

Content:

1. SRS4.0 WebRTC service startup

The initialization and startup interface of SRS WebRTC service module is in the file SRS_ app_ rtc_ In server.cpp, the overall processing logic includes:
1) Generate a self signed certificate for DTLS
2) Start UDP port (8000) listening and process STUN/DTLS/RTP messages
3) Register push-pull stream API interface

srs_error_t RtcServerAdapter::initialize()
{
    ......
    // This function internally calls the openssl library to generate a self signed certificate for subsequent DTLS authentication
    if ((err = _srs_rtc_dtls_certificate->initialize()) != srs_success) {
        return srs_error_wrap(err, "rtc dtls certificate initialize");
    }
    
    // This function internally subscribes to the timeout message of the 5-second timer and completes some periodic work through this message
    if ((err = rtc->initialize()) != srs_success) {
        return srs_error_wrap(err, "rtc server initialize");
    }

    return err;
}

srs_error_t RtcServerAdapter::run()
{
    ......
    // Create a UDP port listening object SrsUdpMuxListener. The default listening port is 8000
    if ((err = rtc->listen_udp()) != srs_success) {
        return srs_error_wrap(err, "listen udp"); 
    }

    // Register the push-pull flow API of the RTC module with the global SrsHttpServeMux object
    if ((err = rtc->listen_api()) != srs_success) {
        return srs_error_wrap(err, "listen api");
    }
    
    // Start_ srs_ rtc_ The manager internal collaboration is used to clean up internal zombie connections and recycle resources
    if ((err = _srs_rtc_manager->start()) != srs_success) {
        return srs_error_wrap(err, "start manager");
    }

    return err;
}

As we know earlier, a socket connection is always established between the RTMP client and the server. The client sends a push-pull stream request command to the server through this connection. After receiving the request command, the server uses the same socket connection to transmit audio and video data streams. Therefore, the push-pull stream control command and audio-video data stream of RTMP protocol are always transmitted through the same socket connection, and socket multiplexing is realized through different RTMP message types.

WebRTC protocol establishes a UDP data channel between two WebRTC terminals based on P2P/ICE technology. When terminal 1 sends a request to establish a connection to terminal 2, it usually completes the forwarding of these control commands through a separate signaling server (the specific method is that the two WebRTC terminals maintain long connections with the signaling server respectively, and use different client IDs to identify different client long connections. When terminal 1 sends a request command to terminal 2, as long as the client ID of terminal 2 is brought in the command message and the request message is sent to the signaling server, the signaling server can pass the request message through the correct client Long connections are forwarded to terminal 2).
Therefore, any practical WebRTC system must have its own signaling server and control commands, and the implementation of this part is usually private, because the WebRTC standard itself does not define these necessary control commands.

The following draft refers to the push-pull stream URL specification of RTMP protocol: https://github.com/rtcdn/rtcdn-draft
Defines the shape such as webrtc://domain/ WebRTC push and pull stream URL of conference ID / push client ID.
Taking video conference as an example, the conference IDs of different conferences must be different, and the streaming IDs of different clients in the same conference must be different.

Meanwhile, SRS4.0 provides streaming API interfaces (/ rtc/v1/publish /) and streaming API interfaces (/ rtc/v1/play /) through HTTP(S) services.

srs_error_t SrsRtcServer::listen_api()
{
    ......
    // Get the global API management object SrsHttpServeMux
    SrsHttpServeMux* http_api_mux = _srs_hybrid->srs()->instance()->api_server();
    
    // Register the processing object SrsGoApiRtcPlay corresponding to the WebRTC streaming API
    if ((err = http_api_mux->handle("/rtc/v1/play/", new SrsGoApiRtcPlay(this))) != srs_success) {
        return srs_error_wrap(err, "handle play");
    }
    // Register the processing object SrsGoApiRtcPublish corresponding to the WebRTC streaming API
    if ((err = http_api_mux->handle("/rtc/v1/publish/", new SrsGoApiRtcPublish(this))) != srs_success) {
        return srs_error_wrap(err, "handle publish");
    }

    return err;
}

2. Processing streaming API

The WebRTC client sends a streaming request command to the SRS server through the streaming API interface (/ rtc/v1/publish /). At this time, the SRS processing flow is as follows:

srs_error_t SrsGoApiRtcPublish::serve_http() { // This function is the processing entry of the streaming API
    do_serve_http(w, r, res); // Process the remote streaming request (including the client SDP information) and construct the request response
    return srs_api_response(w, r, res->dumps()); // Send request response to client (including local SDP information)
}

srs_error_t SrsGoApiRtcPublish::do_serve_http() { // Process the remote streaming request and construct the request response
    ......
    server_->create_session(&ruc, local_sdp, &session); // Create session object and local SDP information
}

srs_error_t SrsRtcServer::create_session() {
    ......
    // Reference WebRTC push and pull stream URL
    // Take the string "/ conference ID / streaming client ID" as the Key to create a corresponding srsrrtcsource object for each streaming client
    _srs_rtc_sources->fetch_or_create(req, &source); 

    // Create a session object of type srsrsrtcconnection for each streaming end
    SrsRtcConnection* session = new SrsRtcConnection(this, cid); 
    do_create_session(ruc, local_sdp, session); // 
}

srs_error_t SrsRtcServer::do_create_session(SrsRtcUserConfig* ruc, SrsSdp& local_sdp, SrsRtcConnection* session){
    if (ruc->publish_) {
        session->add_publisher(ruc, local_sdp); // Add a streaming side processing object for the session
    } 
    
    session->initialize(); //     
    _srs_rtc_manager->add_with_name(username, session);// Save the session object with the local random string ufrag + Remote ufrag as the Key 
}

srs_error_t SrsRtcConnection::add_publisher(SrsRtcUserConfig* ruc, SrsSdp& local_sdp){
    create_publisher(req, stream_desc);
}

srs_error_t SrsRtcConnection::create_publisher(SrsRequest* req, SrsRtcSourceDescription* stream_desc)
{
    // Create the streaming side processing object srsrrtcpublishstream, and start the internal srsrrtcpliworker collaboration
    SrsRtcPublishStream* publisher = new SrsRtcPublishStream();
    publisher->start();
}

After receiving the streaming API(/rtc/v1/publish /) sent by the user, SRS finally creates srsrsrtcconnection object, SrsRtcPublishStream object and srsrrtcsource object through the above function call stack.

3. Handle pull flow API

The WebRTC client sends a streaming request command to the SRS server through the streaming API interface (/ rtc/v1/play /). At this time, the SRS processing flow is as follows:

srs_error_t SrsGoApiRtcPlay::serve_http() { // This function is the processing entry of the pull stream API
    do_serve_http(w, r, res); // Process the remote streaming request (including the client SDP information) and construct the request response
    return srs_api_response(w, r, res->dumps()); // Send request response to client (including local SDP information)
}

srs_error_t SrsGoApiRtcPlay::do_serve_http() { //  Process the remote streaming request and construct the request response
    server_->create_session(&ruc, local_sdp, &session); // Create session and local SDP
}

srs_error_t SrsRtcServer::create_session() {
    ......
    // Reference WebRTC push and pull stream URL
    // With the string "/ conference ID / streaming client ID" as the Key, the streaming end finds the srsrsrtcsource object of the corresponding streaming end
    _srs_rtc_sources->fetch_or_create(req, &source); 

    // Create a session object of type srsrsrtcconnection for each pull stream end
    SrsRtcConnection* session = new SrsRtcConnection(this, cid); 
    do_create_session(ruc, local_sdp, session); // 
}

srs_error_t SrsRtcServer::do_create_session(SrsRtcUserConfig* ruc, SrsSdp& local_sdp, SrsRtcConnection* session)
{
    if (ruc->publish_) {
        ......
    } else {
        session->add_player(ruc, local_sdp); // Add a pull stream side processing object for the session
    }
    
    session->initialize(); //     
    _srs_rtc_manager->add_with_name(username, session);// Save the session object with the local random string ufrag + Remote ufrag as the Key 
}

srs_error_t SrsRtcConnection::add_player(SrsRtcUserConfig* ruc, SrsSdp& local_sdp){
    create_player(req, play_sub_relations);
}

srs_error_t SrsRtcConnection::create_player(SrsRequest* req, std::map<uint32_t, SrsRtcTrackDescription*> sub_relations){
    // Create the pull end processing object srsrsrtcplaystream and start the pull end processing process
    SrsRtcPlayStream* player = new SrsRtcPlayStream();
    player->start();
}

srs_error_t SrsRtcPlayStream::cycle(){ // Pull end processing protocol
    source->create_consumer(consumer); // Create a srsrrtcconsumer consumer object for each pull end
    
    while (true) {
        consumer->dump_packet(&pkt); // Block and get message in srsrsrtconsumer consumer queue
        if (!pkt) {
            consumer->wait(mw_msgs);
            continue;
        }
        
        send_packet(pkt)；// The streaming client that sends the message in the srsrrtcconsumer consumer queue
    }
}

After receiving the streaming API(/rtc/v1/play /) sent by the user, SRS finally creates the srsrrtcconnection object, srsrrtcpaystream object and srsrrtcconsumer object through the above function call stack.

4. SDP exchange establishes connection with ICE

The above process only creates the key objects for the WebRTC service. Next, we need to analyze how to establish a connection between the push-pull stream client and the listening port (8000) of the WebRTC service. The connection between WebRTC client and server is established by P2P like private network penetration. One of the biggest features of this method is that when a WebRTC client sends a connection request to the server, it does not know the IP address and port number of the server in advance. Therefore, the establishment of WebRTC connection generally includes two stages:
1) SDP(Session Description Protocol) messages containing respective IP address + port number information are exchanged between WebRTC client and server in the form of offer and answer.
2) The WebRTC client obtains the IP address and port number of the server from the SDP message of the server, and establishes a connection between the client and the server in the form of ice (Interactive connectivity infrastructure) for subsequent audio and video data transmission.
There are many materials about SDP and ICE on the Internet, which can be learned and referred to as needed
https://segmentfault.com/a/1190000038272539 Detailed explanation and analysis of webrtc SDP
https://segmentfault.com/a/1190000020794391? utm_ Source = SF similar article webrtc session description protocol (SDP) details
https://zhuanlan.zhihu.com/p/60684464 On ICE of webrtc

The following is the offer SDP sent by the browser to the SRS server. Because it is in the trick mode, the SDP does not contain the IP address of the client. Of course, this does not affect the final connection establishment.

v=0
o=- 6308787264381624235 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
a=ice-options:trickle
a=sendonly
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 121 127 120 125 107 108 109 35 36 124 119 123
a=ice-options:trickle
a=sendonly

The answer SDP responded by the SRS server, in which the candidate attribute contains the description information of the SRS server IP address and port (192.168.9.102 8000), and the server adopts the ICE Lite mode to simplify the ICE negotiation process.

v=0
o=SRS/4.0.140(Leo) 32138128 2 IN IP4 0.0.0.0
s=SRSPublishSession
t=0 0
a=ice-lite
a=group:BUNDLE 0 1
m=audio 9 UDP/TLS/RTP/SAVPF 111
a=recvonly
a=candidate:0 1 udp 2130706431 192.168.9.102 8000 typ host generation 0
m=video 9 UDP/TLS/RTP/SAVPF 125 124
a=recvonly
a=candidate:0 1 udp 2130706431 192.168.9.102 8000 typ host generation 0

Next, the browser sends a Binding Request message to the 8000 port of the SRS server, and the server returns a Binding Success Response to the browser. Finally, the push-pull stream client establishes a connection with the SRS server (8000 port).

Subsequently, DTLS verification will be completed on this connection between the client and the server, and audio and video RTP messages will be transmitted.

Summary:

The overall architecture and processing flow of SRS4.0 WebRTC module are:
1) Listen to UDP port (default 8000), and register streaming API interface (/ rtc/v1/publish /) and streaming API interface (/ rtc/v1/play /).

2) The push end processing logic creates the srsrrtcconnection object, srrtcpublishstream object and srrtcsource object, and the pull end processing logic creates the srrtcconnection object, srrtcplaystream object and srsrrtcconsumer object.

3) The push-pull stream client and SRS server exchange through SDP and establish UDP connection by ICE to complete DTLS security negotiation.

4) Finally, the data flow of audio and video data from streaming client to streaming client is shown in the following figure:
a) Streaming client – > server 8000 port – > srsrrtcconnection – > srsrrtcpublishstream – > srsrrtcsource – > srsrrtcconsumer data queue
b) SrsRtcConsumer data queue - > srsrsrtcplaystream:: cycle() process get data - > pull client