How to build an online live broadcasting room (technical post)

background

At present, video live broadcasting is so popular. What technologies are involved in building an online live broadcasting room?

The live video is composed of the live end of the anchor and the viewing end of the audience. A simple viewing terminal should at least include a player and a chat room. The following describes the related technologies around these two modules.

Live video

Live video broadcasting can be divided into video acquisition, pre-processing, encoding and packaging, transmission, unpacking and decoding, and playback.

The live broadcasting end collects audio and video data through hardware equipment, and transmits it to the viewing end after pre-processing, coding and packaging. This step is generally completed by CDN relay. The streaming end will push the video stream to the source station, and the CDN will pull the stream from the source station. After the streaming is successful, it will be encoded and encapsulated into different formats for playback at each viewing end. The simple schematic diagram is as follows:

(network reference picture)

Next, let's talk about the basic requirements of the viewing end:

  • Multi terminal viewing
  • Watch stable, not Caton
  • Low latency and high concurrency

Multi terminal viewing

First, specify the terminals to be supported, including mobile APP (iOS, Android), mobile browser, applet, PC browser and PC client. For PC browsers, IE8 is generally supported. Moreover, since Chrome browser blocks Flash by default, the strategy adopted in PC browser is: give priority to H5 playback in H5 compatible browsers, otherwise downgrade to Flash playback. The differences between the two supported live video formats are as follows:

  • Flash supports rtmp or HTTP flv live broadcast. Delay ~ 3 seconds.
  • H5 supports M3u8 live broadcast. Delay ~ 15 seconds.

Watch stable, not Caton

There may be many reasons for video live broadcast jamming, including live broadcast terminal, network and viewing terminal.

Live broadcast terminal

The main problems are too low hardware configuration, streaming parameter configuration, and asynchronous audio and video timestamps. It can be solved by the following measures:

  • Upgrade hardware and software settings to improve compatibility and fault tolerance (this part is hard installed, and good streaming quality can only be achieved with good equipment).
  • Use hard coding and hard decoding scheme to make full use of GPU acceleration.
  • Reduce the video bit rate, choose smooth, standard definition image quality, or use dynamic bit rate to push the stream.

network

The main problems are network jitter and long link between streaming server and viewing end, which can be solved by the following measures:

  • Select a stable operator network and reasonably arrange CDN nodes.
  • Use sufficient network bandwidth.

Viewing end

When watching video on the network, the buffer is to store part of the video data in advance when you watch the video, and then play the picture when the data reaches a certain amount, so as to play more smoothly. If the setting is too small, it will not play smoothly and continuously when the network is unstable; If the setting is too large, the delay will be accumulated. So set a moderate buffer.

// Flash
netStream.bufferTime = 1 // Unit: Second
// FLV.js
flvjs.createPlayer({
  type: 'flv',
  isLive: true,
  url: 'video.flv'
},{
  stashInitialSize: 120 // Default: 384KB
})

Low latency and high concurrency

We know that video is actually composed of images frame by frame. RTMP is based on TCP and will not lose packets. Therefore, when the network state is poor, the server will cache the packets. When the network condition is good, it will be sent to the viewing end together, resulting in too many video frame data accumulated at the viewing end, and the delay increases with time. For this problem, in addition to setting the appropriate buffer length mentioned above, frame chasing and frame dropping operations can also be added to realize playback chasing.

Flash code:

// Flash realizes frame tracing: the timer polls and detects that when the current buffer length is greater than 30 seconds, it reconnects and pulls the live stream again
netStream.bufferTimeMax = 0.1 // Set bufferTimeMax active frame tracking
​
if(netStream.bufferLength > 30) { 
  // Buffer length greater than 30, reconnect  
  reconnectStream() 
}

H5 Code:

// H5 realizes frame tracing. If the difference between the end time of the current buffer and the current time exceeds 5 seconds, the frame will be traced
if (video.buffered.length > 0 && video.buffered.end(0) - video.currentTime > 5) {
  // If the live stream time is close to the buffer time, the picture is easy to get stuck, so be on the safe side - 1 second 
  
  video.currentTime = video.buffered.end(0) - 1;

}

In addition, if you use FLV.js to play video, you can turn on its Worker feature, multi-threaded parsing, optimize latency, and reduce buffer.

// FLV.js
flvjs.createPlayer({
  type: 'flv',
  isLive: true,
  url: 'video.flv'
},{
  enableWorker: true,
  enableStashBuffer: false,
  stashInitialSize: 120 // Default: 384KB
})

chat room

Instant chat IM service should not only ensure real-time and reliability, but also resist high concurrency. In the implementation process, we use the following methods to solve the problem.

1. WebSocket is the preferred transmission mode. If it is not supported, it will be degraded to polling.

const io = require('socket.io')({ "transports":['websocket', 'polling']})

2. The Node.js server has low performance due to large message concurrency.

The following scheme greatly optimizes the stability and reliability of the chat room.

(1) Features using namespaces

The function of namespace is to limit the propagation of messages within a certain range. Adding a namespace to some messages that do not need to be received globally can greatly save resource transmission.

// Create namespace
const io = require('socket.io')()
const adminNamespace = io.of('/admin')
​
adminNamespace.to('level1').emit('an event', { some: 'data' })

(2) Chat message queue

When the audience enters the chat room, the room will broadcast the login message. For example, there are 2W people in the room at the same time. When each person logs in, they should broadcast "I" login to everyone in the room, which is equivalent to sending 2W messages. If the amount of concurrency is large, it requires very high server performance. Using chat queue message batch display can prevent processing a large number of messages at the same time and improve processing performance.

// Message queue
let scoektMsgArr = [{
   EVENT: 'SPEAK',
   uid: socketId,
   content: 'This is the first chat message' 
},...]
let minCount = 0
​
setInterval(()=>{
    const maxCount = minCount + 100
    const newScoektMsgArr = scoektMsgArr.slice(minCount, maxCount)
    newScoektMsgArr.forEach((item) => {
       socket.emit('message', JSON.stringify(item))
    })
}, 1000)

(3) Server resiliency

Offer the last big move. Those that can be optimized have been optimized. The rest is to configure the server to play (jia), stretch (fu) and shrink (qi).

3. Drop line reconnection mechanism.

Dropping the line will trigger the discrete event. Listen to it and create a socket connection. Heartbeat check is to send a message regularly to maintain the connection status.

// Keep your heart beating
setInterval(()=>{
    socket.emit('message', JSON.stringify({
      EVENT: 'HEARTBEAT',
      uid: socketId
    }))
}, 30 * 60 * 1000)

// Disconnect and reconnect
socket.on('disconnect', () => {
    this.connect()
})

connect() {
  socket.on('connect', () => {
     //TODO
  })
}

POLYV SDK to quickly build an online live broadcasting room

As mentioned above, there are many details to consider in building a live studio, including collection and streaming, CDN distribution, playback experience optimization, chat room performance optimization, etc. Don't worry. Calling the POLYV SDK can quickly build a live studio.

(you need to use POLYV live video service first Free account registration)

STEP1 embedded player:

<script src="https://player.polyv.net/livescript/liveplayer.js"></script>
<div id='player'></div>
<script>
    var player = polyvObject('#player').livePlayer({
        'width':'498',
        'height':'409',
        'uid':'e3wx706i3v',
        'vid':'268682' // polyv live channel number
    });
</script>

STEP2 embedded chat room:

// Default style
<link rel="stylesheet" href="https://player.polyv.net/jssdk/polyv-chatroom.min.css">
​
<script src="https://player.polyv.net/jssdk/polyv-chatroom.min.js"></script>
<div id="wrap"></div>
var chatroom = new PolyvChatRoom({
    roomId: 268682,
    userId: 153075602311,
    nick: 'tourist',
    pic: 'http://livestatic.videocc.net/assets/wimages/missing_face.png',
    token: token,
    container: '#wrap',
    width: 300,
    height: 600,
    userType: '',
    roomMessage: function(data) {
        // TODO
        // data is the chat room socket message. This method will be triggered when there is a chat room message
        console.log(data);
    }
});

As shown in the above figure, the live broadcast room can be created by embedding the live broadcast SDK + chat room SDK. The chat room SDK has its own default skin, send expression, like, send flowers, online list, question and other functions. In addition, for other functions required for live education scenes, such as answer cards, check-in, etc., you can listen to the socket message in the chat room and customize the implementation:

1. Answer sheet

// Monitor answer content
var chatroom = new PolyvChatRoom({
    roomId: 268682,
    userId: 153075602313,
    nick: 'student_1',
    pic: 'http://livestatic.videocc.net/assets/wimages/missing_face.png',
    token: token,
    container: '#wrap',
    width: 300,
    height: 600,
    userType: '',
    roomMessage: function(data) {
        // Listen for chat messages
        switch(data.EVENT){
            case 'GET_TEST_QUESTION_RESULT':
                // Get the content of the answer sheet and display the answer sheet
              break;
            case 'GET_TEST_QUESTION_RESULT': 
              // Obtain the answer sheet results and display the answer results
              break;  
        }
    }
});
​
// Send answer results
chatroom.socket.emit('message', JSON.stringify({
    EVENT: 'ANSWER_TEST_QUESTION',
    roomId: channelId,
    nick: nick,
    userId: userId,
    option: result,
    questionId: questionId
}));

2. Sign in

// Sign in initiated by the live broadcast terminal
roomMessage: function(data) {
    // Listen for chat messages
    switch(data.EVENT){
        case 'SIGN_IN':
            // Initiate sign in
            break;
        case 'STOP_SIGN_IN': 
            // Stop signing in
            break;  
    }
}
​
// Audience sign in
chatroom.socket.emit('message', JSON.stringify({
    EVENT: 'TO_SIGN_IN',
    roomId: roomId,
    checkinId: checkinId,
    user: {
        userId: 123456,
        nick: 'polyv'
    }
}));

Applet


In addition to the browser side, we also provide the corresponding SDK on the applet side.

download Applet SDK After that, the calling component quickly generates live broadcast rooms including players, chat rooms and other functions.

// Embedded polyv page component
<view>
  <polyv />
</view>

// Initialization data
import plv from '*/polyv-sdk/index';
// onLoad
onLoad() {
   const options = {
     channelId: '', // Channel ID
     openId: '', // User openId
     userName: '', // user name
     avatarUrl: '' // User Avatar
   };
   plv.init(options);
}
// onUnload
onUnload() {
  plv.destory();
}

Summary:

If the basic function of educational live broadcasting is realized by completely self-research, it needs at least a technical team of 10 and a product operation team of 5 to complete the launch of the product within 3 months. It takes at least hundreds of thousands or even millions. POLYV provides enterprises with comprehensive solutions and relevant documents to achieve fast access and easily open online education live broadcast.

Information:

Live broadcast using flv.js   https://segmentfault.com/a/1190000009695424

Technology and optimization behind the second opening of live video

Check   https://juejin.im/post/58eaf49a8d6d8100618bc238

http://socket.io https://socket.io/docs/

Tags: html5 SDK Mini Program cdn

Posted on Thu, 02 Sep 2021 18:52:08 -0400 by slshmily