When the media server interacts with the WebRTC client, it can also be regarded as a "client" of WebRTC, so it should also follow the interaction rules of WebRTC. As a passive receiver, the server receives the OfferSdp sent by the client. When the signaling server receives the OfferSdp, it will convert it into the Json format required by MediaSoup, and then transmit it to the media server through the pipeline. The media server processes the corresponding Json format, and then returns the corresponding processing result to the signaling server, The signaling server encapsulates the processing result into AnswerSdp and returns it to the client to complete the SDP negotiation.
Note: the signaling server here is equivalent to the JS part of the native MediaSoup, which can be rewritten according to business needs, and the media server is the C + + part of MediaSoup.
It can be seen from the above analysis that the SDP negotiation of MediaSoup includes parsing OfferSdp, encapsulating MediaSoup messages and encapsulating AnswerSdp. These are explained separately here.
1. OfferSdp
For example, the OfferSdp generated by the test is listed here:
v=0 o=- 2769202419187616652 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=extmap-allow-mixed a=msid-semantic: WMS 6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:V27U a=ice-pwd:j8YGds5JpiHzbkd7Wwdxl5rq a=ice-options:trickle a=fingerprint:sha-256 97:23:44:6B:EC:52:D6:64:D6:02:42:69:EA:9C:99:F8:F3:F3:F4:1B:F3:74:D6:CF:4E:60:5D:89:97:9F:D6:3C a=setup:actpass a=mid:audio a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=sendrecv a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:112 telephone-event/32000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:2096589939 cname:PK+nWUKVj/eH7KG7 a=ssrc:2096589939 msid:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt ef3670e9-615c-4e0b-aa2f-a2b21d1b0b5d a=ssrc:2096589939 mslabel:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt a=ssrc:2096589939 label:ef3670e9-615c-4e0b-aa2f-a2b21d1b0b5d m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 121 127 120 125 107 108 109 35 36 124 119 123 118 114 115 116 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:V27U a=ice-pwd:j8YGds5JpiHzbkd7Wwdxl5rq a=ice-options:trickle a=fingerprint:sha-256 97:23:44:6B:EC:52:D6:64:D6:02:42:69:EA:9C:99:F8:F3:F3:F4:1B:F3:74:D6:CF:4E:60:5D:89:97:9F:D6:3C a=setup:actpass a=mid:video a=extmap:14 urn:ietf:params:rtp-hdrext:toffset a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:13 urn:3gpp:video-orientation a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space a=sendrecv a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 VP9/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 profile-id=2 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:102 H264/90000 a=rtcp-fb:102 goog-remb a=rtcp-fb:102 transport-cc a=rtcp-fb:102 ccm fir a=rtcp-fb:102 nack a=rtcp-fb:102 nack pli a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f a=rtpmap:121 rtx/90000 a=fmtp:121 apt=102 a=rtpmap:127 H264/90000 a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f a=rtpmap:120 rtx/90000 a=fmtp:120 apt=127 a=rtpmap:125 H264/90000 a=rtcp-fb:125 goog-remb a=rtcp-fb:125 transport-cc a=rtcp-fb:125 ccm fir a=rtcp-fb:125 nack a=rtcp-fb:125 nack pli a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtpmap:107 rtx/90000 a=fmtp:107 apt=125 a=rtpmap:108 H264/90000 a=rtcp-fb:108 goog-remb a=rtcp-fb:108 transport-cc a=rtcp-fb:108 ccm fir a=rtcp-fb:108 nack a=rtcp-fb:108 nack pli a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f a=rtpmap:109 rtx/90000 a=fmtp:109 apt=108 a=rtpmap:35 AV1X/90000 a=rtcp-fb:35 goog-remb a=rtcp-fb:35 transport-cc a=rtcp-fb:35 ccm fir a=rtcp-fb:35 nack a=rtcp-fb:35 nack pli a=rtpmap:36 rtx/90000 a=fmtp:36 apt=35 a=rtpmap:124 H264/90000 a=rtcp-fb:124 goog-remb a=rtcp-fb:124 transport-cc a=rtcp-fb:124 ccm fir a=rtcp-fb:124 nack a=rtcp-fb:124 nack pli a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032 a=rtpmap:119 rtx/90000 a=fmtp:119 apt=124 a=rtpmap:123 H264/90000 a=rtcp-fb:123 goog-remb a=rtcp-fb:123 transport-cc a=rtcp-fb:123 ccm fir a=rtcp-fb:123 nack a=rtcp-fb:123 nack pli a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032 a=rtpmap:118 rtx/90000 a=fmtp:118 apt=123 a=rtpmap:114 red/90000 a=rtpmap:115 rtx/90000 a=fmtp:115 apt=114 a=rtpmap:116 ulpfec/90000 a=ssrc-group:FID 3606269878 2056516636 a=ssrc:3606269878 cname:PK+nWUKVj/eH7KG7 a=ssrc:3606269878 msid:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt 9e5ca158-57c2-4900-9cd4-edc08d0cd3cd a=ssrc:3606269878 mslabel:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt a=ssrc:3606269878 label:9e5ca158-57c2-4900-9cd4-edc08d0cd3cd a=ssrc:2056516636 cname:PK+nWUKVj/eH7KG7 a=ssrc:2056516636 msid:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt 9e5ca158-57c2-4900-9cd4-edc08d0cd3cd a=ssrc:2056516636 mslabel:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt a=ssrc:2056516636 label:9e5ca158-57c2-4900-9cd4-edc08d0cd3cd
2. MediaSoup message
When the OfferSdp is received, it is parsed and encapsulated into the Json format required by MediaSoup. There are many types of MediaSoup messages, and the main types are selected here.
2.1 creating a Router
Router is a concept encapsulated by MediaSoup. It is used to manage Transport and has nothing to do with SDP negotiation.
Request:
{
"data":{},
"id":0, #id: the number of signaling is 0
"internal":{
"routerId":"123456" #Specifies the id of the Router
},
"method":"worker.createRouter"
}
reply:
{
"accepted": true,
"id": 0 #Indicates the reply to which signaling
}
Note: MediaSoup requests and replies are associated through the id field.
The SDP negotiation of WebRTC actually includes two types: the negotiation of connection information and the negotiation of media information. The sections 2.2 creating WebRtcTransport and 2.3 connecting to WebRtcTransport are actually to complete the negotiation of SDP connection information, including ICE connection information and DTLS information. 2.4 creating an audio Producer and 2.5 creating a video Producer represent the media information negotiation with the client when MediaSoup is the receiving end. Creating an audio Consumer in 2.6 and creating a video Consumer in 2.7 represent the media information negotiation with the client when MediaSoup is the sender.
2.2 create WebRtcTransport
When transmitting audio and video, the transport layer generally uses UDP protocol. Therefore, creating WebRtcTransport is actually creating a Udp Socket that transmits audio and video streams with the client. Therefore, we need to tell MediaSoup what the created IP address is. MediaSoup has a port manager inside. We don't need to write the port information into the request Json.
Request:
{
"data":{
"enableSctp":false,
"enableTcp":false, #Do not use TCP
"enableUdp":true, #Use UDP
"initialAvailableOutgoingBitrate":1000000,
"isDataChannel":false,
"listenIps":[
{
"announcedIp":"192.168.190.140", #The ip address that WebRtcTransport listens to
"ip":"192.168.190.140" #Public network address
}
],
"maxSctpMessageSize":0,
"numSctpStreams":{
"MIS":0,
"OS":0
},
"preferTcp":false, #Whether TCP is preferred when passing through the wall
"preferUdp":false, #Whether UDP is preferred when passing through the wall
"sctpSendBufferSize":0
},
"id":1,
"internal":{
"routerId":"123456", #id of router corresponding to WebRtcTransport
"transportId":"123456&&1634535081191_0" #id of WebRtcTransport
},
"method":"router.createWebRtcTransport"
}
reply:
The returned information mainly includes ICE information and DTLS information, of which ICE information includes:
a. ice role : controlled MediaSoup in ice Role fixed as controlled
b. iceParameters : UserName / Password return MediaSoup Generated UserName and Password For verification ice
c. iceLite : true use iceLite，Not used fullIce
d. ice candidate
foundation : "udpcandidate" or "tcpcandidate"
priority : establish udpsocket Priority when;
ip : Monitored ip
port : adopt mediasoup Own port manager generation
protocol : "udp" or "tcp"
type : "host" Fixed as host，express candidate The type is real ip
tcptype : "passive" If protocol yes tcp，express tcp Type of(server)
e. ice state : new
DTLS information includes:
a. fingerprints : Indicates the hash algorithm used and the hash value of the certificate b. role : Auto dtls role c. dtlsState : new
{
"accepted":true,
"data":{
# dtls information
"dtlsParameters":{
"fingerprints":[
{
"algorithm":"sha-1",
"value":"D4:94:A7:AF:CA:5A:7C:4A:B2:4A:73:E3:46:5A:7C:FE:0B:A3:D9:25"
},
{
"algorithm":"sha-224",
"value":"85:9A:23:4C:B5:A2:A3:67:1D:85:BD:DC:FE:CC:02:D9:8F:D4:88:FA:A0:55:59:2C:4E:89:0F:CB"
},
{
"algorithm":"sha-256",
"value":"2D:D6:29:29:99:2A:E1:95:59:9D:C2:E9:FE:F1:B2:25:94:80:2A:47:FB:2E:D4:51:58:2E:4A:90:F1:78:F6:97"
},
{
"algorithm":"sha-384",
"value":"A6:8E:DE:3C:16:A1:CB:11:AC:1E:F0:D6:42:07:E8:9F:98:6A:02:9F:A1:FB:38:FB:12:DF:80:A7:3F:F1:2D:D8:20:07:E5:BF:45:49:D2:BC:77:7A:AD:27:32:D9:D7:EE"
},
{
"algorithm":"sha-512",
"value":"7C:E4:E8:1C:42:DF:03:E5:A2:EF:14:D6:10:D3:E4:B1:EA:42:DF:4B:EF:6C:B1:EE:E1:F3:F3:8A:C3:C5:16:BC:87:E2:C4:9D:DC:61:8B:0C:E5:EA:1B:DB:96:1E:FD:4B:97:70:E1:1C:9F:B8:B3:2C:D8:2F:D6:01:57:AD:82:EC"
}
],
"role":"auto"
},
"dtlsState":"new",
# ice information
"iceCandidates":[
{
"foundation":"udpcandidate",
"ip":"192.168.190.140",
"port":42153,
"priority":1076302079,
"protocol":"udp",
"type":"host"
}
],
"iceParameters":{
"iceLite":true,
"password":"cztysmgjc1ahth0u1kyntn1k12vyy61f",
"usernameFragment":"ve4lr5fqvwdg0wzt"
},
"iceRole":"controlled",
"iceState":"new",
# other
"id":"123456&&1634535081191_0",
"mapRtxSsrcConsumerId":{},
"mapSsrcConsumerId":{},
"maxMessageSize":262144,
"producerIds":[],
"recvRtpHeaderExtensions":{},
"rtpListener":{
"midTable":{},
"ridTable":{},
"ssrcTable":{}
},
"consumerIds":[],
"dataConsumerIds":[],
"dataProducerIds":[],
"direct":false,
"traceEventTypes":""
},
"id":1
}
2.3 connecting to WebRtcTransport
The main work of connecting to WebRtcTransport is to set the DTLS role of the server according to the DTLS role of the client.
Here, the DTLS parameters in OfferSdp will be resolved, mainly including two: DTLS role and fingerprint.
The dtls role is generally auto, which means that the client does not decide the dtls role and gives it to the other party for decision. The corresponding sdp is:
a=setup:actpass
Fingerprint is the fingerprint information of the client. After receiving the fingerprint, the server will save it (if there are multiple fingerprints in sdp, only the first one will be taken). Later, the fingerprint information will be verified during dtls handshake. The sdp corresponding to fingerprint information is:
a=fingerprint:sha-256 7E:27:BF:16:30:FC:E3:BE:2F:2C:D3:4A:13:24:E9:09:EF:B1:72:D9:10:C4:8A:F7:E0:42:3A:E3:F1:77:89:DA
Request:
{
"data":{
"dtlsParameters":{
"fingerprints":[
{
"algorithm":"sha-256",
"value":"F6:E0:D1:EC:0F:9F:8F:00:81:34:5C:5E:99:1C:CF:3A:5C:5D:6C:C8:A5:3A:26:32:0E:93:F4:83:AC:A5:A7:E3"
}
],
"role":"auto"
}
},
"id":2,
"internal":{
"routerId":"123456",
"transportId":"123456&&1634535081191_0"
},
"method":"transport.connect"
}
The content of the reply is mainly the DTLS role of the server, usually the client.
reply:
{
"accepted":true,
"data":{
"dtlsLocalRole":"client"
},
"id":2
}
Summary: after creating WebRtcTransport in 2.2 and connecting WebRtcTransport in 2.3, the interaction of SDP connection information between server and client has been completed. The server returns its ICE information and DTLS role information to the client.
2.4 creating an audio Producer
MediaSoup encapsulates Producer and Consumer on Transport to distinguish producers and consumers. Audio and video have independent producers and consumers. Here, the audio Producer is used to push the streaming audio RtpPacket. As mentioned above, Producer actually negotiates SDP media information. The Producer request mainly includes two structures. One is rtpMapping, which maps PayloadType and SSRC. Corresponding logic will be used for data forwarding in MediaSoup; The other is rtpParameters, which is used to encapsulate the negotiated media information, mainly including Codec information, SSRC information, extension header information and RTCP information; Let's talk about the role of this information.
1.Codec information
In fact, it is the negotiation of media Codec types at both ends. The client OfferSdp will bring the Codec types it supports. The server will negotiate a public type according to the types it supports, and then fill this type in the rtpparameters - > codecs parameter. The corresponding SDP is:
# RTP with Payload Type 111 is defined to transmit media in opus format. The media sampling frequency is 48khz and dual channels a=rtpmap:111 opus/48000/2 # Describes that the type of RTCP Feedback supported by opus is TCC a=rtcp-fb:111 transport-cc #The opus media coding parameters are described a=fmtp:111 minptime=10;useinbandfec=1
2.SSRC information
SSRC information is used to identify RTP streams. Different RTP streams cannot have the same SSRC. Therefore, if audio and video exist simultaneously in a session, there will be different audio and video sSRCs. The corresponding SDP is:
a=ssrc:2096589939 cname:PK+nWUKVj/eH7KG7 a=ssrc:2096589939 msid:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt ef3670e9-615c-4e0b-aa2f-a2b21d1b0b5d a=ssrc:2096589939 mslabel:6jWPL5zVxadwFUv3czcxk6OuMJhxwepXrnFt a=ssrc:2096589939 label:ef3670e9-615c-4e0b-aa2f-a2b21d1b0b5d
3. Extension header information
The mapping relationship between the expanded header ID and the expanded content of the RTP header actually transmitted is described. The corresponding SDP is:
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
4.RTCP information
For the negotiation of RTCP information, the corresponding SDP is:
# A = RTCP MUX is defined in RFC 5761 to identify the current session and bind RTP and RTCP in the same connection address and port a=rtcp-mux # A = RTCP rsize is defined in RFC 5506 to identify that the current session supports reduced size RTCP packets (so-called composite RTCP packets). a=rtcp-rsize
Request:
{
"data":{
"kind":"audio", # The type is audio
"paused":false, # Pause receiving audio
"rtpMapping":{
"codecs":[
{
"mappedPayloadType":111,
"payloadType":111
}
],
"encodings":[
{
"mappedSsrc":123,
"ssrc":879445687
}
]
},
# The information in SDP negotiation will be written to the rtpParameters object
"rtpParameters":{
"codecs":[
{
"clockRate":48000,
"mimeType":"audio/opus",
"parameters":{
"minptime":10,
"useinbandfec":1
},
"payloadType":111,
"rtcpFeedback":[
{
"type":"transport-cc"
}
]
}
],
"encodings":[
{
"active":true,
"ssrc":879445687
}
],
"headerExtensions":[
{
"id":1,
"uri":"urn:ietf:params:rtp-hdrext:ssrc-audio-level"
},
{
"id":2,
"uri":"http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time"
},
{
"id":3,
"uri":"http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01"
}
],
"mid":"audio",
"rtcp":{
"cname":"RJaQXo8/KlfMQ/r6",
"mux":true,
"reduceSize":true
}
}
},
"id":3,
"internal":{
"producerId":"123456&&1634535081191_0**audio",
"routerId":"123456",
"transportId":"123456&&1634535081191_0"
},
"method":"transport.produce"
}
reply:
{
"accepted":true,
"data":{
"id":"123456&&1634628364953_0**audio",
"type":"simple"
},
"id":3
}
2.5 creating a video Producer
The meaning of the field is the same as that of the audio Producer created above. There is an additional RTX type here (RTX is generally not enabled for audio). RTX has its own SSRC. When RTX is identified as enabled in SDP, the retransmission packets are transmitted through RTX instead of RTP. This can avoid that when RTX is not enabled, the sender simply retransmits the original RTP packets, This mode will affect the RTCP statistics of the receiver, such as negative packet loss rate. However, it should be noted that RTX does not necessarily transmit retransmission packets, but also Padding packets.
The SDP corresponding to RTX is:
# The payloadtype of rtx is 97 a=rtpmap:97 rtx/90000 # Used to retransmit packets with payloadtype 96 a=fmtp:97 apt=96
Request:
{
"data":{
"kind":"video",
"paused":false,
"rtpMapping":{
"codecs":[
{
"mappedPayloadType":96,
"payloadType":96
},
{
"mappedPayloadType":97,
"payloadType":97
}
],
"encodings":[
{
"mappedSsrc":456,
"ssrc":372799700
}
]
},
"rtpParameters":{
"codecs":[
{
"clockRate":90000,
"mimeType":"video/VP8",
"parameters":{
},
"payloadType":96,
"rtcpFeedback":[
{
"type":"goog-remb"
},
{
"type":"transport-cc"
},
{
"parameter":"fir",
"type":"ccm"
},
{
"type":"nack"
},
{
"parameter":"pli",
"type":"nack"
}
]
},
{
"clockRate":90000,
"mimeType":"video/rtx",
"parameters":{
"apt":96
},
"payloadType":97,
"rtcpFeedback":[
]
}
],
"encodings":[
{
"active":true,
"rtx":{
"ssrc":681460002
},
"ssrc":372799700
}
],
"headerExtensions":[
{
"id":2,
"uri":"http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time"
},
{
"id":3,
"uri":"http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01"
},
{
"id":13,
"uri":"urn:3gpp:video-orientation"
},
{
"id":14,
"uri":"urn:ietf:params:rtp-hdrext:toffset"
}
],
"mid":"video",
"rtcp":{
"cname":"RD6qXr/tsQyoK0bx",
"mux":true,
"reduceSize":true
}
}
},
"id":4,
"internal":{
"producerId":"123456&&1634628364953_0**video",
"routerId":"123456",
"transportId":"123456&&1634628364953_0"
},
"method":"transport.produce"
}
reply:
{
"accepted":true,
"data":{
"id":"123456&&1634629360309_0**video",
"type":"simple"
},
"id":4
}
2.6 creating an audio Consumer
Creating an audio and video Consumer is the SDP media negotiation with the client when MediaSoup is the sender. It also includes two parts. One part is rtpParameters, which is used to encapsulate the media information of negotiation, mainly including Codec information, SSRC information, extension header information and RTCP information. This part is completely consistent with the creation of Producer. However, it should be noted that during Codec negotiation, Codec type negotiation needs to be the actual type of Producer (the Producer ID is used to indicate that the subscription is on a certain Producer); the other part is consumablertcodes. The SSRC passed here identifies the SSRC of the subscribed Producer.
Request:
{
"data":{
"consumableRtpEncodings":[
{
"ssrc":123
}
],
"kind":"audio",
"paused":false,
"rtpParameters":{
"codecs":[
{
"clockRate":48000,
"mimeType":"audio/opus",
"parameters":{
"minptime":10,
"useinbandfec":1
},
"payloadType":111,
"rtcpFeedback":[
{
"type":"transport-cc"
}
]
}
],
"encodings":[
{
"active":true,
"ssrc":123
}
],
"headerExtensions":[
{
"id":1,
"uri":"urn:ietf:params:rtp-hdrext:ssrc-audio-level"
},
{
"id":2,
"uri":"http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time"
},
{
"id":3,
"uri":"http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01"
}
],
"mid":"audio",
"rtcp":{
"cname":"audio",
"mux":true,
"reduceSize":true
}
},
"type":"simple"
},
"id":11,
"internal":{
"consumerId":"123456&&1634629366108_1**audio",
"producerId":"123456&&1634629360309_0**audio",
"routerId":"123456",
"transportId":"123456&&1634629366108_1"
},
"method":"transport.consume"
}
reply:
{
"accepted":true,
"data":{
"id":"123456&&1634629366108_1**audio",
"paused":false,
"producerPaused":false,
"score":{
"producerScore":10,
"producerScores":[
10
],
"score":10
}
},
"id":11
}
2.7 creating a video Consumer
Request:
{
"data":{
"consumableRtpEncodings":[
{
"ssrc":456
}
],
"kind":"video",
"paused":false,
"rtpParameters":{
"codecs":[
{
"clockRate":90000,
"mimeType":"video/VP8",
"parameters":{
},
"payloadType":96,
"rtcpFeedback":[
{
"type":"goog-remb"
},
{
"type":"transport-cc"
},
{
"parameter":"fir",
"type":"ccm"
},
{
"type":"nack"
},
{
"parameter":"pli",
"type":"nack"
}
]
}
],
"encodings":[
{
"active":true,
"ssrc":456
}
],
"headerExtensions":[
{
"id":2,
"uri":"http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time"
},
{
"id":3,
"uri":"http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01"
},
{
"id":13,
"uri":"urn:3gpp:video-orientation"
},
{
"id":14,
"uri":"urn:ietf:params:rtp-hdrext:toffset"
}
],
"mid":"video",
"rtcp":{
"cname":"video",
"mux":true,
"reduceSize":true
}
},
"type":"simple"
},
"id":12,
"internal":{
"consumerId":"123456&&1634629366108_1**video",
"producerId":"123456&&1634629360309_0**video",
"routerId":"123456",
"transportId":"123456&&1634629366108_1"
},
"method":"transport.consume"
}
reply:
{
"accepted":true,
"data":{
"id":"123456&&1634629366108_1**video",
"paused":false,
"producerPaused":false,
"score":{
"producerScore":10,
"producerScores":[
10
],
"score":10
}
},
"id":12
}
3. AnswerSdp
According to the above MediaSoup processing results, we can encapsulate the AnswerSdp and send it back to the client, mainly including media information and connection information, so as to complete SDP negotiation. There are some differences between the AnswerSdp generated by the media server as the receiving end and the sending end. The main differences are:
a. When the server is the receiving end, it indicates that the direction of the media stream is from the client to the server, which is identified by a=recvonly;
If the server is the sender, here is a=sendonly;
b. When the server is the receiver, the SSRC is brought by the client when sending the OfferSdp, so the replied answersdp does not need to bring the SSRC back; if the server is the sender, the SSRC information must be told to the client through answersdp.
As the receiving end:
v=0 o=- 12345678 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS * m=audio 9 UDP/TLS/RTP/SAVPF 111 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=rtpmap:111 opus/48000/2 # Negotiated audio Codec a=fmtp:111 minptime=10;useinbandfec=1 a=rtcp-fb:111 transport-cc a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=setup:active # The server is the DTLS client a=mid:audio a=recvonly # Here, when the server is the receiver, it means that the direction of media stream is from client to server. If the server is the sender, here is a=sendonly a=ice-lite # Use icelite instead of fullness # The following two lines identify ice ufrag/pwd for ice verification a=ice-ufrag:jqkxbdjer0hwgwzx a=ice-pwd:kb9ecuk7khmdd98uxgqjpf949m3iualz #Certificate fingerprint information a=fingerprint:sha-256 4A:C0:C4:DD:76:7A:B5:1D:CA:73:9F:AB:4A:7C:B8:7C:18:C3:22:A9:B3:41:B9:17:75:AF:E7:4F:91:62:51:3C a=rtcp-mux #rtp and rtcp use the same port a=rtcp-rsize #Support composite rtcp a=candidate:udpcandidate 1 udp 1076302079 192.168.190.140 40649 typ host #Describes the icecandidate information of the server a=end-of-candidates a=ice-options:renomination m=video 9 UDP/TLS/RTP/SAVPF 96 97 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:13 urn:3gpp:video-orientation a=extmap:14 urn:ietf:params:rtp-hdrext:toffset a=setup:active a=mid:video a=recvonly a=ice-lite a=ice-ufrag:jqkxbdjer0hwgwzx a=ice-pwd:kb9ecuk7khmdd98uxgqjpf949m3iualz a=fingerprint:sha-256 4A:C0:C4:DD:76:7A:B5:1D:CA:73:9F:AB:4A:7C:B8:7C:18:C3:22:A9:B3:41:B9:17:75:AF:E7:4F:91:62:51:3C a=rtcp-mux a=rtcp-rsize a=candidate:udpcandidate 1 udp 1076302079 192.168.190.140 40649 typ host a=end-of-candidates a=ice-options:renomination
As sender:
v=0 o=- 12345678 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS * m=audio 9 UDP/TLS/RTP/SAVPF 111 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=rtpmap:111 opus/48000/2 # Negotiated audio Codec a=fmtp:111 minptime=10;useinbandfec=1 a=rtcp-fb:111 transport-cc a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=setup:active # The server is the DTLS client a=mid:audio # Here, when the server is the sender, it means that the direction of the media stream is from the server to the client. If the server is the receiver, here is a=recvonly a=sendonly a=ice-lite # Use icelite instead of fullness # The following two lines identify ice ufrag/pwd for ice verification a=ice-ufrag:2923wzjbycr24lj1 a=ice-pwd:vy3g9ww4zx4f1qcq4rg5mthh8ud4ezh7 #Certificate fingerprint information a=fingerprint:sha-256 4A:C0:C4:DD:76:7A:B5:1D:CA:73:9F:AB:4A:7C:B8:7C:18:C3:22:A9:B3:41:B9:17:75:AF:E7:4F:91:62:51:3C a=rtcp-mux #rtp and rtcp use the same port a=rtcp-rsize #Support composite rtcp a=candidate:udpcandidate 1 udp 1076302079 192.168.190.140 45745 typ host #Describes the icecandidate information of the server a=end-of-candidates a=ice-options:renomination a=msid:2idIZOcnbN7nSMW31bkxGmSAv6F4rGp61Sz3 0c05e5ce-4716-47cd-b3eb-5837ccbe3c9d a=ssrc:123 cname:audio #ssrc of streaming sent by media server a=ssrc:123 msid:2idIZOcnbN7nSMW31bkxGmSAv6F4rGp61Sz3 0c05e5ce-4716-47cd-b3eb-5837ccbe3c9d a=ssrc:123 mslabel:2idIZOcnbN7nSMW31bkxGmSAv6F4rGp61Sz3 a=ssrc:123 label:0c05e5ce-4716-47cd-b3eb-5837ccbe3c9d m=video 9 UDP/TLS/RTP/SAVPF 96 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:13 urn:3gpp:video-orientation a=extmap:14 urn:ietf:params:rtp-hdrext:toffset a=setup:active a=mid:video a=sendonly a=ice-lite a=ice-ufrag:2923wzjbycr24lj1 a=ice-pwd:vy3g9ww4zx4f1qcq4rg5mthh8ud4ezh7 a=fingerprint:sha-256 4A:C0:C4:DD:76:7A:B5:1D:CA:73:9F:AB:4A:7C:B8:7C:18:C3:22:A9:B3:41:B9:17:75:AF:E7:4F:91:62:51:3C a=rtcp-mux a=rtcp-rsize a=candidate:udpcandidate 1 udp 1076302079 192.168.190.140 45745 typ host a=end-of-candidates a=ice-options:renomination a=msid:2idIZOcnbN7nSMW31bkxGmSAv6F4rGp61Sz3 17922525-f0fc-4be0-886c-766b8b78ad46 a=ssrc:456 cname:video a=ssrc:456 msid:2idIZOcnbN7nSMW31bkxGmSAv6F4rGp61Sz3 17922525-f0fc-4be0-886c-766b8b78ad46 a=ssrc:456 mslabel:2idIZOcnbN7nSMW31bkxGmSAv6F4rGp61Sz3 a=ssrc:456 label:17922525-f0fc-4be0-886c-766b8b78ad46
Next, write about the connection process of MediaSoup.