When does the NACK and RTX of WebRTC judge to send NACK request and RTX packet loss retransmission in case of packet loss

When does the NACK and RTX of WebRTC judge to send NACK request and RTX packet loss retransmission in case of packet loss

WebRTC special topic open hi duck!!!

1, WebRTC threading model

1. Introduction to thread model and common thread models in WebRTC

2. WSAEventselect model usage of PhysicalSocketServer in WebRTC network

2, WebRTC media negotiation

3, WebRTC audio data acquisition

4, WebRTC audio engine (codec and 3A algorithm)

5, WebRTC video data acquisition

6, WebRTC video engine (codec)

7, WebRTC network transmission

1. STUN protocol of ICE of WebRTC

2. Dtls/ssl/tlsv1 X protocol details

8, WebRTC quality of service (Qos)

1. Detailed explanation of RTCP protocol in WebRTC

2. Detailed explanation of RTP protocol in WebRTC

3. When does the NACK and RTX of WebRTC judge to send NACK request and RTX packet loss retransmission in case of packet loss

9, NetEQ

10, Simulcast and SVC


NACK is used to judge whether the network is lost and retransmitted, and network conditions

1, Role of NACK and RTX

1. NACK is used to notify which packets are lost
2. RTX is used to retransmit lost packets

Whether Nack and RTX protocols are normal in sending offer and answer

a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96

1. Flow chart of NACK/RTX working mechanism

2. Several problems involved in NACK/RTX

1. How to determine whether the algorithm sends a packet loss
When the Sequence Number is not continuous, the packet is lost
2. When will NACK be sent
3. What is the format of NACK and how to handle it when the sender receives NACK?
4. What is the RTX format and how do RTX and NACK work together?

2, Key algorithms for determining packet location

1. A key function: AheadOf

1. AheadOf(a,b), compare the order relationship between a and B
2. a and b must be unsigned integers
3. If bar is ahead of a, it returns true; otherwise, it returns false
4. It should be noted that they are not a simple comparison of numerical values

3, Process flow chart of NACK in WebRTC

1. NackMoudle's creation opportunity

Create NackMoudle class in the constructor of RtpVideoStreamReceiver

RtpVideoStreamReceiver is created in the constructor of VideoReceiveStream class

1. NACK call stack

4, Logic of how WebRTC determines whether packet loss occurs

1. NackModule::OnReceivedPacket function


int NackModule::OnReceivedPacket(uint16_t seq_num,
                                 bool is_keyframe,
                                 bool is_recovered) {
  rtc::CritScope lock(&crit_);
  // TODO(philipel): When the packet includes information whether it is
  //                 retransmitted or not, use that value instead. For
  //                 now set it to true, which will cause the reordering
  //                 statistics to never be updated.
  bool is_retransmitted = true;
  // 1. determine whether to exit after initialization for the first time
  if (!initialized_) {
    newest_seq_num_ = seq_num;
    if (is_keyframe)// Is this package a key frame = = < why should I identify a key frame? >???
    initialized_ = true;
    return 0;

  // Since the |newest_seq_num_| is a packet we have actually received we know
  // that packet has never been Nacked.
  // 2. if this time's seq is the same as the last time, and it is a duplicate package, exit
  if (seq_num == newest_seq_num_)
    return 0;
  // That is to say, if the first package and the duplicate package are not selected, the package order will be determined_ Num in newest_seq_num will be deleted before  
  // 3. if the previous package was processed last time, the package has expired. If it is still in the nack list, it needs to be deleted
  // That means the package arrived late 
  if (AheadOf(newest_seq_num_, seq_num)) {
    // An out of order packet has been received.
    auto nack_list_it = nack_list_.find(seq_num);
    int nacks_sent_for_packet = 0;
    if (nack_list_it != nack_list_.end()) {
      nacks_sent_for_packet = nack_list_it->second.retries;
    if (!is_retransmitted)
    return nacks_sent_for_packet;

  // Keep track of new keyframes.
  // 4. if it is judged whether it is a key frame??? Ha
  if (is_keyframe)
    keyframe_list_.insert(seq_num); // If the message belongs to the key frame, keep it

  // And remove old ones so we don't accumulate keyframes.
  // 5. find the minimum boundary point, and delete the previous data if it exceeds 10000. This is a real-time system
  auto it = keyframe_list_.lower_bound(seq_num - kMaxPacketAge);
  if (it != keyframe_list_.begin())
    keyframe_list_.erase(keyframe_list_.begin(), it);
  // 6. how to judge whether the returned package??? Recovery package
  if (is_recovered) {
    recovered_list_.insert(seq_num); // If the packet belongs to the key frame, keep it

    // Remove old ones so we don't accumulate recovered packets.
	//  Are there any out of range items? The out of range items have also been deleted. The maximum item is 10000
    auto it = recovered_list_.lower_bound(seq_num - kMaxPacketAge);
    if (it != recovered_list_.begin())
      recovered_list_.erase(recovered_list_.begin(), it);

    // Do not send nack for packets recovered by FEC or RTX.
    return 0;
  // 7. what will happen here 
  //     1. Not the first package
  //     2. not a duplicate package
  //     3. Not on New_ Seq_ Packages before num
  //     4. Not a recovery package
  //   There are two situations that will come this way
  //     1. A packet after the last processed packet ha ordered packet
  //     2. Several packets are separated from the last processed packet   
  AddPacketsToNack(newest_seq_num_ + 1, seq_num);
  newest_seq_num_ = seq_num;

  // Are there any nacks that are waiting for this seq_num.
  // 8. tell the other party to resend the packets that are really lost
  std::vector<uint16_t> nack_batch = GetNackBatch(kSeqNumOnly);
  if (!nack_batch.empty()) 
    nack_sender_->SendNack(nack_batch);  //  It needs to be retransmitted to the buffer??????

  return 0;

2. NackModule::AddPacketsToNack preliminarily determines which packets are lost

void NackModule::AddPacketsToNack(uint16_t seq_num_start,
                                  uint16_t seq_num_end) {
  // Remove old packets.
  auto it = nack_list_.lower_bound(seq_num_end - kMaxPacketAge);
  nack_list_.erase(nack_list_.begin(), it);

  // If the nack list is too large, remove packets from the nack list until
  // the latest first packet of a keyframe. If the list is still too large,
  // clear it and request a keyframe.
  // 1. how far is the distance between the beginning and the end 
  uint16_t num_new_nacks = ForwardDiff(seq_num_start, seq_num_end);
  if (nack_list_.size() + num_new_nacks > kMaxNackPackets) {
    while (RemovePacketsUntilKeyFrame() &&
           nack_list_.size() + num_new_nacks > kMaxNackPackets) {
	// 1.1 in extreme cases, if it is not deleted, clear the nack, and then send the request key frame to the other party to make the decoder work again
    if (nack_list_.size() + num_new_nacks > kMaxNackPackets) {
      RTC_LOG(LS_WARNING) << "NACK list full, clearing NACK"
                             " list and requesting keyframe.";
  // 2. Traverse seq_num_start to seq_ Num_ Whether there is packet loss between the end and the nack_list_ China Kazakhstan
  for (uint16_t seq_num = seq_num_start; seq_num != seq_num_end; ++seq_num) {
    // Do not send nack for packets that are already recovered by FEC or RTX
	// 2.1 whether the package has been recovered through FEC or RTX. Once recovered, it does not need to be put into nack_list_ Go to the list
    if (recovered_list_.find(seq_num) != recovered_list_.end())
    NackInfo nack_info(seq_num, seq_num + WaitNumberOfPackets(0.5),
    RTC_DCHECK(nack_list_.find(seq_num) == nack_list_.end());
    nack_list_[seq_num] = nack_info;

3. GetNackBatch (function to determine packet loss)

// Traverse all suspicious packets and insert NACK if the packets meet the conditions_ Batch
std::vector<uint16_t> NackModule::GetNackBatch(NackFilterOptions options) {
	// 1. mark with seq_num is the judgment condition
  bool consider_seq_num = options != kTimeOnly;
  // 2. the identification is judged by timestamp 
  bool consider_timestamp = options != kSeqNumOnly;
  int64_t now_ms = clock_->TimeInMilliseconds();
  std::vector<uint16_t> nack_batch;
  auto it = nack_list_.begin();
  while (it != nack_list_.end()) {
	  // 1. send_nack_delay_ms_ 0 by default, modifiable
    bool delay_timed_out = now_ms - it->second.created_at_time >= send_nack_delay_ms_;
	// 2. whether the duration of an RTT loop has exceeded since the first transmission  
	// Need to get an RTT to prevent repeated transmission 
    bool nack_on_rtt_passed = now_ms - it->second.sent_at_time >= rtt_ms_;
	// 3. Before the first sending and the last processing of packets
    bool nack_on_seq_num_passed = it->second.sent_at_time /*If it is the first time to send*/== -1 &&
        AheadOrAt(newest_seq_num_, it->second.send_at_seq_num)/*The package is before the last processed package*/;
	// Qualified
    if (delay_timed_out && ((consider_seq_num && nack_on_seq_num_passed) ||
                            (consider_timestamp && nack_on_rtt_passed))) {
      it->second.sent_at_time = now_ms;
	  // Try 10 times at NACK_ The list will be deleted if it is not found
      if (it->second.retries >= kMaxNackRetries/*kMaxNackRetries= 10*/) {
        RTC_LOG(LS_WARNING) << "Sequence number " << it->second.seq_num
                            << " removed from NACK list due to max retries.";
        it = nack_list_.erase(it);
      } else {
  return nack_batch;

Judgment conditions

4. Periodically executed function NackModule::Process

void NackModule::Process() {
  if (nack_sender_) {
    std::vector<uint16_t> nack_batch;
      rtc::CritScope lock(&crit_);
      nack_batch = GetNackBatch(kTimeOnly);

    if (!nack_batch.empty())

  // Update the next_process_time_ms_ in intervals to achieve
  // the targeted frequency over time. Also add multiple intervals
  // in case of a skip in time as to not make uneccessary
  // calls to Process in order to catch up.
  int64_t now_ms = clock_->TimeInMilliseconds();
  if (next_process_time_ms_ == -1) {
    next_process_time_ms_ = now_ms + kProcessIntervalMs;
  } else {
    next_process_time_ms_ = next_process_time_ms_ + kProcessIntervalMs +
                            (now_ms - next_process_time_ms_) /
                                kProcessIntervalMs * kProcessIntervalMs;

5. Both the Process function and the OnReceivedPacket function have the purpose of sending the nack function

  1. One is the time to find lost packets
  2. An seq_ Find lost packets in num order

How is this WebRTC designed in the m74 version? I'll give you how to design this packet loss retransmission

5, Determination of VP8 key frames in WebRTC

1. VP8 RTP structure diagram

2. VP8 Payload structure diagram

3. Two structures of VP8 Payload descriptor

3.1 VP8 Payload differences and field meanings

1. The difference between the two

pictureID:  First 7 bits second 15 bits

2. Field meaning

X: Represents whether the following line has extension [I|L|T|K|RSV]
R: Reserved fields
N:  1: Represents non reference frame, 0: is reference frame
S: Whether it belongs to video frame fragment 1: the first fragment of video frame 0: other fragments
R: reserved 
PID: Indicates the serial number of the slice, and the maximum is no more than 8

3.2 extended bytes in descriptor

1. I, the position 1, indicates that there is a PictureID, which immediately follows the extended byte
2. L, the position 1, indicates that TLOPICIDX exists, which follows PictureID, and the T bit must be set to 1
3. T, the position is 1 or (T=0, its K=1), TID/Y/KEYIDX byte exists, otherwise it does not exist
4. K, set to 1, TID/Y/KEYIDX byte exists; T=1 and K=0, KEYIDX field is ignored; Otherwise TID/Y/KEYIDX bytes do not exist
5. SRV, reserved, must be set to 0

Points needing attention
1. The PictureID field consists of M and PictureID.
M=1:PictureID occupies 15 bits;
M = 0: 7 digits
2. TID= 0, TLOPICIDX represents the running index of the time base layer frame
Video layering
3. Tid> 0:TLOPICIDX indicates the base layer frame that the current image depends on
4. In the TID/Y/KEYIDX field, TID occupies two bits, Y occupies one bit, and KEYIDX occupies five bits
TID: represents the time layer. The base layer is 0. The higher the level, the greater the value
Y: Layer synchronization bit, set to 1, indicating that the current frame only depends on the base layer frame (TIDO); Setting 0 means that the current frame does not depend on the base frame
KEYIDX:key frame index value

4,VP8 Payload Header

4.1. VP8 Payload header field meaning

1, Header
1. For internal frames, this field occupies 3 bytes; For key frames, this field occupies 10 bytes
2. The structure of the first three bytes is universal
3. This Header only exists when the upper S position bit and PID=0
2, Field meaning
1. P, 1 bit, indicating frame type: 0-key, 1-infterframe key frame
2. VER, 3 bits, 1-3 defines four profile s with different decoding complexity
3. SIze, 19 bits, SIze of the first fragment byte

5. Meaning of 7 bytes multiplied by KeyFrame Header

1. 3-byte start code: fixed value: 0x9D, 0x01,0x2A
2. Next 16 bits: (2bit horizontal scale < < 14) | Width (14bits)
3. Last 16 bits: (2bits vertical scale < < 14) | Height (14bits)


WebRTC source code analysis address: https://github.com/chensongpoixs/cwebrtc

Tags: webrtc

Posted by brattt on Tue, 31 May 2022 05:03:20 +0530