Before the arrival of MPLS and the advance in silicon technology for ASIC and network processors in early 2000s, many carriers ran multiple single-purpose networks such as TDM, ATM and IP to support their video, voice and data network services. Some enterprises attempted to consolidate their triple-play traffic onto a single IP network. However, due to the silicon technology and QoS implementation on IP routers at that time, they were required to over-engineered their IP networks to support peak traffic usage. In other words, they avoided network traffic Quality of Service (QoS) issues by preventing the switches and the network links from being congested at the first place. The IP network became a Multi-Input Multi-Output First-in First-Out network that was underutilized most of the time until peak network usage happened.
In early 2000s, with the advance in silicon technology and networking standards such as MPLS, network vendors could develop network switching platform such as MPLS that could support triple-play traffic with the desired QoS. In other words, the MPLS network can be over-subscribed to take advantage of the burstiness nature of packetized triple-play traffic and when peak network usage happens, QoS constructs provisioned in the network switches can kick in to ensure high priority traffic such as voice can be forwarded to meet its QoS requirements at the expense of discarding low-priority traffic such as Internet data, if necessary.
The following shows a high-level MPLS ePipe service between PE1 and PE2 for a coast-to-coast MPLS network service to illustrate QoS design for triple-play traffic.
One needs to keep in mind the context of the above ePipe service as far as QoS is concerned. A modern PE router can support 100s of access ports and one Ethernet dot1q access port can support more than 4,000 VLAN IDs for different network services. Therefore, the above ePipe service can be one of the 1,000s or 10,000s network services that the terminating PE router needs to forward the traffic out of its handful high-speed network links. When traffic congestion occurs inside the router such as more access traffic need to exit via a particular network port than the bandwidth of the network port, properly designed Intra-switch and Inter-switch QoS can ensure high priority and time-sensitive traffic can be forwarded to meet their QoS requirements at the expense of delaying or even dropping lower priority traffic such as Internet data.
This article describes network traffic QoS implementation in modern network switches/routers for effective and efficient network resources utilization without network over-engineering. I will describe basic network traffic QoS theories such as classification, queueing, scheduling and marking/remarking in the context of the above MPLS ePipe network service. Although Nokia’s Service Routers are used to illustrate the network QoS design, only commonly available QoS constructs are used in this article so that non-Nokia router users can apply the information in this article for their networks. Simple iperf scripts are used for network throughput verification to verify the ePipe’s QoS design.
In order to contain this article into a manageable length, I will skip the ePipe provisioning on Nokia’s Service Routers used in this article. Readers who are interested in setting up MPLS ePipe services on Nokia’s Service Routers can refer to my previous article, MPLS Pseudowire ePipe Interop – Nokia 7750 and Cisco 7200 on GNS3 at https://clcnetwork.wordpress.com/2017/02/24/first-blog-post for more details.
QoS Objective and Domains
The prime objective of network QoS is to maximize the utilization of network switches and links given the burstiness nature of packetized traffic. When network congestion does happen, the provisioned QoS constructs on the switches will kick in to forward higher priority traffic according to its QoS requirements at the expense of discarding lower priority traffic, if necessary.
With reference to the above network diagram, we can broadly divide network traffic QoS into two domains namely the:
- Inter-Switch QoS Definition
- In order to support end-to-end triple-play traffic properly from QoS perspective, every networking device along the data path needs to honour the QoS requirements of each traffic stream. The mechanism to inform the downstream routers about the QoS requirements of each traffic stream is via the QoS definitions of the two interconnecting routers such as MPLS’ EXP, Ethernet’s 802.1p, and IP’s DSCP etc… We will cover QoS Marking and Remarking in details in Parts 1 and 2 of this article.
- Intra-Switch QoS Implementation
- How different ingress or offered network traffic streams being prioritized and QoS processed inside a switch is very vendor specific though general QoS theories and implementation practices such as traffic classification, queuing, scheduling and marking/Remarking etc.. have been well-established for many years and are being applied by network vendors for QoS implementation inside their networking equipment. This is the main focus of this article.
MPLS ePipe’s Triple-Play Traffic QoS Requirements
The coast-to-coast MPLS ePipe service shown in the above diagram supports video, voice and data traffic. Below is the triple-play traffic QoS requirements for this ePipe service:
Red highlights in the above table are Nokia’s Service Router QoS constructs but similar QoS constructs can be found in almost all carrier-grade routers.
The maximum one-way latency of VoIP traffic is 150ms. Therefore, we set the maximum intra-switch latency to 10ms to allow maximum 15 hops for one-way VoIP traffic.
Nokia’s Service Router supports 8 internal Forwarding Classes (FCs). The priorities of the FCs used in this example are EF > H2 > BE. Most traffic schedulers (to be covered in more details later) priorizes their scheduling services based on the priority of the traffic’s FC.
It is found that 8 FCs are sufficient for most network service deployments. Some older or lower-end routers support only 4 FCs and are somewhat limited to support modern traffic QoS operations. The following shows the typical traffic types and their FCs in descending order that a carrier normally supports in its network:
- Network Control (e.g., OSPF, BGP etc….)
- Voice (phone)
- Video conference
- Video unicast and multicast streaming
- Business Data
- Internet Data
Most carriers define their own QoS standard including latency target, network marking schemes etc… to support the above traffic types in their metro, edge and core networks for end-to-end traffic QoS support. We will cover this topic in more details in Part 2 of this article.
Introduction to Network QoS
With the above MPLS ePipe network service and its Triple-Play QoS Requirements, we can now discuss some basic intra-switch QoS theory and implementation.
If we drill down the above ePipe network service from QoS perspective, we can redraw the diagram with QoS constructs such as sap-ingress, sap-egress, network-ingress and network-egress. These QoS constructs are used in Nokia’s Service Routers but similar QoS constructs are available for all carrier-grade routers.
Triple-play traffic is offered at PE’s sap-ingress and is QoS processed (e.g., classification, queueing, scheduling, and marking/remarking) by the PE before forwarding the traffic to its network-egress to the downstream P router’s network-ingress. The destination PE and P routers further QoS process the triple-play traffic until the traffic leave the destination PE’s sap-egress to the end-device in Vancouver or Toronto.
When a traffic stream arrives or offers at say, PE1’s sap-ingress, the sap-ingress needs a way to differentiate whether the traffic is voice, video or data in order to place the traffic into the appropriate queues or buffers for subsequent QoS processing. In this ePipe QoS example, video and voice traffic use UDP port numbers 6789 and 1234 respectively. Any other raffic that do not match these UDP port numbers are considered as data traffic and is QoS processed as Best Effort traffic according to the ePipe’s Triple-Play QoS Requirements.
After traffic classification, each triple-play traffic stream now has it own queue to buffer their traffic. The size of the queues and frequency of scheduler’s visit are determined by the properties of each queue such as their Committed Information Rate (CIR) and Peak Information Rate (PIR) etc….
In the above ePipe QoS example, video traffic (i.e., UDP port = 6789) needs 3.5Mbps CIR. This means that the scheduler will guarante to empty the video traffic Queue 5 at the rate of 3.5Mbps. This is sufficient as long as the video traffic is arriving at the PEs’ Service Access Port (SAP) at the constant rate of 3.5Mbps. Peak Information Rate (PIR) is another QoS construct to drop traffic from the queue when it exceeds PIR for effective line card’s memory sharing among the network ports.
The operations of CIR and PIR are as follows. For example, if 5Mbps video traffic is offered at the PE’s sap-ingress, on the first second, 3.5Mbps of the video traffic will be forwarded as In-Profile where the PE is committed to send the traffic out of its network-egress port to the downstream P router. 1.0Mbps of the video traffic that is above CIR but below PIR will be forwarded as Out-Profile traffic, which can be subject for discard by routers along the data path under switch and/or network link congestion. The remaining 0.5Mbps above PIR traffic could be dropped but if other QoS buffering constructs such as brust-limit is used to temporarily buffer the over PIR traffic, this over PIR video traffic may be able to forward by the scheduler in the next second instead of being discarded.
Beside CIR and PIR, there are other QoS constructs such as Committed Burst Size (CBS) and Maximum Burst Size (MBS) to fine tune the queuing of bursty traffic and we will discuss CBS and MBS in more details later.
Again, all carrier-grade routers have these kind of QoS queuing constructs or capabilities.
Nokia’s Service Routers support many types of traffic schedulers such as the default Priority, Hierarchical, Class Fair Hierarchical, and Egress Port etc….The Priority and Hierarchical Schedulers are the most common traffic schedulers that are supported by almost all router vendors. Therefore, we will cover the Priority and the Hierarchical Schedulers in Parts 1 and 2 of this article respectively.
In the above ePipe QoS example with Priority Scheduler, the scheduler will prioritize the traffic based on the traffic assigned FC. For example, voice traffic having the highest FC priority of EF among the triple-play traffic will be satisfied first by the Priority Scheduler so that voice traffic within CIR can be sent out by the PE’s network-egress port before other traffic is evaluated.
Before traffic is sent out by a PE at its network-egress or sap-egress port, the PE needs a way to advise the downstream device the QoS priority of the traffic as network traffic QoS is end-to-end. Packet marking and remarking takes care of this process. For example, voice traffic having the highest FC priority in the ePipe’s triple-play traffic should have a high QOS marking (e.g., MPLS EXP 5) than the lower priority video traffic (e.g., MPLS EXP 3). Similarly, at PE’s sap-egress, the appropriate Ethernet’s 802.1p markings should be used based on the FC priority of the traffic. We will cover this topic in more details in Part 2 of this article when we discuss carrier QoS scheme.
PE’s sap-ingress and sap-egress QoS Config
We just cover some basic QoS theory. Let us roll up our sleeves and provision sap-ingress and sap-egress for the ePipe using Priority Scheduler to meet the ePipe’s Triple-Play QoS Requirement.
The following shows sap-ingress 50 for ePipe 1. The ip-criteria performs traffic classification to map voice, and video traffic onto Queues 6 and 5 respectively. By default, traffic that does not match the ip-criteria of voice and video goes Queue 1 having Best Effort FC.
A:PE1>config>qos# sap-ingress 50 A:PE1>config>qos>sap-ingress# info ---------------------------------------------- description "Voice EF Q6, Video H2 Q5 & Data BE Q1" queue 1 create rate 5000 exit queue 5 create rate 4500 cir 3500 cbs 22 burst-limit 120000 bytes exit queue 6 create rate 500 cir 500 exit queue 11 multipoint create exit fc "ef" create queue 6 exit fc "h2" create queue 5 exit ip-criteria entry 10 create match protocol udp dst-port eq 1234 exit action fc "ef" priority high exit entry 20 create match protocol udp dst-port eq 6789 exit action fc "h2" exit exit
In sap-ingress 50, we specify rate 5000 (i.e., 5Mbps PIR) for Queue 1 data traffic. Normally, we do not set any PIR for Best Effort (BE) traffic so that BE traffic can take advantage of the available memory in the line cards to buffer BE traffic instead of discarding BE traffic above PIR. This can introduce additional latency to the BE data traffic (i.e., if a queue is long but the scheduling rate is low, packets can be piled up in the buffer that increase traffic latency) but latency is not important for BE data traffic.
According to the Triple-Play QoS Requirement, video traffic is bursty and Nokia’s Service Router have many QoS constructs for handling bursty traffic. We specify 22KBytes Commited Burst Size (CBS) for bursty video traffic based on the below equation to support max video latency of 50ms:
CBS (KBytes) = CIR (bps) * Max Delay (seconds) / 8000
This is the inital CBS setting and we will adjust CBS, Maximum Burst Size (MBS), CIR, and burst-limit etc… during QoS testing to ensure the settings match the burstiness of the traffic.
The following shows sap-egress 50 for ePipe 1 that is applied to PE1 and PE2. If we are sure that sap-ingress 50 is in use when the traffic is offered to the network, we do not need to repeat the CIR and PIR settings at sap-egress 50. Note that Ethernet 802.1p markings (e..g, Ethernet dot1p 4 for video) are set at sap-egress 50 to indicate to the downstream device the priority of the traffic.
A:PE1>config>qos# sap-egress 50 A:PE1>config>qos>sap-egress# info ---------------------------------------------- description "Voice EF Q6, Video H2 Q5 & Data BE Q1" queue 1 create rate 5000 exit queue 5 create rate 4500 cir 3500 exit queue 6 create rate 500 cir 500 exit fc be create queue 1 dot1p 0 exit fc ef create queue 6 dot1p 7 exit fc h2 create queue 5 dot1p 4 exit
We now have sap-ingress 50 and sap-egress 50 defined with the appropriate queue sizes, CIR and PIR etc… to match the ePipe’s Triple-Play QoS Requirement. This article (Part 1) uses the default Priority Scheduler and we will compare it with the Hierarchical Scheduler in Part 2 of this article.
The following shows sap-ingress 50 and sap-egress 50 being applied to the ePipe service on PE1 and PE2. Since the Priority Scheduler is the default scheduler on Nokia’s Service Router, there is no specific Priority Scheduler CLI keyword in the config.
A:PE1>config>service# epipe 1 A:PE1>config>service>epipe# info ---------------------------------------------- sap 2/1/10 create ingress qos 50 exit egress qos 50 exit exit spoke-sdp 2:100 create no shutdown exit no shutdown
SAP QoS Verification
We have just completed sap-ingress 50 and sap-egress 50 provisioning for ePipe 1 using the default Priority Scheduler. It is time for us to verify whether the triple-play traffic over the ePipe can meet its Triple-Play QoS Requirements.
There are many ways to generate IP traffic to test the QoS behaviours of a network such as the open source traffic generator, Ostinato etc… For quick QoS testing, iperf running on two PCs at each end of the ePipe’s SAP will do the trick.
Connect PC1 (192.168.1.1) and PC2 (192.168.1.2) to the SAP port 2/1/10 on PE1 and PE2 for sending and receiving iperf traffic respectively over the ePipe.
On PC2 connecting to PE2, run the iperf script, rx_all to receive 3 streams of iperf traffic representing voice, video and data traffic respectively.
[root@PC1 ~]# cat rx_all #!/bin/bash # receive voice traffic - FC Q6 EF (pir 0.5M, cir 0.5M) xterm -e bash -c 'iperf -u -s -p 1234; bash' & # video traffic - FC Q5 H2 (pir 4.5M, cir 3.5M) xterm -e bash -c 'iperf -u -s -p 6789; bash' & # data and others traffic - FC Q1 BE (pir 5M) xterm -e bash -c 'iperf -u -s; bash' &
Under CIR Traffic Verification
Once rx_all is run on PC2, we can now run different iperf scripts on PC1 to verify the ePipe’s QoS properities.
On PC1, run the below iperf script, tx_within_cir to generate 0.5Mbps voice, 3.5Mbps video, and 5Mbps data to PC2 over the ePipe.
[root@PC1 ~]# cat tx_within_cir #!/bin/bash # voice traffic - FC Q6 EF (pir 0.5M, cir 0.5M) iperf -u -c 192.168.1.2 -b 0.5M -t 20 -p 1234 & #video traffic - FC Q5 H2 (pir 4.5M, cir 3.5M) iperf -u -c 192.168.1.2 -b 3.5M -t 20 -p 6789 & # data and others traffic - FC Q1 BE (pir 5M) iperf -u -c 192.168.1.2 -b 5.0M -t 20 &
Below Nokia’s SROS commands running on PE1 shows the triple-play traffic received by PE1’s sap-ingress 50 for the ePipe:
A:PE1# clear service statistics sap 2/1/10 all A:PE1# show service id 1 sap 2/1/10 detail << skip >> ------------------------------------------------------- Sap per Queue stats ------------------------------------------------------- Packets Octets Ingress Queue 1 (Unicast) (Priority) Off. HiPrio : 0 0 Off. LowPrio : 8509 12895818 Dro. HiPrio : 0 0 Dro. LowPrio : 204 309264 For. InProf : 0 0 For. OutProf : 8305 12586554 Ingress Queue 5 (Unicast) (Priority) Off. HiPrio : 0 0 Off. LowPrio : 5955 9027780 Dro. HiPrio : 0 0 Dro. LowPrio : 0 0 For. InProf : 5860 8883760 For. OutProf : 95 144020 Ingress Queue 6 (Unicast) (Priority) Off. HiPrio : 855 1296180 Off. LowPrio : 0 0 Dro. HiPrio : 0 0 Dro. LowPrio : 0 0 For. InProf : 855 1296180 For. OutProf : 0 0
The traffic classification process, ip-criteria maps the data, video and voice traffic to Queues 1, 5 and 6 respectively.
Due to the burstiness of the iperf traffic, 204 data packets that are momentarily exceeded the 5Mbps PIR are dropped at sap-ingress 50. As discussed before, we normally do not specify any PIR for BE traffic. If rate max instead of rate 5000 is used for Queue 1, there will be no lost of data packet on sap-ingress 50.
95 video packets are forwarded as Out-Profile (i.e., momentarily exceeded 3.5Mbps CIR with 22Kbytes CBS but below the 4.5Mbps PIR). These 95 Forward Out-Profile packets can be subject for discard if the switch experienced any congestion.
855 voice packets are received by PE1’s sap-ingress 50 as high-priority traffic and since voice traffic is using the highest FC of EF among the triple-play traffic and the offered traffic is within the CIR for Queue 6 voice traffic, all voice traffic are marked as Forward In-Profile traffic and PE1 is committed to forward this In-Profile traffic to its network-egress port to the downstream router, P.
sap-egress 50 on PE2 confirms that all Forward In-Profile and Forward Out-Profile traffic of PE1 are received and delivered to the end-device. This suggests that there is no network congestion after PE1’s network-egress.
B:PE2# clear service statistics sap 2/1/10 all B:PE2# show service id 1 sap 2/1/10 detail << skip >> -------------------------------------------------------- Sap per Queue stats -------------------------------------------------------- Packets Octets Egress Queue 1 For. InProf : 0 0 For. OutProf : 8305 12586554 Dro. InProf : 0 0 Dro. OutProf : 0 0 Egress Queue 5 For. InProf : 5955 9027780 For. OutProf : 0 0 Dro. InProf : 0 0 Dro. OutProf : 0 0 Egress Queue 6 For. InProf : 855 1296180 For. OutProf : 0 0 Dro. InProf : 0 0 Dro. OutProf : 0 0
Above PIR Traffic Verification
Now on PC1, run the iperf script, tx_above_pir to send 0.5Mbps voice, 7Mbps video, and 10Mbps data traffic to PC2. Note that the offered traffic for video and data traffic are 2.5Mbps and 5Mbps above their PIRs respectively.
[root@PC1 ~]# cat tx_above_pir #!/bin/bash # voice traffic - FC Q6 EF (pir 0.5M, cir 0.5M) iperf -u -c 192.168.1.2 -b 0.5M -t 20 -p 1234 & #video traffic - FC Q5 H2 (pir 4.5M, cir 3.5M) iperf -u -c 192.168.1.2 -b 7M -t 20 -p 6789 & # data and others traffic - FC Q1 BE (pir 5M) iperf -u -c 192.168.1.2 -b 10M -t 20 &
Below shows PE1’s sap-ingress 50 after the completion of the iperf script, tx_above_pir:
A:PE1# clear service statistics sap 2/1/10 all A:PE1# show service id 1 sap 2/1/10 detail << skip >> ------------------------------------------------------- Sap per Queue stats ------------------------------------------------------- Packets Octets Ingress Queue 1 (Unicast) (Priority) Off. HiPrio : 0 0 Off. LowPrio : 17013 25787882 Dro. HiPrio : 0 0 Dro. LowPrio : 8708 13201328 For. InProf : 0 0 For. OutProf : 8305 12586554 Ingress Queue 5 (Unicast) (Priority) Off. HiPrio : 0 0 Off. LowPrio : 11907 18051012 Dro. HiPrio : 0 0 Dro. LowPrio : 4385 6647660 For. InProf : 5869 8897404 For. OutProf : 1653 2505948 Ingress Queue 6 (Unicast) (Priority) Off. HiPrio : 855 1296180 Off. LowPrio : 0 0 Dro. HiPrio : 0 0 Dro. LowPrio : 0 0 For. InProf : 855 1296180 For. OutProf : 0 0
17,013 data packets (i.e., 10Mbps) are offered at Queue 1 and 8,708 (or 51%) packets are dropped. This matches the 5Mbps PIR (i.e., rate 5000) of data traffic in sap-ingress 50.
11,907 video packets (i.e., 7Mbps) are offered at Queue 5 and 5,869 packets (or 49% or half of 7Mbps) are forwarded as In-Profile. This matches the 3.5Mbps CIR of video traffic in sap-ingress 50.
4,385 video packets (or 37% of 7Mbps) are dropped. This matches the 2.5Mbps over PIR setting of video traffic in sap-ingress 50.
No voice traffic is dropped as only 0.5Mbps (CIR rate) voice traffic is offered at Queue 6 of sap-ingress 50.
Intra-switch QoS design for network sevices is a complex topic. In this article, we touch upon some basic network traffic QoS settings. Although Nokia’s Service Routers are used to illustrate the QoS design, similar QoS constructs are available for almost all carrier-grade routers from other vendors.
Priority Scheduler forwards traffic to the switch’s network-egress port based on the priority of the FC assigned to the traffic followed by CIR and then PIR. In the case of voice traffic in the above example, since it is assigned with the highest FC priority among the triple-play traffic and the offered traffic is within its CIR, all voice traffic are sent and then received by PE2′ sap-egress 50 without any loss.
We have not yet covered any network-ingress and network-egress QoS constructs. For example, when voice traffic leaves PE1’s network-egress port, what MPLS EXP it is assigned by PE1 so that the downstream P router can process voice traffic accordingly from QoS perspective? Note that FCs are internal to a switch and are not sent to other routers.
All carriers have their own QoS specifications in terms of latency, network QoS marking etc… to support different traffic on their network. We may need to modify network-ingress and network-egress QoS settings to match the carrers’ network QoS specifications.
Beside Priority Scheduler, Hierarchical Scheduler is the 2nd most commonly used traffic scheduler in carrier-grade routers. Hierarchical Scheduler allows bandwidth sharing such that if there is no video and voice traffic for a given moment, data traffic can utilize the idle bandwidth left by video and voice for maximum network resource utilization.
We will cover all the above unanswered questions in part 2 of this article.