Testing Unknown Unicast Limiter Function of Enterprise Switches

Ethernet switches flood frames with a destination MAC address that is not found in the switch's CAM table to all ports (in the same VLAN) except the ingress port. This is known as unknown unicast flooding.

In case of a loop in the Ethernet, every flooded frame will be duplicated as fast as the switches can, which may use up all available bandwidth. The best known example is a so called broadcast storm, but the same phenomenon occurs with multicast and unknown unicast frames.

Some Ethernet switches implement unknown unicast flooding in software, which may result in maxed out CPU during an unknown unicast storm, impeding normal switch functions (e.g. running STP) as well as management access. See my Notes on Enterasys Networks Equipment page for additional information.

To mitigate the impact of broadcast, multicast, and unknown unicast storms, many enterprise class switches implement rate limiters for these frames. This function has different names, e.g., storm-control (Cisco), flood-ctrl (Enterasys), or rate-limit flood (Extreme).

It is usually unproblematic to install broadcast limiters, because very few software requires high data or packet rate broadcast communication. Multicast is often used to transport high data rate traffic that must not be dropped as long as the network provides enough bandwidth, thus it is not possible to generally implement multicast limiters. Unknown unicasts should be transient, because the switches should learn the MAC address from return traffic. A common reason for sustained unknown unicast flooding is an interaction of asymmetric routing or load balancing with ARP cache entry lifetime greater than CAM table entry lifetime on the layer 3 / layer 2 border. Another common cause are cluster systems like Microsoft NLB in unicast mode. As a result unknown unicast limiters are seldomly deployed, because the network team does not know if any applications (ab)use the unknown unicast flooding feature of switched Ethernet networks.

This document reports on tests of unknown unicast limiter functions of several different enterprise class switches. The switch selection depends only on availability of a switch for me to test. The installed firmware was kept. Missing or broken unknown unicast limiter functionality might be documented in release notes or bug trackers, but this was not checked. It is advisable to check release notes and bug trackers regarding the features that shall be used in a network. Building a test network and verifiying that a specific combination of hard- and software works as planned is important as well.

Test Setup

 +----------+              +----------+
 |          |1         eth2| Client 1 |
 |  Device  +--------------+ 10.0.0.1 |
 |   under  |              +----------+
 |   test   |              +----------+
 |  (DUT)   +--------------+ Client 2 |
 |          |2         eth3| 10.0.0.2 |
 +----------+              +----------+

Both Client 1 and Client 2 are implemented using Linux Network Namespaces. They are connected to the DUT via two USB Fast Ethernet NICs (eth2 and eth3).

Linux Network Configuration

Two network namespaces are used to force traffic from one NIC to the other to use the device under test (DUT).

I have used the iproute2 utility to configure the NICs, and to set up and use the namespaces. The official iproute2 documentation is quite bad, but a nice iproute2 cheat sheet is available from Daniil Baturin. An alternative solution to send traffic between two local interfaces of a Linux system through an external device is described in a blog entry from Google's Project Zero on exploiting FireEye security appliances.

sudo ip netns add CLIENT1
sudo ip netns add CLIENT2
sudo ip link set dev eth2 netns CLIENT1
sudo ip link set dev eth3 netns CLIENT2
sudo ip netns exec CLIENT1 ip address add 10.0.0.1/24 dev eth2
sudo ip netns exec CLIENT2 ip address add 10.0.0.2/24 dev eth3
sudo ip netns exec CLIENT1 ip link set dev eth2 up
sudo ip netns exec CLIENT2 ip link set dev eth3 up

Testing Connectivity

Connectivity between network namespaces CLIENT1 and CLIENT2 can be tested using ping. Using a small intervall between ping packets can be used to check if high packet rates result in (known) unicast rate limiting.

sudo ip netns exec CLIENT1 ping -c10000 -i.0001 -q 10.0.0.2
sudo ip netns exec CLIENT2 ping -c10000 -i.0001 -q 10.0.0.1

Creating Traffic

Iperf

Iperf in UDP mode is used to generate traffic, with the server in namespace CLIENT1.

sudo ip netns exec CLIENT1 iperf -u -s

An iperf client in namespace CLIENT2 is used to create traffic from CLIENT2 to CLIENT1. When connecting to CLIENT1's IP address 10.0.0.1, the iperf server shows the received amount of data.

sudo ip netns exec CLIENT2 iperf -u -c10.0.0.1 -b100M -t10

Multi-Generator

The Multi-Generator (MGEN) program is an alternative to using Iperf for traffic generation. It allows fine grained control over packet rate and size. While MGEN usage is in general based on MGEN Script files, everything needed for simple rate-limter tests is available on the command line. To create an MGEN receiver in namespace CLIENT1 and a 200 pps sender (using UDP packets with 64B payload) in namespace CLIENT2, use the following commands:

sudo ip netns exec CLIENT1 mgen port 5001
sudo ip netns exec CLIENT2 mgen event "on 1 udp src 5001 dst 10.0.0.2/5001 periodic [200 64]"

Creating Unknown Unicast Frames

To create unknown unicast frames, invalid ARP entries are added to network namespace CLIENT2.

sudo ip netns exec CLIENT2 ip neighbour add 10.0.0.3 lladdr 02:33:33:33:33:33 dev eth3

Sending data from CLIENT2 to 10.0.0.3 creates unknown unicast flooding. Tcpdump in namespace CLIENT1 can be used to verify that the frames actually reach CLIENT1, that is they are flooded by the switch.

sudo ip netns exec CLIENT2 iperf -u -c10.0.0.3 -b100M -t10

Cleaning Up

To clean up the GNU/Linux system after testing, just delete the network namespaces created for testing.

sudo ip netns delete CLIENT1
sudo ip netns delete CLIENT2

Test Procedure

The DUT's configuration is cleared, the (virtual) clients connected to two, ports, and connectivity is tested with ping. There should be no packet loss.

Next, iperf using the correct IP addresses is used to get a baseline. The DUT's CPU usage should stay low, since frame forwarding should be done by hardware.

The unknown unicast flooding is tested using iperf. If the DUT implements unknown unicast flooding in software, the CPU usage should be significantly higher than in the preceding test.

After these initial tests, unknown unicast rate limiting is configured. The three tests above are repeated. The first two should show the same results since they primarily use (known) unicast forwarding in the DUT. The third should show dropped packets at the switch, and possibly reduced CPU usage for software based flooding. This can be verified using show commands on the DUT, observing the port LEDs of the DUT, and tcpdump on CLIENT1.

sudo ip netns exec CLIENT1 tcpdump -i eth2 -n -l -q -t host 10.0.0.3

The tests shall show if unknown unicast rate limiting is working and if it affects normal unicast traffic as well. The tests do not verify the actual traffic rate permitted by the rate limiters.

Tests

The test results follow. Only unknown unicast flooding behaviour is tested. The switches are sorted alphabetically by name (manufacturer, model). Each test includes information on how to configure an unknown unicast rate limit, if the switch actually supports to limit unknown unicast frames. In some cases, this limit might affect all unicast frames, not just unknown unicast.

Alcatel OmniSwitch 6850

The DUT is an Alcatel OmniSwitch 6850-P48 layer 3 switch using AOS firmware version 6.1.2.175.R03. The switch is based on a PowerPC CPU and unknown switching ASICs.

Rate limiting for unknown unicast (and all other flooded) frames is configured by specifying the maximum rate in Mbps as follows:

interfaces 1/1 flood rate 1
interfaces 1/2 flood rate 1

Rate limiting is enabled by default, with a maximum bandwidth of slightly below half the interface bandwidth. To test without rate limiting, the flood bandwidth is set to the interface bandwidth.

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Arista Networks 7050T-64

The DUT is an Arista Networks DCS-7050T-64 layer 3 switch using EOS firmware version 4.14.9M. The switch is based on the Broadcom Trident+ and an i386 compatible CPU.

Rate limiting for unknown unicast frames is not supported.

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting is not supported.

Cisco Systems Catalyst 2960G

The DUT is a Cisco Systems WS-C2960G-8TC-L layer 2 switch using IOS firmware version 12.2(53)SE1 (LAN Base). The switch is based on a PowerPC CPU and proprietary switching ASICs.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

configure terminal
interface range GigabitEthernet 0/1 - 2
storm-control unicast level pps 100
end

This is not really rate limiting, as the port will start blocking unicast traffic if the rising threshold is crossed and re-open only after the falling threshold is crossed.

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting is not supported, as all unicast frames are affected by the "storm-control" feature.

When a "storm" is detected, Cisco IOS blocks all traffic of the type causing the storm for as long as the packet rate is at or above the lower threshold, instead of rate limiting.

Cisco Systems Catalyst 3550

The DUT is a Cisco Systems WS-C3550-24-SMI layer 3 switch using IOS firmware version 12.2(50)SE3 (IP Services). The switch is based on a PowerPC CPU and proprietary switching ASICs.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

configure terminal
interface range FastEthernet 0/1 - 2
storm-control unicast level pps 100
end

This is not really rate limiting, as the port will start blocking unicast traffic if the rising threshold is crossed and re-open only after the falling threshold is crossed.

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting is not supported, as all unicast frames are affected by the "storm-control" feature.

When a "storm" is detected, Cisco IOS blocks all traffic of the type causing the storm for as long as the packet rate is at or above the lower threshold, instead of rate limiting.

Cisco Systems Catalyst 3560

The DUT is a Cisco Systems WS-C3560-8PC layer 3 switch using IOS firmware version 15.0(2)SE1 (IP Services). The switch is based on a PowerPC CPU and proprietary switching ASICs.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

configure terminal
interface range FastEthernet 0/1 - 2
storm-control unicast level pps 100
end

This is not really rate limiting, as the port will start blocking unicast traffic if the rising threshold is crossed and re-open only after the falling threshold is crossed.

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting is not supported, as all unicast frames are affected by the "storm-control" feature.

When a "storm" is detected, Cisco IOS blocks all traffic of the type causing the storm for as long as the packet rate is at or above the lower threshold, instead of rate limiting.

Cisco Systems Nexus 5672UP (third party testing)

The DUT is a Cisco Systems N5K-5 5672UP layer 3 switch using NX-OS firmware version 7.0(4)N1(1). The switch is based on an Intel Xeon CPU and Broadcom switching ASICs.

Rate-limiting to 1% of link bandwidth (PPS rate limiting is not supported) for unknown unicast frames is implemented with the commands:

configure terminal
interface Ethernet 1/1
storm-control unicast level 1
interface Ethernet 1/2
storm-control unicast level 1
end

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works.

Note that this test was conducted by a third party with a different test setup. Not all the tests done for the other devices were replicated and reported exactly. Thus the test results are not directly comparable.

Enterasys Networks 800

The DUT is an Enterasys Networks 08G20G2-08P layer 3 switch using firmware version 01.02.02.0006. The switch is based on some Broadcom SoC.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the command:

config traffic control 1-2 unicast enable threshold 100

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Enterasys Networks C2

The DUT is an Enterasys Networks C2H124-48 layer 3 switch using EOS firmware version 05.02.17.0006. The switch is based on the Broadcom BCM5665 REV 18.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

set cos port-resource flood-ctrl 0.0 unicast rate 100
set cos state enable

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Class-of-Service processing needs to be enabled for unknown unicast rate limiting.

Enterasys Networks C5

The DUT is an Enterasys Networks C5G124-24 layer 3 switch using EOS firmware version 06.61.02.0007. The switch is based on the Broadcom BCM56620 REV 18.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

set cos port-resource flood-ctrl 0.0 unicast rate 100
set cos state enable

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Class-of-Service processing needs to be enabled for unknown unicast rate limiting.

Enterasys Networks D2

The DUT is an Enterasys Networks D2G124-12 layer 2 switch using EOS firmware version 06.03.01.0008. The switch is based on the Broadcom BCM5698 REV 17.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

set cos port-resource flood-ctrl 0.0 unicast rate 100
set cos state enable

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Class-of-Service processing needs to be enabled for unknown unicast rate limiting.

Enterasys Networks E1

The DUT is an Enterasys Networks 1H582-25 layer 3 switch using EOS firmware version 03.07.20. The switch is based on a PowerPC CPU and proprietary switching ASICs.

Rate limiting for unknown unicast frames is not supported.

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting is not supported.

Enterasys Networks N1

The DUT is an Enterasys Networks 7H4382-25 layer 3 switch using EOS firmware version 07.63.01.0006. The switch is based on a PowerPC CPU and the proprietary CoreFlow switching ASIC.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

set cos port-resource flood-ctrl 0.0 unknown-unicast rate 100 unit pps
set cos settings 0 flood-ctrl enable
set cos state enable

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works, but the CPU usage during unknown unicast forwarding is still quite high. Further tests are needed to verify the effectiveness of limiting CPU usage by using unknown unicast limiters.

Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Class-of-Service processing needs to be enabled for unknown unicast rate limiting.

Enterasys Networks SSA

The DUT is an Enterasys Networks SSA-T1068-0652 layer 3 switch using EOS firmware version 08.41.01.0005. The switch is based on a PowerPC CPU and the proprietary CoreFlow2 switching ASIC.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the commands:

set cos port-resource flood-ctrl 0.0 unknown-unicast rate 100 unit pps
set cos settings 0 flood-ctrl enable
set cos state enable

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works, but the CPU usage during unknown unicast forwarding is still quite high. Further tests are needed to verify the effectiveness of limiting CPU usage by using unknown unicast limiters.

Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Class-of-Service processing needs to be enabled for unknown unicast rate limiting.

Extreme Networks Summit X450e

The DUT is an Extreme Networks X450e-24p layer 3 switch using ExtremeXOS (EXOS) firmware version 15.3.5.2patch1-8. The switch is based on a SiByte SB1 CPU and Broadcom SoC.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the command:

configure ports 1-2 rate-limit flood unknown-destmac 100

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Extreme Networks Summit X460

The DUT is an Extreme Networks X460-24t layer 3 switch using ExtremeXOS (EXOS) firmware version 16.1.2.14. The switch is based on an RMI Phoenix CPU and Broadcom SoC.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the command:

configure ports 1-2 rate-limit flood unknown-destmac 100

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Extreme Networks Summit X460-G2

The DUT is an Extreme Networks X460G2-24t-G4 layer 3 switch using ExtremeXOS (EXOS) firmware version 16.1.2.14. The switch is based on a Cavium Octeon II CPU and Broadcom Apollo 2 SoC.

Rate-limiting to 100 frames per second for unknown unicast frames is implemented with the command:

configure ports 1-2 rate-limit flood unknown-destmac 100

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Quanta LB4M

The DUT us a Quanta model LB4M, type LB4G 48x1G 2x10G, layer 2 switch using FASTPATH Switching firmware version 1.0.2.17. The switch is based on a Broadcom BCM56514_A0 SoC.

Rate-limiting can be configured based in interface bandwidth percentage only. A 1% limit is configured with the commands:

configure
interface 0/1
storm-control unicast level 1
exit
interface 0/2
storm-control unicast level 1
exit
exit

No Rate-Limiting

Rate-Limiting

Conclusion

Unknown unicast rate limiting works as expected. Rate limiting allows the specified rate of traffic into the port, dropping exceeding traffic only.

Back to my homepage.