System-Engineering | 7 min read
Carrier-Grade Traffic Generation
While iperf/iperf3 are suitable to test the bandwidth of a 10 gig link they can not be used to test specific traffic patters or reliably test even faster links.
In the past, I used various performance testing tools like Apache ab, Apache JMeter, iperf and tcpbench. While the first two primarily target web servers, the following two can be used for general bandwidth testing.
Usually, if specific traffic patterns or high PPS (packet per second) rates need to be tested, a commercial product is used. The dominant vendors in this area are Spirent and Ixia. These products come in the form of an appliance and are very expensive.
An alternative traffic generator was released under the Apache 2 license by Cisco in 2015: TRex realistic traffic generator. Its promise is to be able to generate realistic traffic at more than 100 Gbps while just running on standard server hardware. The Apache 2 license means it is free to use. The source code is hosted on GitHub and binaries can be downloaded from the project website.
I was eager to try TRex. Fortunately, I have a small server available in my home office with supported network cards. I could also test TRex in a virtual machine but the real fun comes with real hardware!
The server is a SuperMicro E300-8D which provides 6 Intel 1G interfaces (2x I210, 4x I350) and 2 Intel 10G interfaces (2x X552). The CPU is a Xeon D-1518 with 4 cores and I added 64GB memory. In addition, I put a Mellanox Connect-X 4 Lx in the available PCI slot which adds two 25 gig nics. For all the parts I spent less than CHF 2k!
I was not sure what TRex was able to deliver on this small server but hoped to fill a 10 Gbps link with 64 byte packages.
TRex runs on Linux and uses DPDK at its core. It is using the network interfaces directly without sending the packets through the Linux kernel. With this, it can achieve very high packet rates. But that also means TRex/DPDK needs its own drivers for the nics. Fortunately, the nics in my SuperMicro server are supported. Alternatively, there is an abstraction which allows TRex to use nics via Linux kernel. But according to the documentation that would limit the packet rate to about 1 Mpps.
Installation and Setup
TRex should run on any current Linux distribution. But in order to support the Mellanox nic I have, the documentation recommends only CentOS 7.6. Nevertheless, I tried CentOS 8.1 and was lucky.
After installing a minimal CentOS I performed these steps:
- Installation of RPMs needed by TRex and Mellanox OFED
dnf install per-Term-ANSIColor tk python36 make tcl
- Follow the instructions to install the Mellanox OFED driver here
- Additional RPMs are needed for TRex to compile DPDK drivers for the Intel nics (happens automatically on the first run):
dnf group install "Development Tools" dns install elfutils-libelf-devel kernel-devel
- Download the latest release of TRex and unpack it. At the time of this writing version 2.80 was the latest one.
mkdir -p /opt/trex; cd /opt/trex wget --no-check-certificate https://trex-tgn.cisco.com/trex/release/latest tar xvfz latest cd v2.80
After these steps, the only missing thing is the TRex config file. It tells TRex which nics to use and a few other things. The easiest way to create it is to use the wizard “dpdk_setup_ports.py” and just use the recommended settings.
To get started I chose the two ports on the Mellanox card and physically connected them with a DAC (loopback).
Different modes of operations are provided by TRex:
- Advanced Stateful
This is the original mode that TRex provided first. It does not provide a full TCP/IP stack but can inject smart replies on real flows. The traffic it generates is based on templates of captured traffic, then gets manipulated (eg. Source and destination address) and amplified.
In normal operations, it generates the traffic on one nic (client) and measures/replies on another nic (server). At the same time, it can inject additional packets to measure latency. NAT can dynamically be detected.
A simple test case definition looks like this:
[root@super1 v2.80]# cat cap2/dns.yaml - duration : 10.0 generator : distribution : "seq" clients_start : "184.108.40.206" clients_end : "220.127.116.11" servers_start : "18.104.22.168" servers_end : "22.214.171.124" clients_per_gb : 201 min_clients : 101 dual_port_mask : "126.96.36.199" tcp_aging : 1 udp_aging : 1 cap_info : - name: cap2/dns.pcap cps : 1.0 ipg : 10000 rtt : 10000 w : 1
The “generator” section defines what IP addresses will be used while the “cap_info” section references a file (or multiple files) with captured traffic.
TRex includes many more such profiles. Besides IMIX examples it also has profiles for a realistic traffic mix defined by SFR France that consists of typical enterprise flows like Citrix, Oracle, voice traffic and more.
These profiles can be run on the command line with additional options to define parameters like the amplification rate. The following command will generate 5 Gbps of SFR traffic and measure the latency 1000 times per Second:
[root@super1 v2.80]# ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -m 5 -l 1000
The result looks like this:
In contrast to stateful mode, the stateless mode offers many more possibilities to craft packets but it is not aware of flows. It is best suited for basic L2/L3 testing.
Profiles are not written as YAML files but as Python scripts. To define the characteristics of the generated packets it leverages the capabilities of Scapy.
Not only are the profiles defined differently in this mode but also the usage is different. First, the TRex server needs to be started with the interactive option (t-rex-64 -i) and then in another shell the TRex client can be started.
The above output shows, that TRex can deliver more packets on my SuperMicro than I hoped for: almost 30 Gbps with 64 byte packets (53 Mpps)! With larger packets, it’s no problem to fill both available 25 Gbps links.
The stateless mode also offers a simulation of routing protocols like BGP, OSPF and others. It includes Bird for this purpose, which I haven’t looked further into yet.
There is a graphical client for the stateless mode that allows us to define the packages with a graphical editor and then run the tests. The client can be installed on Windows, Mac or Linux and connects remotely to the Linux server hosting the TRex server.
Advanced Stateful Mode
The third mode is the “Advanced Stateful Mode”. In contrast to the regular stateful mode, it can also deal with Layer 7. For example, it is possible for a user to define different user-agent strings in HTTP headers. It incorporates a real TCP stack which sits on top of DPDK in userland. Pcap files are used as with the regular stateful mode. It is ideal for testing firewalls that inspect TCP and protocols on top of it.
The usage of this mode is similar to stateless mode: the server is started in interactive mode (with the additional –astf flag) and trex-console can be used to launch the tests. Profiles are defined in the form of a Python script. A simple profile looks like this (astf/http_simple.py):
from trex.astf.api import * class Prof1(): def __init__(self): pass def get_profile(self, **kwargs): # ip generator ip_gen_c = ASTFIPGenDist(ip_range=["188.8.131.52", "184.108.40.206"], distribution="seq") ip_gen_s = ASTFIPGenDist(ip_range=["220.127.116.11", "18.104.22.168"], distribution="seq") ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="22.214.171.124"), dist_client=ip_gen_c, dist_server=ip_gen_s) return ASTFProfile(default_ip_gen=ip_gen, cap_list=[ASTFCapInfo(file="../avl/delay_10_http_browsing_0.pcap", cps=2.776)]) def register(): return Prof1()
It was very exciting for me to see, that with TRex this little SuperMicro server is able to generate more than 50 Mpps. According to the documentation, a bigger server can easily max out 100 Gbps nics.
I also liked the fact, that the focus of the TRex project is not so much on a graphical user interface but on automation. Having the test as YAML files or Python scripts makes it easy to version control the tests in GIT and make them repeatable. This also allows it to fully automate the full test setup for example by integrating TRex scripts with the RobotFramework test that also takes care of the configuration of the device under test and generating reports.
TRex is not only attractive to save on license fees for commercial traffic generators but also because it can be a real time-saver when the testing is fully automated by leveraging the Python SDK.