3. Setting up a System to Run DPDK Applications

This section provides step-by-step instructions for setting up your system to run DPDK applications. It covers system configurations for Linux, FreeBSD, and Windows. Each section details the necessary memory and device setups for these operating systems.

Navigate to the section that corresponds to your operating system to begin the setup process.

3.1. System Setup for Linux

3.1.1. Memory Setup: Reserving Hugepages

For Linux, DPDK requires hugepages be reserved for its use on the system. To check if hugepages are reserved on your system, you can run the following command:

grep HugePages_Total /proc/meminfo

If hugepages are not reserved, you will need to reserve them by following these steps:

  1. Determine the number of hugepages you want to allocate. For example, to allocate 1024 hugepages of 2MB each, you can use the following command:

    echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

  2. To make the hugepages configuration persistent across reboots, add the following line to your /etc/sysctl.conf file, adjusting the number of hugepages as needed:

    vm.nr_hugepages = 1024

  3. Check if your distribution already provides access to hugepages via /dev/hugepages. If it doesn’t, you will need to manually mount the hugepages filesystem. To do this, add the following line to your /etc/fstab file:

    nodev /mnt/huge hugetlbfs defaults 0 0

    Then, create the mount directory and mount the filesystem:

    mkdir -p /mnt/huge
mount -a

Note

For information on configuring different hugepage sizes, including 1G pages on x86 and various sizes on ARM and other architectures, refer to Hugepages Configuration for Multiple Architectures.

3.1.2. Device Setup: VFIO

VFIO is a robust and secure driver that relies on IOMMU protection. To make use of VFIO on Linux, the vfio-pci module must be loaded:

sudo modprobe vfio-pci

VFIO kernel is usually present by default in all distributions, however please consult your distribution’s documentation to make sure that is the case.

To make use of full VFIO functionality, both kernel and BIOS must support and be configured to use IO virtualization (such as Intel® VT-d).

Note

In most cases, specifying “iommu=on” as kernel parameter should be enough to configure the Linux kernel to use IOMMU.

For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up. For more information, refer to Running DPDK Applications Without Root Privileges.

Refer to VFIO no-IOMMU mode when there is no IOMMU available on the system.

3.1.3. Binding Network Ports to VFIO-PCI Module

To bind network ports to the vfio-pci module on Linux, use the dpdk-devbind.py script with the following command. This script will automatically unbind the device from its current driver, if necessary, before binding it to vfio-pci.

sudo dpdk-devbind.py -b vfio-pci <pci_id>

Replace <pci_id> with the PCI device identifier of the network port. For example:

# Example for a network port with PCI ID 01:00.0
sudo dpdk-devbind.py -b vfio-pci 01:00.0

3.2. System Setup for FreeBSD

3.2.1. Memory Setup: Loading the DPDK contigmem Module

To run a DPDK application on FreeBSD, physically contiguous memory is required. In the absence of non-transparent superpages, the included sources for the contigmem kernel module provides the ability to present contiguous blocks of memory for the DPDK to use. The contigmem module must be loaded into the running kernel before any DPDK application is run. Once DPDK is installed on the system, the module can be found in the /boot/modules directory.

The amount of physically contiguous memory along with the number of physically contiguous blocks to be reserved by the module can be set at runtime prior to module loading using:

kenv hw.contigmem.num_buffers=n
kenv hw.contigmem.buffer_size=m

The kernel environment variables can also be specified during boot by placing the following in /boot/loader.conf:

hw.contigmem.num_buffers=n
hw.contigmem.buffer_size=m

The variables can be inspected using the following command:

sysctl -a hw.contigmem

The module can then be loaded using kldload:

cd /boot/modules
kldload contigmem

3.2.2. Device Setup: Loading the DPDK nic_uio Module

After Memory Setup: Loading the DPDK contigmem Module on FreeBSD, the nic_uio module must be loaded into the running kernel before running any DPDK application.

Manual Loading:

To load the module manually, execute the following command:

cd /boot/modules
kldload nic_uio
Automatic Loading at System Startup:

For the module to be automatically loaded at system startup, add this line to /boot/loader.conf:

nic_uio_load="YES"

This configuration step allows the system to automatically load the nic_uio module during the boot process. Additionally, to bind specific network ports to the nic_uio module, set the hw.nic_uio.bdfs variable in the /boot/loader.conf file. Use a comma-separated list of PCI device identifiers for the network ports without any whitespace. For example:

hw.nic_uio.bdfs="2:0:0,2:0:1"

3.2.3. Binding and Unbinding Network Ports to/from nic_uio Module

If the original driver for a network port has been compiled into the kernel, it is necessary to reboot FreeBSD to restore the original device binding. Before doing so, update or remove the hw.nic_uio.bdfs in /boot/loader.conf.

If rebinding to a driver that is a loadable module, the network port binding can be reset without rebooting. To do so, unload both the target kernel module and the nic_uio module, modify or clear the hw.nic_uio.bdfs kernel environment (kenv) value, and reload the two drivers - first the original kernel driver, and then the nic_uio driver.

Example commands to perform these steps are shown below:

kldunload nic_uio
kldunload <original_driver>
kenv -u hw.nic_uio.bdfs
kldload <original_driver>
kldload nic_uio  # optional

3.3. System Setup for Windows

3.3.1. Memory Setup: Installing Windows Modules

Before running DPDK applications on Windows, certain kernel-mode drivers must be installed. Note that these drivers are not signed, so you’ll need to disable signature enforcement. However, be cautious as this can weaken your OS security and is generally not recommended in production environments.

3.3.2. Device Setup: Install Drivers

To run DPDK applications on Windows, you’ll need to install specific kernel-mode drivers:

  • virt2phys: This driver is essential for providing access to physical addresses and is mandatory for allocating physically-contiguous memory, which is required by hardware PMDs. Once loaded successfully, this driver will appear in the Device Manager as Virtual to physical address translator device under the Kernel bypass category. If DPDK cannot communicate with the driver, a warning will be printed during initialization.

  • NetUIO: This driver provides access to device hardware resources and is mandatory for all hardware PMDs, except for the mlx5 PMD. Devices supported by NetUIO are listed in netuio.inf. You can extend this list to try running DPDK with new devices.