Executing any of tools (ifconfig, netstat, ipfw) breaks running primary process

[blaa]

Hello,
I'm chasing a issue as in the title.

Reproduction steps:

1. Build f-stack 1.24 on Debian Bookworm VM using VirtIO network drivers and DPDK from packages (22.11.7)
2. Build nginx app, run the app.
3. Curl can correctly connect to the nginx.
4. Execute ff_ifconfig --help (so it doesn't really do anything at all. Only ff_ipc_init and rte_eal_init)
5. nginx stops accepting connections.

ifconfig output:

/data/f-stack # ff_ifconfig --help
Device 0000:00:12.0 is not driven by the primary process
EAL: Requested device 0000:00:12.0 cannot be used
ff_ifconfig: illegal option -- -
usage: ifconfig -p <f-stack proc_id> [-f type:format] [-g groupname] [-C] [-L] interface address_family
                [address [dest_address]] [parameters]
       ifconfig -p <f-stack proc_id> interface create
       ifconfig -p <f-stack proc_id> -a [-g groupname] [-C] [-L] [-d] [-m] [-u] [-v] [address_family]
       ifconfig -p <f-stack proc_id> -l [-d] [-u] [address_family]
       ifconfig -p <f-stack proc_id> [-g groupname] [-C] [-L] [-d] [-m] [-u] [-v]

Devbind:

Network devices using DPDK-compatible driver
============================================
0000:00:13.0 'Virtio network device 1000' drv=uio_pci_generic unused=virtio_pci
0000:00:14.0 'Virtio network device 1000' drv=uio_pci_generic unused=virtio_pci

Network devices using kernel driver
===================================
0000:00:12.0 'Virtio network device 1000' if=ens18 drv=virtio-pci unused=virtio_pci,uio_pci_generic *Active*

The same happens on our own code that we were trying to use TRANSPARENT IP sockets - using ipfw.
Both codes use kqueue and loop keeps executing, just there are no incoming events on new connections.

[blaa]

If I remove ff_ipc_init from ifconfig.c the ifconfig --help won't break the primary process. But, ifconfig prints the interfaces correctly still... and then breaks the process. It calls rte_eal_init later then:

#0  0x0000555556de4290 in rte_eal_init ()
#1  0x0000555556498f59 in ff_ipc_init () at ff_ipc.c:68
#2  0x0000555556498ff7 in ff_ipc_msg_alloc () at ff_ipc.c:94
#3  0x00005555564a593a in sysctl (name=0x7fffffffdc30, namelen=6, old=0x0, oldlenp=0x7fffffffdc28, new=0x0, newlen=0) at sysctl.c:46
#4  0x00005555564a3035 in getifaddrs (pif=0x7fffffffe140) at getifaddrs.c:125
#5  0x000055555648f6a9 in main (argc=0, argv=0x7fffffffe300) at ifconfig.c:691

When stepping through the code it seems rte_eal_init is not a problem, but quitting the process after execution is. Disabling rte_eal_cleanup calls doesn't help.

[blaa]

This seems to be the case:
https://lore.kernel.org/r/dpdk-dev/20241128091552.2406f1a3@hermes.local/T/
So it's a DPDK bug that is present on Debian Stable / Bookworm.

EDIT: Also the same bug seems to be present in the included DPDK with f-stack.

[jfb8856606]

Root Cause Analysis

After investigating the DPDK source code and upstream mailing list, here is the complete picture.

Bug-introducing commit

1cab1a40ea9b — "bus: cleanup devices on shutdown" (2022-10-04, DPDK 22.11)

This commit added eal_bus_cleanup() to rte_eal_cleanup() without distinguishing between primary and secondary processes.

What happens when a secondary process exits

rte_eal_cleanup()
  rte_mp_channel_cleanup()   <- closes IPC socket (mp_fd = -1)
  eal_bus_cleanup()          <- calls drv->remove() on ALL PCI devices  ← BUG
    pci_cleanup()
      drv->remove(dev)       <- eth_virtio_pci_remove()
        virtio_dev_close()
          virtio_reset(hw)   <- writes 0x00 to VIRTIO_CONFIG_STATUS_RESET !

virtio_reset() issues a full hardware reset of the NIC. The primary process still holds the device but its hardware state has been destroyed — it stops receiving packets.

This is why even ff_ifconfig --help (which triggers rte_eal_init → rte_eal_cleanup in secondary mode) is enough to break the primary process.

Affected DPDK / F-Stack versions

Version	Affected
DPDK 22.11.x (all point releases)	✅ Yes
DPDK 23.11.x incl. F-Stack bundled 23.11.5	✅ Yes
DPDK 24.11.x (up to 24.11.4)	✅ Yes
DPDK 25.07+	✅ Fixed

Upstream fix

Upstream DPDK commit 4bc53f8f0d64 — "eal: fix MP socket cleanup" (2025-07-19, merged for DPDK 25.07) by Yang Ming (Nokia).

The fix reorders eal_bus_cleanup() before rte_mp_channel_cleanup() and adds per-driver secondary-process guards.

Fix for F-Stack bundled DPDK 23.11.5

PR #1050 backports the fix using the simpler guard approach:

// lib/eal/linux/eal.c  rte_eal_cleanup()
- eal_bus_cleanup();
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+     eal_bus_cleanup();

PR: #1050

[jfb8856606]

Fixed by PR #1050, now merged into dev branch.