NetVyn BNG — Installation¶

Ubuntu 22.04 LTS (kernel 5.15), bare metal, Debian packages.

This is the supported installation path. Everything below was validated on an HP ProLiant DL360 G9 (2 sockets, 48 threads, 64 GB, 4x Intel X710) and a Dell quad-socket E5-4640 with Intel X520.

Two packages are installed, in this order:

Package	Contents
`netvyn-dpdk`	bundled DPDK runtime + PMDs, `dpdk-devbind.py`, hugepage / vfio-pci setup, `netvyn-bind-nics`, `netvyn-enroll`
`netvyn-bng`	the BNG daemon, `bng-cli`, systemd unit, configuration examples

0. Obtaining the packages¶

The packages are not published to a public repository. To request them, email support@netvynx.com with:

the intended deployment (operator name, site, expected subscriber count),
the OS and version (lsb_release -d),
the dataplane NIC model (lspci -nn | grep -i ethernet).

You will receive netvyn-dpdk_<version>_amd64.deb and netvyn-bng_<version>_amd64.deb. Keep both — they are installed together and their versions must match.

A licence is requested separately, after installation, using the netvyn-enroll tool the package provides (§5). The BNG serves no subscribers until it is licensed.

1. Hardware and BIOS¶

Required

x86-64 server, 8 physical cores minimum; 16 or more for production.
16 GB RAM minimum. Memory is dominated by QoS scheduler pipes — see §7.
An Intel dataplane NIC: ixgbe (X520), i40e (X710/XL710) or ice (E810). Mellanox mlx5 is supported but is bifurcated — it must never be bound to vfio-pci, and needs rdma-core present.
A separate management interface. Dataplane ports are taken over by DPDK and disappear from the kernel; they can never carry SSH or the WebUI.

BIOS

Intel VT-d: Enabled. Without it vfio-pci cannot bind and the daemon will not start. On HPE Gen9 this is under System Configuration → BIOS/Platform Configuration → Virtualization Options.
Power Regulator: Static High Performance, and disable C-states if offered. A DPDK worker spins in a poll loop and looks idle to the power governor; letting it downclock shows up as packet jitter.
Leave Hyper-Threading enabled — the planner places one worker per physical core.

Confirm 1 GB page support before going further:

grep -c pdpe1gb /proc/cpuinfo

Any non-zero result is fine.

2. Kernel command line¶

This is the single most common installation failure. Ubuntu does not enable the Intel IOMMU by default, even when VT-d is on in the BIOS. Without intel_iommu=on, every NIC bind fails with "IOMMU support is disabled".

Edit /etc/default/grub and append to GRUB_CMDLINE_LINUX_DEFAULT:

intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=32

On AMD, use amd_iommu=on instead.

Sizing hugepages: 32 (that is, 32 GB) suits a 64 GB box carrying up to 8k subscribers. It must cover the QoS scheduler (§7), which is by far the largest consumer. Never reserve so much that the OS is left short — hugepages are permanently removed from normal memory, and an OOM kill takes the daemon with it. Leave at least 16 GB to the operating system.

Apply and reboot:

sudo update-grub && sudo reboot

Apply the edit exactly once. Running a sed twice appends the whole flag set twice; the kernel then adds the hugepage counts together and can reserve far more than intended. Check /proc/cmdline for duplicates before rebooting.

Verify after the reboot — all three must be right:

grep -o 'intel_iommu=on' /proc/cmdline; ls /sys/kernel/iommu_groups | wc -l; grep HugePages_Total /proc/meminfo

You need the flag present, an IOMMU group count greater than zero, and the hugepage total you asked for. If the group count is 0 while the flag is present, VT-d is not actually enabled in the BIOS — return to §1.

3. Install the packages¶

Order matters — netvyn-bng links against the DPDK runtime:

sudo apt-get install ./netvyn-dpdk_<version>_amd64.deb ./netvyn-bng_<version>_amd64.deb

Installing both in one command lets apt resolve the order itself. Installing netvyn-bng first fails with a dependency error.

The netvyn-dpdk post-install registers the DPDK runtime with ldconfig, configures vfio-pci, creates the vendor support account netvyn, and sets the hugepage count. If you tuned nr_hugepages by hand, re-check it afterwards — see §10.

4. Bind the dataplane NICs¶

sudo -E netvyn-bind-nics

The -E matters: it preserves SSH_CONNECTION, which the tool uses to identify the interface you are logged in over so it can never hand that link to DPDK. The default-route interface is excluded regardless.

The tool lists every NIC, marks the management ones excluded, and asks which to bind. Select by interface name — do not answer all, which would take spare onboard ports as well:

Bind which? ('all', or space-separated ifnames/pci) [all]: ens1f0 ens1f1

Watch each line for -> vfio-pci with no WARN after it. The selection is saved to /etc/netvyn/dataplane-nics and re-applied at boot by netvyn-bind.service — a vfio-pci bind does not survive a reboot on its own.

Confirm:

dpdk-devbind.py --status | grep -A6 'using DPDK'

Every dataplane port should show drv=vfio-pci. Mellanox adapters correctly stay on mlx5_core and are recorded without rebinding.

5. Request a licence¶

sudo netvyn-enroll request

It asks for your name, company and reseller code, attaches the node's unique id automatically, and submits the request. The BNG accepts no subscribers until a licence is issued.

6. Start the daemon¶

sudo systemctl enable --now netvyn-bng
systemctl status netvyn-bng

enable is not optional — without it the daemon does not return after a reboot.

Startup produces no output on stdout. Diagnostics go to an in-memory ring (show log in the CLI); core events go to /var/log/bng/bng.log:

tail -40 /var/log/bng/bng.log

Look for the topology plan, the port inventory, and a final BNG up: N workers, M control, P ports.

Older VT-d silicon can pause for one to three minutes after EAL: Using IOMMU type 1 on the first start. That is normal — wait.

Then open the CLI. The control socket is root-owned, so sudo is required:

sudo bng-cli

The default login is admin. The account is in a forced first-login state until its password is changed (system user change admin <new-password>).

7. Size the box¶

The defaults start the daemon but are not production values. Three settings matter, all under /usr/local/etc/bng/.

Workers — `system.conf`¶

The planner otherwise claims every spare lcore as a worker. On a 48-thread box with 40 Gbps of ports that is 42 poll loops for no benefit, and it exhausts the mbuf pool during start.

max-workers-per-10g 2

Workers become ceil(total_gbps / 10) x 2 — eight for 4x10G. That is ample: one worker handles well over a gigabit of PPPoE with CGNAT and shaping.

QoS pipes — `qos.conf`¶

Scheduler memory is:

max-pipes  x  number of profiles  x  2 directions  x  workers

scaled by each profile's qsize. Every profile line costs memory whether or not a subscriber uses it, so define only the disciplines you actually assign. Measured on the DL360: 2.29 MB per pipe at 8 workers with two profiles — 8192 pipes cost 20.4 GB.

max-pipes 8192
profile fq    fq_codel qsize 512  flows 2048 target 10000 interval 10000 ecn on
profile fqpie fq_pie   qsize 2048 flows 1024 target 15000 interval 15000 ecn on
assign ppp   fq
assign ipoe  fqpie
default-profile fq
overhead-compensation 12

max-pipes must be a power of two above your peak concurrent session count. Sessions beyond the cap forward unshaped. It is tunable live, so raise it as you grow rather than over-reserving up front.

overhead-compensation 12 shapes each pipe slightly above its plan so a speed test measures the plan number — the shaper counts PPPoE and L1/L2 framing that the test does not. Without it, a 300 Mbps plan measures around 268 Mbps.

Conntrack — `nat.conf`¶

conntrack-max 2000000

This is concurrent flows, not subscribers — size it as subscribers x peak-flows-each, with headroom. 2,000,000 suits 8k residential subscribers. Changing it requires a restart, so set it before going live.

Public addressing — `nat.conf`¶

Subscribers per public IP is fixed arithmetic:

(port-range size) / block = subscribers per public IP
64512 / 2048 = 31

A pool of 8 public IPs therefore serves 248 subscribers, and subscriber 249 gets no block — its session comes up with no internet. Size the pool for your target before turn-up, not after.

8. First configuration¶

A minimal PPPoE service. The WAN port is derived from the default route, so no port-role file is needed.

sudo bng-cli

system-view
ip ether1 203.0.113.2/30
ip route add 0.0.0.0/0 203.0.113.1
interface ether2
add vlan pppoe0 100
quit
commit
save

Then bind PPPoE to the VLAN sub-interface in /usr/local/etc/bng/pppoe.conf:

interface pppoe0

A VLAN interface is pure L2 until a protocol claims it. With no interface line the access concentrator answers discovery on nothing — the daemon warns about this at start.

Full command reference: CLI.md.

9. Verify¶

show interfaces        every port and VLAN, with state and speed
show ip route          the default route must show flags SR (selected, resolved)
show health            workers, memory, time sync, temperature, disk
show topology          the lcore plan

The default route resolves only once the gateway answers ARP. Allow up to a minute — if the upstream switch port is not configured as an edge port, spanning tree withholds forwarding for 30 to 50 seconds after link-up and nothing works until it completes.

Reboot once and confirm everything returns unattended: the service starts, the NICs re-bind, hugepages are reserved, and interface names persist.

10. Upgrading¶

sudo apt-get install ./netvyn-dpdk_<new>_amd64.deb ./netvyn-bng_<new>_amd64.deb
sudo systemctl restart netvyn-bng

Your .conf files are never touched — the packages ship only *.conf.example.

Two things to re-check after every upgrade:

systemctl is-enabled netvyn-bng; grep HugePages_Total /proc/meminfo

Upgrading netvyn-dpdk removes and reinstalls netvyn-bng, which loses the systemd enable symlink — the daemon runs but will not return after a reboot. The DPDK post-install also rewrites the hugepage count.

A restart drops every PPPoE session. On a box carrying subscribers, schedule it.

11. Troubleshooting¶

Symptom	Cause and fix
`IOMMU support is disabled` on every bind	`intel_iommu=on` missing from `/proc/cmdline` — §2
IOMMU groups = 0 with the flag present	VT-d not enabled in BIOS — §1
`netvyn-bind-nics` exits silently under `sudo`	pre-1.0.1 package; use `sudo -E`
Daemon aborts at start, `Cannot allocate memory` on the last port	mbuf pool exhausted by too many workers — set `max-workers-per-10g` (§7)
`qos init failed; forwarding without shaping`	scheduler memory exceeded — fewer profiles, smaller `qsize`, or more hugepages (§7)
No `cli.sock`, CLI reports not reachable	the daemon aborted during init — read `/var/log/bng/bng.log`
`cli.sock` exists, Permission denied	the socket is root-owned; run `sudo bng-cli`
Daemon runs, default route stays unresolved	gateway not answering ARP: check VLAN tagging on the handoff, and STP edge-port on the switch
Session shows UP but has no internet	CGNAT pool exhausted — `show nat block`; the pool is too small for the subscriber count (§7)
Subscribers forward unshaped beyond a certain count	`max-pipes` below peak sessions (§7)
Speed test reads about 10% under plan	`overhead-compensation` unset (§7)
Daemon does not start after a reboot	run `systemctl enable netvyn-bng`; re-check after every upgrade (§10)
Link up, no traffic, `mac_local_errors` climbing	transceiver rejected by the far end (common with third-party DACs on Huawei) — use a cable coded for that switch
`PPPoE RSS: not installed (PMD rejected)`	expected on X520 and X710 — use `system dispatch enable`; see CLI.md §4.3
Terminal unresponsive after leaving the CLI	raw terminal mode left behind — type `stty sane` blind