NetFlow v9 / IPFIX datagrams with options data are dropped entirely

[Mesverrum]

Summary

NetFlow v9 / IPFIX datagrams that include a standard options data flowset are discarded entirely. The real flow records in the same UDP packet never reach the flow handler, so flow_in_bytes / flow_records_flows stay empty against common exporters (softflowd, Cisco, etc.).

Clean v9 and IPFIX (template + data only) parse correctly. Adding a spec-compliant options template + options data set is enough to make pktvisor increment packet_errors and drop the packet.

Environment

• pktvisord 4.5.0-develop-fb6dc23 (Orb agent embedded backend)
• Flow tap: flow_type: netflow, UDP listen
• Exporter: Alpine softflowd 1.1 NetFlow v9 (also reproduced with synthetic v9 and IPFIX)

Observed vs expected

Datagram	flow.packet_errors (20 copies)	Flow handler
NetFlow v9 template + data	+0	parsed (203.0.113.10:12345/203.0.113.20:8080)
NetFlow v9 + options data (flowset id 256)	+20	dropped
IPFIX set 2 + data	+0	parsed
IPFIX set 3 + options data 256 + data	+20	dropped

Expected: skip unknown / options flowsets and still ingest the data records (RFC 3954 / RFC 7011). Collectors such as ktranslate do this on the same softflowd copy.

Softflowd is not doing anything unusual: v9 packets contain template flowset 0, options template flowset 1, options data with template id 256, then IPv4/IPv6 data (ids 1024/1025/…). 256 is the first legal data template id.

Root cause

Options templates are unimplemented, then a missing-template lookup fails the whole packet.

src/inputs/flow/NetflowData.h process_netflow_v9:

case NF9_OPTIONS_FLOWSET_ID:
    /* XXX: implement this (maybe) */
    break;
default:
    if (flowset_id < NF9_MIN_RECORD_FLOWSET_ID) {
        break;
    }
    if (!process_netflow_v9_data(...)) {
        return false;  // aborts remaining flowsets, including real data
    }

process_netflow_v9_data returns false when (source_id, flowset_id) is not in nf9_template_map (also when the map is empty, or num_flowsets == 0).

The same pattern exists in process_netflow_v10 for IPFIX (options set skipped; unknown data set → return false).

FlowInputStream.cpp then increments _error_count and never calls netflow_cb:

if (process_netflow_packet(&sample)) {
    ... netflow_cb(...)
} else {
    ++_error_count;
}

On a live tap this shows up as input.*.flow.packet_errors climbing in lockstep with exporter PPS, empty metrics/bucket/1, and no lab_netflow flow_in_bytes series.

Suggested fix

1. Do not fail the datagram when a data flowset has no template (or fails length checks). Skip that flowset and continue. Template-only packets should also not count as packet_errors (process_netflow_v9 currently return false when total_flows == 0).
2. Optionally parse options templates (v9 flowset 1 / IPFIX set 3) so options data is not mistaken for missing flow templates. Skipping them is enough for volume metrics.

Reproduction

Minimal flow tap:

version: "1.0"
visor:
  taps:
    lab_netflow:
      input_type: flow
      config:
        flow_type: netflow
        port: 2055
        bind: 0.0.0.0
  policies:
    lab_netflow_summary:
      kind: collection
      input:
        input_type: flow
        tap: lab_netflow
      handlers:
        modules:
          flow_overview:
            type: flow
            metric_groups:
              enable: [counters, by_bytes, conversations]

Send 20 clean vs 20 options-bearing packets to 127.0.0.1:2055, then compare GET /api/v1/policies → flow.packet_errors and .../metrics/bucket/1.

Sketch of the failing v9 layout (all in one UDP payload):

1. Header version=9, source_id unique
2. Template flowset 0, template id 3000 (IPv4 5-tuple + IN_BYTES/IN_PKTS)
3. Options template flowset 1, template id 256
4. Options data flowset 256
5. Data flowset 3000 (the actual conversation)

Omit steps 3–4 and the same data record is accepted. IPFIX equivalent: set id 2 (templates), set id 3 (options template), set 256 (options data), set 3000 (data).

Happy to attach a small Python sender if useful.

[leoparente]

hi @Mesverrum thanks for the detailed feedback. We will start working on this ASAP. We have some data already to validate this, but if needed, we will ask for more input like sender data.