linux/net/openvswitch
Ilya Maximets ea966b6698 openvswitch: fix lockup on tx to unregistering netdev with carrier
[ Upstream commit 47e55e4b41 ]

Commit in a fixes tag attempted to fix the issue in the following
sequence of calls:

    do_output
    -> ovs_vport_send
       -> dev_queue_xmit
          -> __dev_queue_xmit
             -> netdev_core_pick_tx
                -> skb_tx_hash

When device is unregistering, the 'dev->real_num_tx_queues' goes to
zero and the 'while (unlikely(hash >= qcount))' loop inside the
'skb_tx_hash' becomes infinite, locking up the core forever.

But unfortunately, checking just the carrier status is not enough to
fix the issue, because some devices may still be in unregistering
state while reporting carrier status OK.

One example of such device is a net/dummy.  It sets carrier ON
on start, but it doesn't implement .ndo_stop to set the carrier off.
And it makes sense, because dummy doesn't really have a carrier.
Therefore, while this device is unregistering, it's still easy to hit
the infinite loop in the skb_tx_hash() from the OVS datapath.  There
might be other drivers that do the same, but dummy by itself is
important for the OVS ecosystem, because it is frequently used as a
packet sink for tcpdump while debugging OVS deployments.  And when the
issue is hit, the only way to recover is to reboot.

Fix that by also checking if the device is running.  The running
state is handled by the net core during unregistering, so it covers
unregistering case better, and we don't really need to send packets
to devices that are not running anyway.

While only checking the running state might be enough, the carrier
check is preserved.  The running and the carrier states seem disjoined
throughout the code and different drivers.  And other core functions
like __dev_direct_xmit() check both before attempting to transmit
a packet.  So, it seems safer to check both flags in OVS as well.

Fixes: 066b86787f ("net: openvswitch: fix race on port output")
Reported-by: Friedrich Weber <f.weber@proxmox.com>
Closes: https://mail.openvswitch.org/pipermail/ovs-discuss/2025-January/053423.html
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
Reviewed-by: Aaron Conole <aconole@redhat.com>
Link: https://patch.msgid.link/20250109122225.4034688-1-i.maximets@ovn.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
History 2025-01-23 17:21:09 +01:00
..
actions.c openvswitch: fix lockup on tx to unregistering netdev with carrier 2025-01-23 17:21:09 +01:00
conntrack.c openvswitch: get related ct labels from its master if it is not confirmed 2024-07-05 09:33:46 +02:00
conntrack.h net/sched: act_api: fix miss set post_ct for ovs after do conntrack in act_ct 2021-03-16 15:22:18 -07:00
datapath.c net: ovs: fix ovs_drop_reasons error 2024-08-29 17:33:50 +02:00
datapath.h net/sched: Enable tc skb ext allocation on chain miss only when needed 2022-02-05 10:12:53 +00:00
dp_notify.c net: openvswitch: use netif_ovs_is_port() instead of opencode 2019-07-08 15:53:25 -07:00
drop.h net: openvswitch: add misc error drop reasons 2023-08-14 08:01:06 +01:00
flow.c net: openvswitch: fix overwriting ct original tuple for ICMPv6 2024-06-12 11:11:52 +02:00
flow.h net: openvswitch: reduce cpu_used_mask memory 2023-02-06 22:36:29 -08:00
flow_netlink.c net: openvswitch: limit the number of recursions from action sets 2024-02-23 09:24:51 +01:00
flow_netlink.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 269 2019-06-05 17:30:29 +02:00
flow_table.c There is no particular theme here - mainly quick hits all over the tree. 2023-02-23 17:55:40 -08:00
flow_table.h net: openvswitch: fix to make sure flow_lookup() is not preempted 2020-10-18 12:29:36 -07:00
Kconfig net: create nf_conntrack_ovs for ovs and tc use 2023-02-10 16:23:03 -08:00
Makefile openvswitch: add trace points 2021-06-22 10:47:32 -07:00
meter.c genetlink: remove userhdr from struct genl_info 2023-08-15 14:54:44 -07:00
meter.h net: openvswitch: use u64 for meter bucket 2020-04-23 18:26:11 -07:00
openvswitch_trace.c openvswitch: add trace points 2021-06-22 10:47:32 -07:00
openvswitch_trace.h openvswitch: add trace points 2021-06-22 10:47:32 -07:00
vport-geneve.c rtnetlink: Honour NLM_F_ECHO flag in rtnl_delete_link 2022-10-31 18:10:21 -07:00
vport-gre.c rtnetlink: Honour NLM_F_ECHO flag in rtnl_delete_link 2022-10-31 18:10:21 -07:00
vport-internal_dev.c openvswitch: Change the return type for vport_ops.send function hook to int 2022-09-19 18:28:50 -07:00
vport-internal_dev.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 269 2019-06-05 17:30:29 +02:00
vport-netdev.c rtnetlink: Honour NLM_F_ECHO flag in rtnl_delete_link 2022-10-31 18:10:21 -07:00
vport-netdev.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 269 2019-06-05 17:30:29 +02:00
vport-vxlan.c rtnetlink: Honour NLM_F_ECHO flag in rtnl_delete_link 2022-10-31 18:10:21 -07:00
vport.c net: openvswitch: fix upcall counter access before allocation 2023-06-07 12:25:05 +01:00
vport.h net: openvswitch: Add support to count upcall packets 2022-12-09 10:43:46 +00:00