2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-12-05 22:08:36 +01:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
2020-08-28 10:26:15 +02:00
|
|
|
|
2020-08-28 10:26:17 +02:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-01-19 14:15:33 -08:00
|
|
|
|
|
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
2023-01-19 14:15:33 -08:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-10-07 14:24:54 +02:00
|
|
|
|
xsk: align &xdp_buff_xsk harder
After the series "XSk buff on a diet" by Maciej, the greatest pow-2
which &xdp_buff_xsk can be divided got reduced from 16 to 8 on x86_64.
Also, sizeof(xdp_buff_xsk) now is 120 bytes, which, taking the previous
sentence into account, leads to that it leaves 8 bytes at the end of
cacheline, which means an array of buffs will have its elements
messed between the cachelines chaotically.
Use __aligned_largest for this struct. This alignment is usually 16
bytes, which makes it fill two full cachelines and align an array
nicely. ___cacheline_aligned may be excessive here, especially on
arches with 128-256 byte CLs, as well as 32-bit arches (76 -> 96
bytes on MIPS32R2), while not doing better than _largest.
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com>
Link: https://patch.msgid.link/20241203173733.3181246-2-aleksander.lobakin@intel.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-12-03 18:37:24 +01:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
2023-01-19 14:15:33 -08:00
|
|
|
|
2023-11-27 11:03:08 -08:00
|
|
|
|
2023-01-19 14:15:33 -08:00
|
|
|
|
2020-08-28 10:26:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
2020-08-28 10:26:23 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-04-16 12:19:08 +02:00
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
2023-07-19 15:24:08 +02:00
|
|
|
|
2020-08-28 10:26:23 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-08-28 10:26:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
2022-01-25 17:04:43 +01:00
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
2024-10-07 14:24:56 +02:00
|
|
|
|
2023-11-27 11:03:07 -08:00
|
|
|
|
2020-08-28 10:26:19 +02:00
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
2023-11-27 11:03:14 -08:00
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
xsk: use a smaller new lock for shared pool case
- Split cq_lock into two smaller locks: cq_prod_lock and
cq_cached_prod_lock
- Avoid disabling/enabling interrupts in the hot xmit path
In either xsk_cq_cancel_locked() or xsk_cq_reserve_locked() function,
the race condition is only between multiple xsks sharing the same
pool. They are all in the process context rather than interrupt context,
so now the small lock named cq_cached_prod_lock can be used without
handling interrupts.
While cq_cached_prod_lock ensures the exclusive modification of
@cached_prod, cq_prod_lock in xsk_cq_submit_addr_locked() only cares
about @producer and corresponding @desc. Both of them don't necessarily
be consistent with @cached_prod protected by cq_cached_prod_lock.
That's the reason why the previous big lock can be split into two
smaller ones. Please note that SPSC rule is all about the global state
of producer and consumer that can affect both layers instead of local
or cached ones.
Frequently disabling and enabling interrupt are very time consuming
in some cases, especially in a per-descriptor granularity, which now
can be avoided after this optimization, even when the pool is shared by
multiple xsks.
With this patch, the performance number[1] could go from 1,872,565 pps
to 1,961,009 pps. It's a minor rise of around 5%.
[1]: taskset -c 1 ./xdpsock -i enp2s0f1 -q 0 -t -S -s 64
Signed-off-by: Jason Xing <kernelxing@tencent.com>
Acked-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Link: https://patch.msgid.link/20251030000646.18859-3-kerneljasonxing@gmail.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-10-30 08:06:46 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-12-18 14:45:24 +01:00
|
|
|
|
xsk: use a smaller new lock for shared pool case
- Split cq_lock into two smaller locks: cq_prod_lock and
cq_cached_prod_lock
- Avoid disabling/enabling interrupts in the hot xmit path
In either xsk_cq_cancel_locked() or xsk_cq_reserve_locked() function,
the race condition is only between multiple xsks sharing the same
pool. They are all in the process context rather than interrupt context,
so now the small lock named cq_cached_prod_lock can be used without
handling interrupts.
While cq_cached_prod_lock ensures the exclusive modification of
@cached_prod, cq_prod_lock in xsk_cq_submit_addr_locked() only cares
about @producer and corresponding @desc. Both of them don't necessarily
be consistent with @cached_prod protected by cq_cached_prod_lock.
That's the reason why the previous big lock can be split into two
smaller ones. Please note that SPSC rule is all about the global state
of producer and consumer that can affect both layers instead of local
or cached ones.
Frequently disabling and enabling interrupt are very time consuming
in some cases, especially in a per-descriptor granularity, which now
can be avoided after this optimization, even when the pool is shared by
multiple xsks.
With this patch, the performance number[1] could go from 1,872,565 pps
to 1,961,009 pps. It's a minor rise of around 5%.
[1]: taskset -c 1 ./xdpsock -i enp2s0f1 -q 0 -t -S -s 64
Signed-off-by: Jason Xing <kernelxing@tencent.com>
Acked-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Link: https://patch.msgid.link/20251030000646.18859-3-kerneljasonxing@gmail.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-10-30 08:06:46 +08:00
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
2020-08-28 10:26:17 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-21 13:57:01 +00:00
|
|
|
|
2020-08-28 10:26:25 +02:00
|
|
|
|
2022-04-25 17:37:45 +02:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
2020-08-28 10:26:17 +02:00
|
|
|
|
2020-10-27 13:32:01 +01:00
|
|
|
|
2020-08-28 10:26:17 +02:00
|
|
|
|
2020-08-28 10:26:20 +02:00
|
|
|
|
|
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
2023-12-05 22:08:36 +01:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
xsk: Batched buffer allocation for the pool
Add a new driver interface xsk_buff_alloc_batch() offering batched
buffer allocations to improve performance. The new interface takes
three arguments: the buffer pool to allocated from, a pointer to an
array of struct xdp_buff pointers which will contain pointers to the
allocated xdp_buffs, and an unsigned integer specifying the max number
of buffers to allocate. The return value is the actual number of
buffers that the allocator managed to allocate and it will be in the
range 0 <= N <= max, where max is the third parameter to the function.
u32 xsk_buff_alloc_batch(struct xsk_buff_pool *pool, struct xdp_buff **xdp,
u32 max);
A second driver interface is also introduced that need to be used in
conjunction with xsk_buff_alloc_batch(). It is a helper that sets the
size of struct xdp_buff and is used by the NIC Rx irq routine when
receiving a packet. This helper sets the three struct members data,
data_meta, and data_end. The two first ones is in the xsk_buff_alloc()
case set in the allocation routine and data_end is set when a packet
is received in the receive irq function. This unfortunately leads to
worse performance since the xdp_buff is touched twice with a long time
period in between leading to an extra cache miss. Instead, we fill out
the xdp_buff with all 3 fields at one single point in time in the
driver, when the size of the packet is known. Hence this helper. Note
that the driver has to use this helper (or set all three fields
itself) when using xsk_buff_alloc_batch(). xsk_buff_alloc() works as
before and does not require this.
void xsk_buff_set_size(struct xdp_buff *xdp, u32 size);
Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20210922075613.12186-3-magnus.karlsson@gmail.com
2021-09-22 09:56:02 +02:00
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
xsk: add helper to get &xdp_desc's DMA and meta pointer in one go
Currently, when your driver supports XSk Tx metadata and you want to
send an XSk frame, you need to do the following:
* call external xsk_buff_raw_get_dma();
* call inline xsk_buff_get_metadata(), which calls external
xsk_buff_raw_get_data() and then do some inline checks.
This effectively means that the following piece:
addr = pool->unaligned ? xp_unaligned_add_offset_to_addr(addr) : addr;
is done twice per frame, plus you have 2 external calls per frame, plus
this:
meta = pool->addrs + addr - pool->tx_metadata_len;
if (unlikely(!xsk_buff_valid_tx_metadata(meta)))
is always inlined, even if there's no meta or it's invalid.
Add xsk_buff_raw_get_ctx() (xp_raw_get_ctx() to be precise) to do that
in one go. It returns a small structure with 2 fields: DMA address,
filled unconditionally, and metadata pointer, non-NULL only if it's
present and valid. The address correction is performed only once and
you also have only 1 external call per XSk frame, which does all the
calculations and checks outside of your hotpath. You only need to
check `if (ctx.meta)` for the metadata presence.
To not copy any existing code, derive address correction and getting
virtual and DMA address into small helpers. bloat-o-meter reports no
object code changes for the existing functionality.
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Link: https://patch.msgid.link/20250206182630.3914318-5-aleksander.lobakin@intel.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-02-06 19:26:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-07 13:20:26 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-07 13:20:26 +02:00
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-17 11:22:55 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-04-23 20:01:56 +02:00
|
|
|
|
2023-04-06 01:59:18 +02:00
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
2024-12-03 18:37:26 +01:00
|
|
|
|
2023-07-19 15:24:04 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:21:02 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-10-07 14:24:55 +02:00
|
|
|
|
|
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
2024-10-07 14:24:55 +02:00
|
|
|
|
|
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
2024-10-07 14:24:55 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-04-16 13:29:25 +02:00
|
|
|
|
2024-10-07 14:24:55 +02:00
|
|
|
|
2021-09-22 09:56:06 +02:00
|
|
|
|
|
|
|
|
|
2023-11-27 11:03:08 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-20 21:20:53 +02:00
|
|
|
|