[Why]
YELLOW_CARP_B0 address was not correct
[How]
Set YELLOW_CARP_B0 to 0x1A.
Reviewed-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jude Shih <shenshih@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Condition variable sometimes terminated unexpectedly
[How]
Use wait_for_completion_timeout to avoid unexpected termination of CV
Reviewed-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jude Shih <shenshih@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[why]
1. HPD callback function has deadlock problem
2. HPD status is not assigned
3. There is crash due to null pointer
4. link_enc is NULL in DPIA case
[How]
1. Fix deadlock problem by moving it out of the
drm_modeset_lock
2. Assign HPD status from the notify of outbox
from dmub FW
3. Fix the crash by checking if pin or enc exists
4. Use link_enc_cfg_get_link_enc_used_by_link to
dynamically assign
Reviewed-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jude Shih <shenshih@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why & How]
1. Remove unnecessary dummy interrupt source for USB4 HPD & HPD RX
2. Adjust parameter for DPCD writing of link training process of DPIA link
3. Adjust specific AUX defer delay for DPIA link
Reviewed-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Meenakshikumar Somasundaram <meenakshikumar.somasundaram@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why & How]
Additional debug flags that can be useful for testing USB4 DP
link training.
Add flags:
- 0x2 : Forces USB4 DP link to non-LTTPR mode
- 0x4 : Extends status read intervals to about 60s.
Reviewed-by: Meenakshikumar Somasundaram <meenakshikumar.somasundaram@amd.com>
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
DIB_BE_CNTL<i>.DIG_HPD_SELECT selects the HPD block being used
by the display endpoint assigned to DIG<i>. In the case of USB4
display endpoints, no physical HPD block is assigned.
[How]
Setting DIB_BE_CNTL<i>.DIG_HPD_SELECT to 5 indicates that no HPD
is assigned to a display endpoint. Firmware decrements the
HPD_SELECT value by 1 before writing it to the register.
Reviewed-by: Meenakshikumar Somasundaram <meenakshikumar.somasundaram@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
To process SET_CONFIG transactions with DMUB using inbox1 and
outbox1 mail boxes.
[How]
1) DMUB posts SET_CONFIG reply as an Outbox1 message of type
DMUB_OUT_CMD__SET_CONFIG_REPLY.
2) The dmub async to sync mechanism for AUX is modified to accommodate
SET_CONFIG commands for both command issue and reply code paths.
Reviewed-by: Nicholas Kazlauskas <Nicholas.Kazlauskas@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jude Shih <shenshih@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why & How]
To add support for dpia debug options.
Reviewed-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Meenakshikumar Somasundaram <meenakshikumar.somasundaram@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
We requires information from DPCD in order to identify USB4 DP
tunneling targets.
[How]
Add USB4 DP tunneling fields to DPCD struct and populate these fields
during sink detection.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
To process SET_CONFIG transactions with DMUB using inbox1 and
outbox1 mail boxes.
[How]
1) Added inbox1 DPIA command subtype DMUB_CMD__DPIA_SET_CONFIG_ACCESS to
issue SET_CONFIG command to DMUB in dc_process_dmub_set_config_async().
DMUB processes the command with DPIA sends reply back immediately or
in an outbox1 message triggering an outbox1 interrupt to driver.
2) DMUB posts SET_CONFIG reply as an Outbox1 message of type
DMUB_OUT_CMD__SET_CONFIG_REPLY.
3) The dmub async to sync mechanism for AUX is modified to accommodate
SET_CONFIG commands for both command issue and reply code paths.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Meenakshikumar Somasundaram <meenakshikumar.somasundaram@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why & How]
Clear training pattern sequence for hop in display path once clock
recovery and equalization phases of DP tunnel link training completed.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Equalisation is the mandatory second phase of DisplayPort link training
over a USB4 DP tunnel.
[How]
Implement equalisation phase for DP tunneled over USB4 in DPIA
training module.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Clock recovery is the mandatory first phase of DP link training.
[How]
- Implement clock recovery phase in DPIA training module.
- Add helper functions for building SET_CONFIG messages.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Training settings need to be applied to DPIA link at start of each
training loop. Note: FEC readiness should be configured before link
training while FEC enablement should be configured once training is
complete.
[How]
- Implement DPIA link configuration function.
- Account for dynamically assigned link encoders during link
configuration.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Training of DPIA link differs enough from that of conventional
DP link to warrant a separate implementation.
[How]
- Implement top-level of DPIA training loop.
- Make functions shared between DP and DPIA link training "public".
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why & How]
Conventional links are trained with fallback during sink detection.
Have DPIA links trained with fallback too.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[why & how]
Driver does not need to train the first hop.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[why & how]
1. Add stub for getting tunneling device data
2. Add check for phy_repeater_cnt < 0xff to LTTPR check
3. Add two more bits of information to DPIA links
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[why & how]
Add stub for DPIA link training and define new DPIA DMUB commands
to support it.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[why & how]
We will need a way to distinguish physically connected
links and DPIA endpoints.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[WHY]
To add support for HPD & HPD RX interrupt handling for USB4 DPIA in
YELLOW_CARP_B0. USB4 DPIA HPD & HPD RX interrupts are issued from DMUB to
driver as a outbox1 message.
[HOW]
1) Created get_link_index_from_dpia_port_index() to retrieve link index
from dpia port index for HPD & HPD RX dmub notifications.
2) Added DMUB HPD & HPD RX handling in dmub_srv_stat_get_notification().
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Meenakshikumar Somasundaram <meenakshikumar.somasundaram@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[WHY]
To enable dc links for USB4 DPIA ports and AUX command tunneling
for YELLOW_CARP_B0.
[HOW]
1) Created dc links for all USB4 DPIA ports in create_links().
dc_link_construct() implementation is split for legacy DDC and DPIAs.
As usb4 has no ddc, ddc->ddc_pin will be set to NULL for its dc link
and this parameter will be used to identify the dc links as DPIA. The
dc link for DPIA is further to be enhanced with implementation for link
encoder and link initialization.
2) usb4_dpia_count in struct resource_pool will be initialized to 4 in
dcn31_resource_construct() if the DCN is YELLOW_CARP_B0.
3) Enabled DMUB AUX via outbox for YELLOW_CARP_B0.
Reviewed-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Meenakshikumar Somasundaram <meenakshikumar.somasundaram@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why & How]
USB4 endpoints are dynamically mapped. We create additional link
encoders for USB4 use when DC is created and destroy them when DC
is destructed
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Wayne Lin <Wayne.Lin@amd.com>
Acked-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Acked-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
ind_block_64b_no_128bcl means INDEP_64B && INDEP_128B &&
MAX_COMPRESSED_BLOCK_SIZE == 64B. Only used by gfx10.3.
ind_block_64b means INDEP_64B && !INDEP_128B &&
MAX_COMPRESSED_BLOCK_SIZE == 64B. Only used by gfx9 and gfx10.
Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
During DQE's promotion test, error appears in dmesg at boot
on dcn3.1;
[How]
Add NULL pointor check for the pointor to the amdgpu_dm_connector;
Reviewed-by: Nicholas Kazlauskas <Nicholas.Kazlauskas@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Leo (Hanghong) Ma <hanghong.ma@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Trying to enable multiple displays simultaneously exposed shortcomings
with the algorithm for dynamic link encoder assignment.
The main problems were:
- Assuming stream order remained constant across states would
sometimes lead to invalid DIG encoder assignment.
- Incorrect logic for deciding whether or not a DIG could support a
stream would also sometimes lead to invalid DIG encoder assignment.
- Changes in encoder assignment were wholesale while updating of the
pipe backend is incremental. This would lead to the hardware state
not matching the software state even with valid encoder assignments.
[How]
The following changes fix the identified problems.
- Use stream pointer rather than stream index to track streams across
states.
- Fix DIG compatibility check by examining the link signal type
rather than the stream signal type.
- Modify assignment algorithm to make incremental updates so software
and hardware states remain coherent.
Additionally:
- Add assertions and an encoder assignment validation
function link_enc_cfg_validate() to detect potential problems with
encoder assignment closer to their root cause.
- Reduce the frequency with which the assignment algorithm is
executed. It should not be necessary for fast state validation.
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Jimmy Kizito <Jimmy.Kizito@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why & How]
The codes to blank all dp display have been called many times,
so add a helper in dc_link to make it more concise.
Reviewed-by: Aric Cyr <Aric.Cyr@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Leo (Hanghong) Ma <hanghong.ma@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
This version brings along following fixes:
- New firmware version
- Fix DMUB problems on stress test.
- Improve link training by skip overrride for preferred link
- Refinement of FPU code structure for DCN2
- Fix 3DLUT skipped programming
- Fix detection of 4 lane for DPALT
- Fix dcn3 failure due to dmcbu_abm not created
- Limit display scaling to up to 4k for DCN 3.1
- Add helper for blanking all dp displays
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Aric Cyr <aric.cyr@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
DPALT detection for B0 PHY has its own set of RDPCSPIPE registers
[How]
Use RDPCSPIPE registers to detect if DPALT lane is 4 lane
Reviewed-by: Charlene Liu <Charlene.Liu@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Hansen <Hansen.Dsouza@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[why]
The existing limit was mistakenly bigger than 4k for DCN 3.1
Reviewed-by: Zhan Liu <Zhan.Liu@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Nikola Cornij <nikola.cornij@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Current FPU code for DCN2x is located under dml/dcn2x.
This is not aligned with DC's general source tree
structure.
[How]
Move FPU code for DCN2x to dml/dcn20.
Reviewed-by: Rodrigo Siqueira <Rodrigo.Siqueira@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Qingqing Zhuo <qingqing.zhuo@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
[Why]
Overriding link setting inside override_training_settings
result in fallback link settings being ignored. This can
potentially cause link training to always fail and consequently
result in an infinite loop of link training to occur in
dp_verify_link_cap during detection.
[How]
Since preferred link settings are already considered inside
decide_link_settings, skip the check in override_training_settings
to avoid infinite link training loops.
Reviewed-by: Wenjing Liu <wenjing.liu@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: George Shen <george.shen@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The size of the exception stacks was increased by the commit in Fixes,
resulting in stack sizes greater than a page in size. The #VC exception
handling was only mapping the first (bottom) page, resulting in an
SEV-ES guest failing to boot.
Make the #VC exception stacks part of the default exception stacks
storage and allocate them with a CONFIG_AMD_MEM_ENCRYPT=y .config. Map
them only when a SEV-ES guest has been detected.
Rip out the custom VC stacks mapping and storage code.
[ bp: Steal and adapt Tom's commit message. ]
Fixes: 7fae4c24a2 ("x86: Increase exception stack sizes")
Signed-off-by: Borislav Petkov <bp@suse.de>
Tested-by: Tom Lendacky <thomas.lendacky@amd.com>
Tested-by: Brijesh Singh <brijesh.singh@amd.com>
Link: https://lkml.kernel.org/r/YVt1IMjIs7pIZTRR@zn.tnic
Currently, the driver doesn't set the PCP-based priority for DCT, hence
DCT response packets are transmitted without user priority.
Fix it by setting user provided priority in the eth_prio field in the DCT
context, which in turn sets the value in the transmitted packet.
Fixes: 776a3906b6 ("IB/mlx5: Add support for DC target QP")
Link: https://lore.kernel.org/r/5fd2d94a13f5742d8803c218927322257d53205c.1633512672.git.leonro@nvidia.com
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Maor Gottlieb <maorg@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
The valid bit for extended CQE's written by HW is retrieved from the
incorrect quad-word. This leads to missed completions for any UD traffic
particularly after a wrap-around.
Get the valid bit for extended CQE's from the correct quad-word in the
descriptor.
Fixes: 551c46edc7 ("RDMA/irdma: Add user/kernel shared libraries")
Link: https://lore.kernel.org/r/20211005182302.374-1-shiraz.saleem@intel.com
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Hengqi Chen says:
====================
bpf_{map,program}__{prev,next} don't follow the libbpf API naming
convention. Deprecate them and replace them with a new set of APIs
named bpf_object__{prev,next}_{program,map}.
v1->v2: [0]
* Addressed Andrii's comments
[0]: https://patchwork.kernel.org/project/netdevbpf/patch/20210906165456.325999-1-hengqi.chen@gmail.com/
====================
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
BPF objects are not reloadable after unload. Users are expected to use
bpf_object__close() to unload and free up resources in one operation.
No need to expose bpf_object__unload() as a public API, deprecate it
([0]). Add bpf_object__unload() as an alias to internal
bpf_object_unload() and replace all bpf_object__unload() uses to avoid
compilation errors.
[0] Closes: https://github.com/libbpf/libbpf/issues/290
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Song Liu <songliubraving@fb.com>
Link: https://lore.kernel.org/bpf/20211002161000.3854559-1-hengqi.chen@gmail.com
Replace deprecated bpf_{map,program}__next APIs with newly added
bpf_object__next_{map,program} APIs, so that no compilation warnings
emit.
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Song Liu <songliubraving@fb.com>
Link: https://lore.kernel.org/bpf/20211003165844.4054931-3-hengqi.chen@gmail.com
This adds a section to the documentation for libbpf
naming convention which describes how to document
API features in libbpf, specifically the format of
which API doc comments need to conform to.
Signed-off-by: Grant Seltzer <grantseltzer@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Song Liu <songliubraving@fb.com>
Link: https://lore.kernel.org/bpf/20211004215644.497327-1-grantseltzer@gmail.com
Deprecate bpf_{map,program}__{prev,next} APIs. Replace them with
a new set of APIs named bpf_object__{prev,next}_{program,map} which
follow the libbpf API naming convention ([0]). No functionality changes.
[0] Closes: https://github.com/libbpf/libbpf/issues/296
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Song Liu <songliubraving@fb.com>
Link: https://lore.kernel.org/bpf/20211003165844.4054931-2-hengqi.chen@gmail.com
Currently the recursion test is hooking __htab_map_lookup_elem
function, which is invoked both from bpf_prog and bpf syscall.
But in our kernel build, the __htab_map_lookup_elem gets inlined
within the htab_map_lookup_elem, so it's not trigered and the
test fails.
Fixing this by using htab_map_delete_elem, which is not inlined
for bpf_prog calls (like htab_map_lookup_elem is) and is used
directly as pointer for map_delete_elem, so it won't disappear
by inlining.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Song Liu <songliubraving@fb.com>
Link: https://lore.kernel.org/bpf/YVnfFTL/3T6jOwHI@krava
Recent-ish versions of make do no longer consider number signs ("#") as
comment symbols when they are inserted inside of a macro reference or in
a function invocation. In such cases, the symbols should not be escaped.
There are a few occurrences of "\#" in libbpf's and samples' Makefiles.
In the former, the backslash is harmless, because grep associates no
particular meaning to the escaped symbol and reads it as a regular "#".
In samples' Makefile, recent versions of make will pass the backslash
down to the compiler, making the probe fail all the time and resulting
in the display of a warning about "make headers_install" being required,
even after headers have been installed.
A similar issue has been addressed at some other locations by commit
9564a8cf42 ("Kbuild: fix # escaping in .cmd files for future Make").
Let's address it for libbpf's and samples' Makefiles in the same
fashion, by using a "$(pound)" variable (pulled from
tools/scripts/Makefile.include for libbpf, or re-defined for the
samples).
Reference for the change in make:
https://git.savannah.gnu.org/cgit/make.git/commit/?id=c6966b323811c37acedff05b57
Fixes: 2f38304127 ("libbpf: Make libbpf_version.h non-auto-generated")
Fixes: 07c3bbdb1a ("samples: bpf: print a warning about headers_install")
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20211006111049.20708-1-quentin@isovalent.com
The BPF core defines a __weak bpf_jit_compile() dummy function already
which should only be overridden by JITs if they actually implement a
legacy cBPF JIT. Given arm implements an eBPF JIT, this stub is not
needed.
Now that MIPS cBPF JIT is finally gone, the only JIT left that is still
implementing bpf_jit_compile() is the sparc32 one.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Johan Almbladh says:
====================
This is an implementation of an eBPF JIT for MIPS I-V and MIPS32/64 r1-r6.
The new JIT is written from scratch, but uses the same overall structure
as other eBPF JITs.
Before, the MIPS JIT situation looked like this.
- 32-bit: MIPS32, cBPF-only, tests fail
- 64-bit: MIPS64r2-r6, eBPF, tests fail, incomplete eBPF ISA support
The new JIT implementation raises the bar to the following level.
- 32/64-bit: all MIPS ISA, eBPF, all tests pass, full eBPF ISA support
Overview
--------
The implementation supports all 32-bit and 64-bit eBPF instructions
defined as of this writing, including the recently-added atomics. It is
intended to provide good performance for native word size operations,
while also being complete so the JIT never has to fall back to the
interpreter. The new JIT replaces the current cBPF and eBPF JITs for MIPS.
The implementation is divided into separate files as follows. The source
files contains comments describing internal mechanisms and details on
things like eBPF-to-CPU register mappings, so I won't repeat that here.
- jit_comp.[ch] code shared between 32-bit and 64-bit JITs
- jit_comp32.c 32-bit JIT implementation
- jit_comp64.c 64-bit JIT implementation
Both the 32-bit and 64-bit versions map all eBPF registers to native MIPS
CPU registers. There are also enough unmapped CPU registers available to
allow all eBPF operations implemented natively by the JIT to use only CPU
registers without having to resort to stack scratch space.
Some operations are deemed too complex to implement natively in the JIT.
Those are instead implemented as a function call to a helper that performs
the operation. This is done in the following cases.
- 64-bit div and mod on a 32-bit CPU
- 64-bit atomics on a 32-bit CPU
- 32-bit atomics on a 32-bit CPU that lacks ll/sc instructions
CPU errata workarounds
----------------------
The JIT implements workarounds for R10000, Loongson-2F and Loongson-3 CPU
errata. For the Loongson workarounds, I have used the public information
available on the matter.
Link: https://sourceware.org/legacy-ml/binutils/2009-11/msg00387.html
Testing
-------
During the development of the JIT, I have added a number of new test cases
to the test_bpf.ko test suite to be able to verify correctness of JIT
implementations in a more systematic way. The new additions increase the
test suite roughly three-fold, with many of the new tests being very
extensive and even exhaustive when feasible.
Link: https://lore.kernel.org/bpf/20211001130348.3670534-1-johan.almbladh@anyfinetworks.com/
Link: https://lore.kernel.org/bpf/20210914091842.4186267-1-johan.almbladh@anyfinetworks.com/
Link: https://lore.kernel.org/bpf/20210809091829.810076-1-johan.almbladh@anyfinetworks.com/
The JIT has been tested by running the test_bpf.ko test suite in QEMU with
the following MIPS ISAs, in both big and little endian mode, with and
without JIT hardening enabled.
MIPS32r2, MIPS32r6, MIPS64r2, MIPS64r6
For the QEMU r2 targets, the correctness of pre-r2 code emitted has been
tested by manually overriding each of the following macros with 0.
cpu_has_llsc, cpu_has_mips_2, cpu_has_mips_r1, cpu_has_mips_r2
Similarly, CPU errata workaround code has been tested by enabling the
each of the following configurations for the MIPS64r2 targets.
CONFIG_WAR_R10000
CONFIG_CPU_LOONGSON3_WORKAROUNDS
CONFIG_CPU_NOP_WORKAROUNDS
CONFIG_CPU_JUMP_WORKAROUNDS
The JIT passes all tests in all configurations. Below is the summary for
MIPS32r2 in little endian mode.
test_bpf: Summary: 1006 PASSED, 0 FAILED, [994/994 JIT'ed]
test_bpf: test_tail_calls: Summary: 8 PASSED, 0 FAILED, [8/8 JIT'ed]
test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
According to MIPS ISA reference documentation, the result of a 32-bit ALU
arithmetic operation on a 64-bit CPU is unpredictable if an operand
register value is not properly sign-extended to 64 bits. To verify the
code emitted by the JIT, the code generation engine in QEMU was modifed to
flip all low 32 bits if the above condition was not met. With this
trip-wire installed, the kernel booted properly in qemu-system-mips64el
and all test_bpf.ko tests passed.
Remaining features
------------------
While the JIT is complete is terms of eBPF ISA support, this series does
not include support for BPF-to-BPF calls and BPF trampolines. Those
features are planned to be added in another patch series.
The BPF_ST | BPF_NOSPEC instruction currently emits nothing. This is
consistent with the behavior if the MIPS interpreter and the existing
eBPF JIT.
Why not build on the existing eBPF JIT?
---------------------------------------
The existing eBPF JIT was originally written for MIPS64. An effort was
made to add MIPS32 support to it in commit 716850ab10 ("MIPS: eBPF:
Initial eBPF support for MIPS32 architecture."). That turned out to
contain a number of flaws, so eBPF support for MIPS32 was disabled in
commit 36366e367e ("MIPS: BPF: Restore MIPS32 cBPF JIT").
Link: https://lore.kernel.org/bpf/5deaa994.1c69fb81.97561.647e@mx.google.com/
The current eBPF JIT for MIPS64 lacks a lot of functionality regarding
ALU32, JMP32 and atomic operations. It also lacks 32-bit CPU support on a
fundamental level, for example 32-bit CPU register mappings and o32 ABI
calling conventions. For optimization purposes, it tracks register usage
through the program control flow in order to do zero-extension and sign-
extension only when necessary, a static analysis of sorts. In my opinion,
having this kind of complexity in JITs, and for which there is not
adequate test coverage, is a problem. Such analysis should be done by the
verifier, if needed at all. Finally, when I run the BPF test suite
test_bpf.ko on the current JIT, there are errors and warnings.
I believe that an eBPF JIT should strive to be correct, complete and
optimized, and in that order. The JIT runs after the verifer has audited
the program and given its approval. If the JIT then emits code that does
something else, it will undermine the eBPF security model. A simple
implementation is easier to get correct than a complex one. Furthermore,
the real performance hit is not an extra CPU instruction here and there,
but when the JIT bails on an unimplemented eBPF instruction and cause the
whole program to fall back to the interpreter. My reasoning here boils
down to the following.
* The JIT should not contain a static analyzer that tracks branches.
* It is acceptable to emit possibly superfluous sign-/zero-extensions for
ALU32 and JMP32 operations on a 64-bit MIPS to guarantee correctness.
* The JIT should handle all eBPF instructions on all MIPS CPUs.
I conclude that the current eBPF MIPS JIT is complex, incomplete and
incorrect. For the reasons stated above, I decided to not use the existing
JIT implementation.
====================
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
This patch removes the old 32-bit cBPF and 64-bit eBPF JIT implementations.
They are replaced by a new eBPF implementation that supports both 32-bit
and 64-bit MIPS CPUs.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-8-johan.almbladh@anyfinetworks.com
This patch enables the new eBPF JITs for 32-bit and 64-bit MIPS. It also
disables the old cBPF JIT to so cBPF programs are converted to use the
new JIT.
Workarounds for R4000 CPU errata are not implemented by the JIT, so the
JIT is disabled if any of those workarounds are configured.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-7-johan.almbladh@anyfinetworks.com
This patch adds workarounds for the following CPU errata to the MIPS
eBPF JIT, if enabled in the kernel configuration.
- R10000 ll/sc weak ordering
- Loongson-3 ll/sc weak ordering
- Loongson-2F jump hang
The Loongson-2F nop errata is implemented in uasm, which the JIT uses,
so no additional mitigations are needed for that.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-6-johan.almbladh@anyfinetworks.com
