mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
synced 2026-11-15 19:56:03 +00:00
drm/vkms: Fix limited-range YCbCr to RGB conversion scaling
The limited-/studio-range YCbCr to RGB conversion matrices were generated
with colour.matrix_YCbCr(is_legal=True, bits=8), which normalises the
narrow range by 2^bits (256 for 8-bit) rather than by the full-range
maximum 2^bits - 1 (255). As a result the luma scale was 256/219 and the
chroma scale 256/224 instead of the correct 255/219 and 255/224.
This over-scales every limited-range conversion by a factor of 256/255,
producing an error of up to ~3/255 (8-bit) and causing IGT tests that
compare software-computed surfaces against VKMS-processed ones to fail.
Recompute the three limited-range matrices from first principles rather
than relying on colour's is_legal scaling:
1. Start from the standard ITU-R YCbCr -> RGB relations for the luma
weights Kr, Kb (Kg = 1 - Kr - Kb) of each encoding (BT.601, BT.709
and BT.2020 non-constant luminance). These are the same relations
that produce the existing full-range matrices.
2. Expand the studio input range to full range relative to a full-range
maximum of 2^n - 1: the luma coefficient by 255/(235 - 16) and the
chroma coefficients by 255/(240 - 16). This matches the DRM UAPI
definition and IGT's igt_ycbcr_to_rgb_matrix().
3. Convert each coefficient to S31.32 fixed point (round(coeff * 2^32)).
Update the limited-range reference values in the vkms-format KUnit test
accordingly. Provide a python script to show how the kunit test value
are calculated.
Fixes: fe22d21e93 ("drm/vkms: Add YUV support")
Assisted-by: Copilot:claude-opus-4.8
Signed-off-by: Harry Wentland <harry.wentland@amd.com>
Reviewed-by: Alex Hung <alex.hung@amd.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Link: https://patch.msgid.link/20260814211816.1219597-3-harry.wentland@amd.com
This commit is contained in:
parent
f6c58e4ae8
commit
8af5fccd9e
3 changed files with 144 additions and 36 deletions
Loading…
Add table
Add a link
Reference in a new issue