drm: prevent integer overflows in dumb buffer creation helpers

Fix integer overflow issues in the dumb buffer creation path:

1. drm_mode_create_dumb() does not bound width, height, or bpp
   before passing them to driver callbacks.  Downstream helpers
   (e.g. drm_gem_dma_dumb_create_internal) perform pitch/size
   alignment in u32 arithmetic that can overflow for extreme
   values.  Add hard limits: width and height < 8192, bpp <= 32.
   No legitimate software rendering use case exceeds these.

2. drm_mode_align_dumb() uses roundup(pitch, hw_pitch_align)
   without checking for overflow.  If pitch is near U32_MAX,
   roundup() wraps to a small value, making subsequent
   check_mul_overflow() pass with a much smaller pitch than
   intended.  Add an overflow check after roundup.

3. drm_mode_align_dumb() uses ALIGN(size, hw_size_align) which
   only works correctly for power-of-two alignment values.
   Replace with roundup() which works for any alignment.

Suggested-by: Thomas Zimmermann <tzimmermann@suse.de>
Signed-off-by: Rajat Gupta <rajat.gupta@oss.qualcomm.com>
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
This commit is contained in:
Rajat Gupta 2026-05-20 22:11:21 -07:00 • committed by Thomas Zimmermann
commit 5ab62dd368