0
Fork 0
mirror of https://github.com/Droid-VM/crosvm.git synced 2026-11-17 06:58:03 +00:00
Commit graph

7 commits

Author SHA1 Message Date
lateautumn233
fe24182b62 virtio-media: MediaCodec decoders within a pixel budget, asked to decode as fast as they can
A 4096x2160 hardware decoder holds about 500 MB of the host's memory in
its own output buffers, which no guest memory limit bounds: macOS's
lock-screen video opened a new 4K session every few seconds without
closing the last, and five took 2.5 GB in half a minute until the host's
low-memory killer ended the VM. The hardware decoders of the process
now share a budget of two 4K pictures (CROSVM_MEDIA_DECODE_MAX_MPX
megapixels); a session past it is refused at STREAMON(OUTPUT) with
EBUSY, which a guest takes as a decoder it cannot have.

A guest's timestamps are its own (macOS's VideoToolbox host sends frame
numbers a second apart), so a codec setting its clocks by them read
1 fps and a 4K Main10 stream came out at ~25 fps: KEY_PRIORITY 0 at the
highest KEY_OPERATING_RATE, and configure retried without the two keys
where a component refuses them. KEY_MAX_INPUT_SIZE is capped at 2 MiB
(23 x 6.3 MiB of input dma-buf per 4K HEVC session before); a larger
access unit goes in over several buffers as before.
2026-11-04 11:20:40 -05:00
HuJK
8f9ffc3f92 media: put the QTI AVC decoder in decode order so the reorder cannot deadlock a client (D91)
An adaptive/MSE H.264 client wedges against c2.qti.avc.decoder a few
frames into a stream: it feeds 7-8 access units, then blocks waiting for
the picture it has just submitted, and that picture is still inside the
decoder's display-reorder window. D85 measured the client side (8 AUs in,
5 frames out, then an idle drain and "Stateful sync failed"); D91 measured
the decoder side directly, host-direct with an NDK probe and no VM, and
found the two numbers meet exactly at the boundary. The metric that
decides it is self-delay -- how many further AUs have to be fed before the
picture for AU i comes back. On a High-profile 1080p stream the QTI AVC
decoder's default is p50 4, p95 7, max 7 (54 of 300 pictures need the full
7), and 3 pictures are never released at all without an EOS. A client
whose feed-ahead budget is 7-8 therefore has no margin, and its only way
out is DEC_CMD_STOP -- a drain that resets the DPB and cuts the reference
chain, which is why D85 ends in a broken stream rather than a stall.

vendor.qti-ext-dec-picture-order.enable=1 removes the cause instead of
working around it: the component stops holding pictures for display order
and emits in decode order. Measured on the same stream: self-delay
p50 = p95 = 1 (295 of 300 pictures return on the same step), max 5 and
only during the opening burst, nothing held to EOS; nonmono 157 -> 0; the
EOS tail 3 -> 0; 300/300 frames out with no errors. The pictures
themselves are untouched -- per-frame md5 aligned by PTS is identical in
300 of 300 frames, and the negotiated output format is byte-for-byte the
same (stride, slice-height, buffer counts, crop, colour format), so
nothing downstream in libva or the driver sees a new surface. Decode order
is in any case what this backend wants: every CAPTURE buffer is claimed by
its access unit's own timestamp tag and the VA client reorders by POC
itself, so display order is work the guest side has to undo.

What this is not: KEY_LOW_LATENCY and the c2.qti.avc.decoder.low_latency
sibling were both measured and both are inert here -- neither moves first
output, steady lag, nonmono or the EOS tail by a single frame, and the
sibling additionally drops the bitrate ceiling from 220 to 70 Mbps and
halves blocks-per-second. KEY_LOW_LATENCY is left exactly as it was (set
only when the component advertises FEATURE_LowLatency, which plain AVC
does not), and no drain or duplicate-AU trick is needed.

Two properties of the key shape the code. It is only honoured in the
MediaFormat passed to configure(): an AMediaCodec_setParameters after
start() returns success and does nothing. And the component does not echo
it on the input format, it reflects it on the output one -- reading the
input format is what made an earlier probe record this key as unsupported.
So the ledger reports the value read back from output_format() after
start, not the value we set (configure() returning OK proves nothing about
a vendor key; unknown ones are dropped in silence), and the "started for
..." line gains a trailing ", picture-order on|off" with everything before
it unchanged. Absent or unreadable reads as off.

Scoped to video/avc. The key also survives readback on HEVC, VP9 and AV1,
but its effect there is unmeasured, so those components keep the behaviour
they ship with today.

For the deploy WP, the bar: every AVC session's ledger line must say
picture-order on; h264.html must stay at a single session 300 in / 300 out
with gbm-dmabuf and zero idle drains; Firefox ESR on adaptive High MSE
must reach in == out in one session with zero idle drains, zero "Stateful
sync failed" and no software fallback; and the three H.264 canonical clips
must stay bit-exact (bf32f00e, d0c7f223, 3db3538b) -- the decoder now
hands libva-v4l2 its buffers in a different order, and that claim_locked
path is the one thing D91 could not exercise host-direct. VP9 and AV1 must
be bit-identical, unchanged by construction.

Investigated in logs/vpu_wp/D91-lowlat-probe.md (§2-§6), with the client
side in logs/vpu_wp/D85-drain.md. Not built or deployed here; the
orchestrator builds crosvm and a later WP verifies on the phone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011BkN7vYhuc1w9iVuqsHzSn
2026-10-28 06:23:29 +08:00
HuJK
b60d066aa4 android_codec: release_output ends the output borrow, and a bottom-up layout is refused, not indexed
Two review-m6 findings in the crate (logs/vpu_wp/review-m6.md).

R6-13 (nit): Codec::output_buffer returned a slice borrowed from &self while
release_output also took &self, so `let src = codec.output_buffer(i)?;
codec.release_output(i)?; use(src)` compiled: a use after release
expressible with no unsafe. release_output now takes &mut self, which ends
every output_buffer borrow at the release by construction; the two callers
in devices (pump_output, take_output) and codec_probe already hold the
codec mutably, and are adjusted here only where the signature requires it.

R6-9 (minor): MediaImage2::plane_end clamps a negative increment out of its
sum, so check_bounds looked at the first row only of a bottom-up image-data
layout (which the format allows) and tight_nv12_rows then computed a row
start below zero, cast it to usize and panicked on the index -- in the
media helper, i.e. the VM down -- where the function's contract is that
nothing is emitted for a refused layout. A plane_start companion finds the
lowest byte a plane reaches, and check_bounds refuses a layout that reaches
before the buffer with the new ImageError::BelowStart. A bottom-up layout
whose offsets account for the walk is still walked, rows in the order the
picture has them. Not observed on 5566 (B5 §5.2: every rowInc positive); the
codec controls the value, not the guest.

Test: a_bottom_up_layout_is_walked_or_refused_never_indexed_below_zero (31
passed in deploy/vpu/harness.sh acc).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011BkN7vYhuc1w9iVuqsHzSn
2026-10-28 06:23:29 +08:00
HuJK
a72caf6cd4 android_codec: codec_probe takes hex levels, keeps its banners off stdout, and leaves through _exit
Three B5 findings (logs/vpu_wp/B5-acceptance.md §6). --level 0x1000 as
M6-probe's own recipe spells it was refused with "invalid digit found in
string" (D25 item 1): --level and --profile now take the 0x form
MediaCodecConstants.h writes them in, and the decimal one still. The two banner
lines (running as uid, libmediandk loaded) went to stdout ahead of the --json
document, so `list --json > f.json` was not JSON (D25 item 2): they go to
stderr. And `list` returned 139 after printing everything (D24):
AMediaCodecStore's file-scope vector of AMediaCodecInfo is destroyed by
__cxa_finalize at exit(), after its allocator arena, and the process dies
inside libmediandk. After a Store walk main flushes stdout and stderr and
leaves through _exit, which runs no destructor; the exit code is what it was.
The other commands never touch the Store and exit as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011BkN7vYhuc1w9iVuqsHzSn
2026-10-28 06:23:29 +08:00
HuJK
25a803358e android_codec: an event carries the generation it was posted under, and the encoder's keys
B5 found the HEVC decoder handing out a dead output index after flush() +
start() in five of seven runs (logs/vpu_wp/B5-acceptance.md D23): a callback
posted before the flush was delivered after it, and a getOutputBuffer on its
index returned null. The queue now stamps every event with a generation that
flush_and_restart bumps under the queue lock -- after AMediaCodec_flush has
returned, so an event posted while the flush ran is old too -- and take_events
hands out only the current generation, counting what it dropped
(Codec::stale_events). The stamp cannot catch a callback the NDK looper had
already queued before the flush: that one arrives afterwards, with the new
generation and a dead index, and the docs now say what a consumer does with the
null it gets (skip it; never a session error). A unit test drives the queue
directly, since a Codec needs the NDK and the generation logic does not.

Also the constants the encoder backend configures with: COLOR_RANGE_*,
COLOR_STANDARD_*, COLOR_TRANSFER_* (MediaCodecConstants.h:1036-1045) and the
three encoder keys that have no AMEDIAFORMAT_KEY_* symbol
(prepend-sps-pps-to-idr-frames, video-qp-min, video-qp-max).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011BkN7vYhuc1w9iVuqsHzSn
2026-10-28 06:23:29 +08:00
HuJK
7b567e8a71 android_codec: let a consumer wake its own descriptor, and repack NV12 a row at a time
Three additions the MediaCodec decoder backend (M6-backend) needs, none of which
changes what codec_probe does.

A wake hook on the async queue. The fork's decoder device keeps the poll
descriptor on the device session and hands the backend a DecoderSink to bump
(M6-crate section 2), while Codec's four callbacks bump the crate's own eventfd.
Rather than a thread sitting between the two, Codec::set_wake_hook installs a
closure the callbacks run after queueing their event -- one eventfd write, nothing
else, as NdkMediaCodec.h:506 asks of a callback -- so the whole session can run on
the device's worker thread.

tight_nv12_rows, the same plane walk as tight_nv12 handing out one row at a time.
The backend writes into a CAPTURE buffer the guest also maps, and does so through
a raw pointer rather than a slice over guest-visible memory (the camera backend's
rule); a Vec destination would have meant a second copy per frame. tight_nv12 is
now the Vec wrapper over it, so the nine existing tests cover both.

queue_input_with_flags: the fill closure returns the buffer flags along with the
bytes used. Whether a bitstream buffer is parameter sets alone (BUFFER_FLAG_
CODEC_CONFIG) is decided on the bytes once they are in the codec's own buffer,
never by reading the guest's memory twice.

MediaImage2::semiplanar is codec_probe's assumed_nv12 moved into the crate, so the
backend's fallback for a component that publishes no image-data is the probe's.
Two tests added (29 in the acc harness).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011BkN7vYhuc1w9iVuqsHzSn
2026-10-28 06:23:28 +08:00
HuJK
423ddccebf android_codec: the MediaCodec NDK crate and codec_probe (M6, design 7.2)
The decoder and encoder devices (VPU_DESIGN.md 7.2, 7.3) must map the
phone's real codec capabilities, not a table, and must drive MediaCodec in
asynchronous mode from a device worker. Before either device exists, a
standalone probe has to enumerate what the platform reports and prove decode
and encode round trips through the NDK under the app uid. This is that
crate, the android_camera pattern applied to libmediandk:

- lib.rs: ndk_api! bindings resolved with dlopen (libmediandk, libbinder_ndk;
  the binder thread pool is started at load because MediaCodec serves a
  BnResourceManagerClient). Everything up to API 28 is required; the API 36
  AMediaCodecStore_* / AMediaCodecInfo_* / ACodec*Capabilities_* surface and
  the AMediaCodecInfo_FEATURE_* data globals are optional groups, so the
  library loads on an older Android and list_codecs falls back to
  create*ByType per mime. list_codecs walks the Store (media types, every
  decoder and encoder with kind/type/vendor/instances, video ranges and
  alignment, per-size rate checks, performance-point coverage, encoder
  quality/complexity/bitrate modes, features) and asks isFormatSupported
  about a candidate profile/level table, since the NDK has no
  getProfileLevels (survey 2.3). Codec is one session: the four async
  callbacks only push onto a mutex-protected queue and signal a base::Event
  eventfd (design 7.2's rule), with flush_and_restart for the async seek.
  MediaFormat wraps AMediaFormat with the keys the survey names, including
  the literal "image-data".
- image.rs: MediaImage2 parsed from the 104-byte "image-data" blob, and
  tight_nv12, the one per-frame repack the decoder device will make
  (memcpy rows for NV12, byte swap for NV21, interleave for planar, generic
  walk otherwise), plus rebased() for survey open question 1.
- bitstream.rs: Annex-B access-unit splitter (H.264/H.265, spec boundary
  rules, csd extraction) and IVF reader/writer with VP8/VP9 key-frame bits.
- synth.rs: the moving test pattern, the padded-layout packer for encoder
  input (stride, slice-height, semiplanar or planar), luma PSNR and digest.
- probe.rs: codec_probe list [--json] / decode / encode / roundtrip, with
  --uid as camera_probe, --csd config|format|inband, the mid-stream
  flush-then-start, the mPlane[0].mOffset verdict, and a >= 30 dB luma
  PSNR round-trip check.

Workspace member; Android.bp mirrors android_camera (installable: false,
apex_available com.android.virt, no libmediandk in shared_libs, phony
codec_probe). Host build, 27 unit tests, clippy and rustfmt pass in the
scratch harness (logs/vpu_wp/scratch-m6); nothing has run on a phone yet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011BkN7vYhuc1w9iVuqsHzSn
2026-10-28 06:23:28 +08:00