Good info on DRM_IOCTL_MODE_RMFB. My app preallocates framebuffers and nothing freed until app shutdown. Hardware h265 requires DRM_IOCTL_PRIME_FD_TO_HANDLE once per unique buffer, but that was not the cause.
After much experimenting, have replicated/logged the issue. With my nonblocking DRM_IOCTL_MODE_ATOMIC followed by blocking DRM_IOCTL_WAIT_VBLANK approach, can replicate under bullseye but not bookworm. The triggering event is high page-fault activity that presumably delays the vblank interrupt handler dispatch. If the delay is significant enough, looks like the driver pushes the atomic update to the subsequent frames vblank and any nonblocking DRM_IOCTL_MODE_ATOMIC submitted for that frame receives an EBUSY.
The cause of the excessive page faults was my updated buffer management. To effectively use all available free memory for buffering (returning it to the O/S on demand), completed GOPs are copied from heap to MAP_ANONYMOUS memory. These particular playback tests used local-storage high-bitrate h265 media resulting in fast loading and mmap+memcpy page-fault hotspots. Refactored the code to rate limit the mmaps, and to madvise(MADV_WILLNEED) the memory (which seems to premap the pages vs the memcpy faulting on every page).
Since I cannot easily reproduce under bookworm and faster hardware like Pi5 should make it almost impossible, am considering this resolved for my purposes. Now I can move on to experimenting with the Pi5. Appreciate the feedback. Thanks.
For completeness, here is a successful DRM_IOCTL_MODE_ATOMIC+DRM_IOCTL_WAIT_VBLANK:Believe this is the failure case (note the extra "calculating vblank" lines). EBUSY is never indicated in the logs ironically.
After much experimenting, have replicated/logged the issue. With my nonblocking DRM_IOCTL_MODE_ATOMIC followed by blocking DRM_IOCTL_WAIT_VBLANK approach, can replicate under bullseye but not bookworm. The triggering event is high page-fault activity that presumably delays the vblank interrupt handler dispatch. If the delay is significant enough, looks like the driver pushes the atomic update to the subsequent frames vblank and any nonblocking DRM_IOCTL_MODE_ATOMIC submitted for that frame receives an EBUSY.
The cause of the excessive page faults was my updated buffer management. To effectively use all available free memory for buffering (returning it to the O/S on demand), completed GOPs are copied from heap to MAP_ANONYMOUS memory. These particular playback tests used local-storage high-bitrate h265 media resulting in fast loading and mmap+memcpy page-fault hotspots. Refactored the code to rate limit the mmaps, and to madvise(MADV_WILLNEED) the memory (which seems to premap the pages vs the memcpy faulting on every page).
Since I cannot easily reproduce under bookworm and faster hardware like Pi5 should make it almost impossible, am considering this resolved for my purposes. Now I can move on to experimenting with the Pi5. Appreciate the feedback. Thanks.
For completeness, here is a successful DRM_IOCTL_MODE_ATOMIC+DRM_IOCTL_WAIT_VBLANK:
Code:
[758156.233480] [drm:drm_ioctl [drm]] comm="fotron" pid=64244, dev=0xe201, auth=1, [b]DRM_IOCTL_MODE_ATOMIC[/b][758156.234078] [drm:drm_atomic_state_init [drm]] Allocated atomic state 00000000b781d519[758156.234294] [drm:drm_mode_object_get [drm]] OBJ ID: 346 (2)[758156.234400] [drm:drm_atomic_get_plane_state [drm]] Added [PLANE:117:plane-6] 00000000bd966e44 state to 00000000b781d519[758156.234492] [drm:drm_mode_object_get [drm]] OBJ ID: 339 (1)[758156.234571] [drm:drm_atomic_get_crtc_state [drm]] Added [CRTC:105:crtc-4] 0000000075607465 state to 00000000b781d519[758156.234667] vc4-drm gpu: [drm:drm_atomic_set_fb_for_plane [drm]] Set [FB:367] for [PLANE:117:plane-6] state 00000000bd966e44[758156.234747] [drm:drm_mode_object_get [drm]] OBJ ID: 367 (3)[758156.234816] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 346 (3)[758156.234886] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 367 (4)[758156.234952] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 367 (3)[758156.235068] [drm:drm_mode_object_get [drm]] OBJ ID: 341 (2)[758156.235149] [drm:drm_atomic_get_plane_state [drm]] Added [PLANE:127:plane-7] 000000004d74f5d3 state to 00000000b781d519[758156.235240] vc4-drm gpu: [drm:drm_atomic_set_fb_for_plane [drm]] Set [FB:340] for [PLANE:127:plane-7] state 000000004d74f5d3[758156.235330] [drm:drm_mode_object_get [drm]] OBJ ID: 340 (3)[758156.235406] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 341 (3)[758156.235492] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 340 (4)[758156.235558] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 340 (3)[758156.236689] [drm:drm_atomic_check_only [drm]] checking 00000000b781d519[758156.237078] [drm:drm_atomic_get_private_obj_state [drm]] Added new private object 000000003738f977 state 0000000019c8e742 to 00000000b781d519[758156.237182] vc4-drm gpu: [drm:vc4_atomic_check [vc4]] crtc-4: Trying to find a channel.[758156.237304] vc4-drm gpu: [drm:vc4_atomic_check [vc4]] crtc-4: Already enabled, reusing channel 0.[758156.237505] [drm:drm_atomic_get_private_obj_state [drm]] Added new private object 00000000bd7e6787 state 0000000002466764 to 00000000b781d519[758156.238957] [drm:drm_atomic_nonblocking_commit [drm]] committing 00000000b781d519 nonblocking[758156.239597] [drm:drm_ioctl [drm]] comm="fotron" pid=64244, dev=0xe201, auth=1, [b]DRM_IOCTL_WAIT_VBLANK[/b][758156.239794] vc4-drm gpu: [drm:drm_wait_vblank_ioctl [drm]] waiting on vblank count 45488942, crtc 4[758156.241416] vc4-drm gpu: [drm:drm_calc_timestamping_constants [drm]] crtc 105: hwmode: htotal 2200, vtotal 1125, vdisplay 1080[758156.241853] vc4-drm gpu: [drm:drm_calc_timestamping_constants [drm]] crtc 105: clock 148500 kHz framedur 16666666 linedur 14814[758156.243450] vc4-drm gpu: [drm:drm_crtc_vblank_helper_get_vblank_timestamp_internal [drm]] crtc 4 : v p(0,-45)@ 758155.893811 -> 758155.894478 [e 0 us, 0 rep][758156.243748] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] crtc 4: Calculating number of vblanks. diff_ns = 16686883, framedur_ns = 16666666)[758156.243819] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] updating vblank count on crtc 4: current=45488941, diff=1, hw=0 hw_last=0[758156.244045] vc4-drm gpu: [drm:vc4_atomic_commit_tail [vc4]] Running the core clock at 148500000 Hz[758156.244077] vc4-drm gpu: [drm:drm_wait_vblank_ioctl [drm]] crtc 4 returning 45488942 to client[758156.246129] vc4-drm gpu: [drm:vc4_atomic_commit_tail [vc4]] Core clock actual rate: 200000000 Hz[758156.246366] [drm:drm_atomic_state_default_clear [drm]] Clearing atomic state 00000000b781d519[758156.246815] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 339 (2)[758156.246958] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 346 (2)[758156.247054] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 341 (2)[758156.247137] [drm:__drm_atomic_state_free [drm]] Freeing atomic state 00000000b781d519Code:
[758156.254429] [drm:drm_ioctl [drm]] comm="fotron" pid=64244, dev=0xe201, auth=1, DRM_IOCTL_MODE_ATOMIC[758156.255431] [drm:drm_atomic_state_init [drm]] Allocated atomic state 000000002e055859[758156.255610] [drm:drm_mode_object_get [drm]] OBJ ID: 367 (2)[758156.255706] [drm:drm_atomic_get_plane_state [drm]] Added [PLANE:117:plane-6] 0000000062fb73ed state to 000000002e055859[758156.255809] [drm:drm_mode_object_get [drm]] OBJ ID: 339 (1)[758156.255892] [drm:drm_atomic_get_crtc_state [drm]] Added [CRTC:105:crtc-4] 000000003344754b state to 000000002e055859[758156.256014] vc4-drm gpu: [drm:drm_atomic_set_fb_for_plane [drm]] Set [FB:349] for [PLANE:117:plane-6] state 0000000062fb73ed[758156.256106] [drm:drm_mode_object_get [drm]] OBJ ID: 349 (3)[758156.256186] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 367 (3)[758156.256275] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 349 (4)[758156.256354] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 349 (3)[758156.256491] [drm:drm_mode_object_get [drm]] OBJ ID: 340 (2)[758156.256590] [drm:drm_atomic_get_plane_state [drm]] Added [PLANE:127:plane-7] 00000000e242623b state to 000000002e055859[758156.256726] vc4-drm gpu: [drm:drm_atomic_set_fb_for_plane [drm]] Set [FB:342] for [PLANE:127:plane-7] state 00000000e242623b[758156.256819] [drm:drm_mode_object_get [drm]] OBJ ID: 342 (3)[758156.256907] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 340 (3)[758156.256990] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 342 (4)[758156.257079] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 342 (3)[758156.257197] [drm:drm_atomic_check_only [drm]] checking 000000002e055859[758156.257328] [drm:drm_atomic_get_private_obj_state [drm]] Added new private object 000000003738f977 state 0000000021a4185b to 000000002e055859[758156.257438] vc4-drm gpu: [drm:vc4_atomic_check [vc4]] crtc-4: Trying to find a channel.[758156.259923] vc4-drm gpu: [drm:vc4_atomic_check [vc4]] crtc-4: Already enabled, reusing channel 0.[758156.260378] [drm:drm_atomic_get_private_obj_state [drm]] Added new private object 00000000bd7e6787 state 000000004f74c1d3 to 000000002e055859[758156.260136] vc4-drm gpu: [drm:drm_crtc_vblank_helper_get_vblank_timestamp_internal [drm]] crtc 4 : v p(0,-45)@ 758155.910494 -> 758155.911161 [e 0 us, 0 rep][758156.260676] [drm:drm_atomic_nonblocking_commit [drm]] committing 000000002e055859 nonblocking[758156.260693] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] crtc 4: Calculating number of vblanks. diff_ns = 16683013, framedur_ns = 16666666)[758156.260782] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] updating vblank count on crtc 4: current=45488942, diff=1, hw=0 hw_last=0[758156.260885] vc4-drm gpu: [drm:vblank_disable_fn [drm]] disabling vblank on crtc 4[758156.260975] vc4-drm gpu: [drm:drm_crtc_vblank_helper_get_vblank_timestamp_internal [drm]] crtc 4 : v p(0,18)@ 758155.911370 -> 758155.911103 [e 0 us, 0 rep][758156.261055] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] crtc 4: Calculating number of vblanks. diff_ns = -57498, framedur_ns = 16666666)[758156.261088] [drm:drm_ioctl [drm]] comm="fotron" pid=64244, dev=0xe201, auth=1, DRM_IOCTL_WAIT_VBLANK[758156.261141] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] updating vblank count on crtc 4: current=45488943, diff=0, hw=0 hw_last=0[758156.261246] vc4-drm gpu: [drm:drm_vblank_enable [drm]] enabling vblank on crtc 4, ret: 0[758156.261359] vc4-drm gpu: [drm:drm_crtc_vblank_helper_get_vblank_timestamp_internal [drm]] crtc 4 : v p(0,43)@ 758155.911752 -> 758155.911115 [e 3 us, 0 rep][758156.261465] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] crtc 4: Calculating number of vblanks. diff_ns = -45721, framedur_ns = 16666666)[758156.261540] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] updating vblank count on crtc 4: current=45488943, diff=0, hw=0 hw_last=0[758156.263476] vc4-drm gpu: [drm:drm_wait_vblank_ioctl [drm]] waiting on vblank count 45488944, crtc 4[758156.263053] vc4-drm gpu: [drm:drm_calc_timestamping_constants [drm]] crtc 105: hwmode: htotal 2200, vtotal 1125, vdisplay 1080[758156.263846] vc4-drm gpu: [drm:drm_calc_timestamping_constants [drm]] crtc 105: clock 148500 kHz framedur 16666666 linedur 14814[758156.276801] vc4-drm gpu: [drm:drm_crtc_vblank_helper_get_vblank_timestamp_internal [drm]] crtc 4 : v p(0,-45)@ 758155.927150 -> 758155.927816 [e 0 us, 0 rep][758156.277402] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] crtc 4: Calculating number of vblanks. diff_ns = 16655605, framedur_ns = 16666666)[758156.277482] vc4-drm gpu: [drm:drm_update_vblank_count [drm]] updating vblank count on crtc 4: current=45488943, diff=1, hw=0 hw_last=0[758156.277834] vc4-drm gpu: [drm:vc4_atomic_commit_tail [vc4]] Running the core clock at 148500000 Hz[758156.279231] vc4-drm gpu: [drm:drm_wait_vblank_ioctl [drm]] crtc 4 returning 45488944 to client[758156.280343] vc4-drm gpu: [drm:vc4_atomic_commit_tail [vc4]] Core clock actual rate: 200000000 Hz[758156.280589] [drm:drm_atomic_state_default_clear [drm]] Clearing atomic state 000000002e055859[758156.280985] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 339 (2)[758156.281104] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 367 (2)[758156.281195] [drm:drm_mode_object_put.part.0 [drm]] OBJ ID: 340 (2)[758156.281276] [drm:__drm_atomic_state_free [drm]] Freeing atomic state 000000002e055859[/code]Statistics: Posted by Vraz — Sun Sep 15, 2024 10:03 pm