I’m facing the same issue.
The nearest to native solution that I have found is to add
add_filter( ‘should_load_separate_core_block_assets’, ‘__return_true’ );
This will include only css files of the blocks that are used in the page.
But the css files are injected in the end of the body and I’m not sure if the user internet is slow how will effect the layout shift of the page
