How about unload it with wp_dequeue_style?
The second option is to make it with wp_deregister_style.
if( ! is_admin() ) {
add_action( 'wp_enqueue_scripts', 'my_dequeue_style', 11 );
function my_dequeue_style() {
wp_dequeue_style( 'my-handle' ); // the handle of style of the block
}
}