Increasing the priority from 20 to 99 solved the problem.
Asking the question in public is always the fastest way to find the answer yourself. 😉 This needs some refinement, but setting the $field[‘default_value’] in a acf/load_field hooked function that runs after most other things is working:
add_filter('acf/load_field/type=taxonomy', function( $field ) {
$screen = get_current_screen();
if ( $screen->base = 'term' && $screen->parent_base == 'edit' ) {
$term_id = (int) $_REQUEST['tag_ID'];
$field['default_value'] = get_term_meta( $term_id, $field['name'] );
}
return $field;
}, 99, 1);
