Indeed, it’s not possible with get_terms (get_categories) args, cause final query contains “INNER JOIN wp_termmeta” but acf doesn’t use this table
So loop through it with usort()
the shortest variant for now is
usort($terms, function($a. $b) use ($taxonomy) {
return get_field('sort_order', $taxonomy.'_'.$a->term_id) - get_field('sort_order', $taxonomy.'_'.$b->term_id);
});