It works now. I peeked a bit in the code of the class \acf_form_taxonomy. There are two methods which were added since version 5, add_term and edit_term. There are also two actions defined in which you can hook into: {$taxonomy}_add_form_fields and {$taxonomy}_edit_form.
I am using now the actions to determine which term is currently viewed. I save the current term in the $GLOBAS variable (current hotfix only).
Next, I am using the acf/load_field/name={$field_name} filter which checks the previously defined term in the $GLOBALS variable on which level we are and to determine if we display the current field or not.
