I think I narrowed it further down. I think it is somehow some javascript which does not remove the error message from a field with custom validation.
If I added this in the beginning of my code in the validation function
if (($key = array_search($input, $GLOBALS['acf_validation_errors'])) !== false) {
unset($GLOBALS['acf_validation_errors'][$key]);
}
The text in the message dissapears, but the red error bar still stays