Some more information. When making the field required in the field settings. Then the message will disappear after the first validation fails, but after you corrected it and it passes validation again, the error message will be cleared. But I don’t want all fields with custom validation to be required.
