We narrowed it down to Oembed now, if we use the embed code Vimeo provides and use a text field it works. But we also discovered that the same problems occur in Gutenberg if you add a video so it may not be am ACF Problem at all but a WordPress Core Problem.
