This is part of how ACF works. ACF loads field groups for the current site and it does not load field groups for other sites. This means that any field that you want to access when using switch_to_blog() must exist in the field groups on both sites. Values do not need to be present but the field definition must exist.
Welcome to the Advanced Custom Fields community forum.
Browse through ideas, snippets of code, questions and answers between fellow ACF users