Very interesting. Did you get a chance to try out the WP Debug Bar plugin? That initial html/text load is the server running its queries and grabbing post info to display – it might be worthwhile to do a comparison of the debug bar readings between your slow site and one of the other faster sites you have on that server.
Welcome to the Advanced Custom Fields community forum.
Browse through ideas, snippets of code, questions and answers between fellow ACF users