Follow-up: the remove_filter/add_filter method ended up preventing the filter from running at all (even after re-adding), so get_post_meta() was the solution in my case.
Thank you for saving my sanity! Whew.
I used the remove and add filter method above and it avoids the infinite looping.