Hi,
I’m just finding myself in the same situation. (about 1,5 years later.. )
Luckily for me my needs are in the wp admin backend so performance is not crucial.
The only solution I can think of is to fetch all terms, loop through them and check for the value using get_field. Not pretty and wont work when you have thousands of terms but for a smaller amount and in backend it should be fine.