Thanks for the tip. This may be viable. The problem is that the list of required term IDs is dynamic. Basically User A has access to Term ID 1, 2 and 3 and User B has access to Term ID 2, 3 and 4.
I can filter the taxonomy for user A that will show him IDs 1, 2, 3 and NOT 4. But as soon as I do this and User B select Term ID 4 and afterwards User A select Term ID 1, the reference to term ID 4 will disappear. I would probably need to first check which Term IDs are already associated with the field then remove all ids that user has access to and merge submitted data with trimmed data using acf-update_value filter as suggested. This seems bit overkill but I’ll probably have to go this route:)
