Is ‘contact_info’ a field for Category terms or Brand terms? Based on the field that works, I’m guessing the latter, in which case you’ll need to change the 2nd get_field argument:
$custom_field = get_field('contact_info', 'product_brand_' . $term->term_id);