made it work using this code instead
$cat_lookup_key = 'quote_category_'.$term->term_id;
$data[$i]['image'] = get_field('category_image', $cat_lookup_key);
$data[$i]['color'] = get_field('category_color', $cat_lookup_key);
$data[$i]['priority'] = get_field('category_priority', $cat_lookup_key);
hope it helps someone else.