Yes, if you want to get fields from a category and not the post then you must supply the category ID in the form of "term_{$term_id}" or you can supply the term object as in your first code example.
You can get the terms of the post using get_the_terms()
