The rule adds the field to any page where you can add or edit a term. Since you can add a term on the page that lists terms the field group is also shown there. The next version of WP (or maybe in 2 versions, I don’t remember exactly) will be removing that form from the term list page to be more like the post listing page. Once that’s done I’m sure that the field groups won’t appear there either.
