I would start by looking for a PHP error, or maybe even a JS error during the AJAX request to get the terms. I just did a test of adding a taxonomy field and then logged in as a contributor. The taxonomy field works and a contributor does not have ‘manage_terms’, so this cannot be causing your issue.
