Do you also have save terms set to true? I think it’s probably a bad idea to have both of these settings, I can’t think of any case where you’d want to load values but not save them.
Anyway, what is happening if save terms is not true is that it’s always loading the terms set using the standard WP term metabox for your post type/taxonomy and without save terms being true the values are never updated so nothing is being changed when you save the post.
