When using the taxonomy field for a custom block (in combination with a custom taxonomy), if you add a term to that field and save it – then it saves no problem. If you remove it again, it doesn’t remove it.
In addition the ‘save terms’ and ‘load terms’ options do not work at all. As such the field works in isolation to the pages actual taxonomy.
These are issues with guberbuggutenberg. You should submit these issues to the developers. As far as I know there is no fix at this time.