Hi @nhankla
Have you tried removing the default taxonomy meta box?
I’m not sure if it’s possible to do with the attachment as those are as you say a bit different but there’s a function called remove_meta_box() that you can try: https://codex.wordpress.org/Function_Reference/remove_meta_box
