It was not it, I just tried something and it worked.
I was updating the field with term IDs as integers: 1, 2, 3…
Then I tried updating the field using strings instead: “1”, “2”, “3”…
Now it’s working, that was the difference when updating on the frontend/backend.
Anyway, just another day in the life of a developer.
Thanks for helping me clarify!