I think I have fixed it with a workaround by using save_post and storing the value through update_post_meta.
I had to make an exception so this doesn’t run when v5 is active (because it works there already), so when the v4 fields are loaded I store an option value which is checked in the save_post. If it’s v4 the function runs, otherwise not.
While it does appear to be working now as I want it, I’d still like to have it working the ‘right way’. Maybe @hube2 has some insight ?