@hube2 thanks this seems like a solution.
I thought maybe ACF had some hooks to check if the upload is going through a file field and than you could set the max upload to the ACF max upload size on the file and else set a default, something like this https://gist.github.com/mvaneijgen/4db072899ae54ef7469519b50f5ddacc but maybe that is just wishful thinking.
Thanks again.