What do you mean by this
I’ve set my gallery field to NOT use the wordpress media library for upload
Do you mean Library - Uploaded to Post
All images added to a gallery are uploaded to the media library. This just limits the selection to images that have been uploaded while editing a post.
Anyway, no, it is not possible to disable this without altering ACF JavaScriot code
