As far as I know, there is not downside of removing it. It’s a drain on resources and slows down the site. Most plugins add their own custom field metaboxes. The standard WP custom fields display is confusing and as far as I’m concerned is pretty much useless. I’ve been disabling it for as long as it’s been possible to do so and have not run into any issues. I too only use ACF for custom fields.
