Yes, you will need to add a meta_query to do this.
There was a new feature added in WP4.2 that would allow you to do this.
And you’ll need to query by the field you are ordering by as well. See the city_clause example.
https://make.wordpress.org/core/2015/03/30/query-improvements-in-wp-4-2-orderby-and-meta_query/
