The short answer is that you can’t order your terms by a custom meta field unless you loop through them and create your own array of objects in that order.
However since you say they each have a number maybe this number is just so you can order them the way you want and nothing else really. If so you can use a plugin like this: https://wordpress.org/plugins/taxonomy-order/
There are also other plugins that do the same thing I just grabbed the first I could find 🙂
