Sorry I didn’t get back to this sooner.
I would start by looking and code available for bidirectional relationship fields. The user field works similar to the relationship field but stores data in the usermeta table. I would code a bidirectional user field. I do not know of any existing code for this but it’s possible that someone else has gone down this path already.