I kept the field as “library:all”, but filtered the media that could be seen like so (fill in your own USERTYPE):
/**
* Limit media access to only their own media for certain users
*/
add_filter( 'ajax_query_attachments_args', 'mr_mp_filter_media' );
function mr_mp_filter_media( $query ) {
if ( current_user_can( 'USERTYPE' ) || current_user_can( 'USERTYPE2' ) ) {
$query['author'] = get_current_user_id();
}
return $query;
}