I did more digging and I think I might have made this work in a different way, by adding an Image subfield for each row and thus creating the relationship in the repeater, instead of the other way around.
In the gallery code, I can the use get_field to get all the rows and for every images search for a row which has a linked image ID matching the current one.
I’d still be curious to know if it could also be done the other way around.