My theory of Ganondorf was wrong.

It turns out that the Gerudo just go to Hyrule Castle Town to look for boyfriends.