From Jaffa, an ancient port city in Israel from which the oranges were exported. The sense "impotent male" is a reference to being "seedless"; see seed. The cricketing sense, of a 'sweet ball' (from the bowler's perspective), is a metaphor further reinforced in Australian English where Jaffa is also a trademark for a popular lolly that is red and round.