It would be hard to make so many characters different and memorable and at the same time all involved in the story, unless your whole story is centered around the harem and the harem members' struggles. One night stands and exes are not harem members, I am only talking about long-term, formal ones. But I don't think you should go into the details too much even for the short-term ones. The Ultimate Past's Rene fucked half of the book but the main ones are Anton, Lorry, Sam, Leon, (Alex and Jack can each count as half?), (one night stands, rapists and Higgins don't count) and I think that's as much as I can handle so I voted for 5s.