That is kind of a driving part of the story, at least the little bits I read. Realistically with everything that happened at the beginning as soon as he got a guild card so he could leave and enter towns he should have left that place to go somewhere else and never came back. I have only read a little ways in, but to much seem to be the world being jerks and him having problems as a result of not paying attention, or reacting badly.
Hard to remember the details my brain treats them like an infection and tries to expel it but bottom line Spoiler 1. he never has sex it never happens 2. he makes his barony a going concern adds more women to the harem 3. he gets promoted to higher nobility can't remember what title count or maybe marquis 4 he fights and defeats the demon king 5. just when it looks like he might actually have sex he falls down a well and is isekaied to another world
The last LN vol already diverged from the WN. So expect different ending to the trollfest the author made