I first read it in volarenovels(I think). All the other people was transferred to another world to make them adapt to mana, he was the only one left because he has superior stealth skill. And it took the other people 2000(?) years to get back, but by the time they got back he was OP.
you just reminded me of some manga about an MC and 3 girls find them self the only one left in the entire world