Valhalla is a ruin. It needs to be built from the ground up. This city is already more or less in working order. The people are also a valuable resource that he doesn't want to abandon. Smiths, food, new army recruits and other such things will all be coming from the city's population.