Ahh it took me a while to find your upload. sorry about that. Well the reason your peasants are standing around is because you made your mines, and mills with the red and blue colors selected.
All buildings, mines and mills included need to be built in the neutral or brown color. They can then be assigned a color in the settlementinfo. menu.
Delete the buildings, (this will need to be done in the correct color, as red mill, mine selected while the red color is selected, and blue while blue is selected etc.)
And then replace them with the neutral color buildings and go from there.
Looks like a nice start though.
Anything else just let me know.
Daddio