The leftovers of planet formation : small body populations of our solar system and exoplanet systems