In 1842, Manhattan (of course) showed settlement down at the tip of the island, and in a few locations northward. In this view, Harlem is the densest settlement north of 59th Street:

Note how West Harlem (Manhattanville, mostly) and East Harlem (then known as just Harlem). were settled, whereas what we call Central Harlem was not yet settled.
How To Build Community
