I prefer to take a top-down approach. I want the high level picture of the campaign setting sketched out before I start drawing the first staircase into the dungeon. For the Black City, that meant spending some time thinking about who built the city before starting the maps. I'm pretty well set on the story of the Greys, just working through a bit of the city's history now. Soon, pencils will hit the graph paper.
The upper works are going to be modular. The frozen, windswept streets and plazas of the city are mostly barren, with various open ruins used as lairs or dens. Any ancient technology or artifacts have long since been stripped from these open areas, but conventional treasures are possible. Somewhere in the ruins, tribes of primitive, cold-adapted humanoids like the Voormis of Hyperborea (or the Quaggoths in the Fiend Folio) will prowl the rubble. Maybe yeti?
Many of the structures are still completely sealed, guarded by ancient defense systems, or otherwise unassailable - areas like the Spire of Thaumaturgy or the Institute of the Arcane will be closed until further along in the campaign; the means to open them will be deep in the dungeon.
I like that I'll have upper buildings and areas of the ruins that will be closed at first; I won't have to populate it all up front (and can supplement the stocked open areas with random contents tables for on-the-fly exploration).
Level 1 will be partially mapped and explored by others. The Northmen have been making summer encampments on the island of the Black City for the past couple of years; new adventuring parties just arriving at the island will be able to gather rumors in the large base camp, and speak to some veterans.
Level 2 is where tribes of atavistic cannibals inhabit; the first opportunity to encounter the functional servitors and automatons of the Ancients will be on level 3.