The urban scaling framework views cities as integrated socioeconomic networks of interactions embedded in physical space. A crucial property of cities highlighted by this approach is that cities act to mix populations, a mixing both facilitated and constrained by physical infrastructure. Operationalizing a view of cities as settings for social interactions and population mixing—assembling a set of spatial units of analysis which contain the relevant social aspects of urban settlements—implies choices about the use of existing data, the assignation of data to locations, and the delineation of the boundaries of urban areas, all of which are far from trivial research decisions. Metropolitan areas have become the spatial unit of choice in urban economics and economic geography for investigating urban life as they are seen as encompassing the distinct phenomena of “urbanity” (proximity, density) and social interactions indirectly captured through a unified labor market. However, the population size and areal extent of metropolitan areas, as most often defined, render opaque the distinction between two salient types of urban population: those who work and those who reside within a metropolitan area. These two sets of individuals, among whom of course there is great overlap, putatively engage in different economic and social interactions which are in turn differently embedded in physical space. Availing ourselves of Swedish micro-level data for two distinct spatial units, tätorts (“dense localites”) and local labor markets, we can distinguish which types of populations and which types of spatial agglomerations are responsible for the observed scaling effects on productivity and physical infrastructure. We find that spatially contiguous labor markets are not enough to generate some of the most salient urban scaling phenomena.