This paper examines the geographic factors that are associated with the spread of COVID-19 during the first wave in Sweden. We focus particularly on the role of place-based factors versus factors associated with the spread or diffusion of COVID-19 across places. Sweden is a useful case study to examine the interplay of these factors because it did not impose mandatory lockdowns and because there were essentially no regional differences in the pandemic policies or strategies during the first wave of COVID-19. We examine the role of place-based factors like density, age structures and different socioeconomic factors on the geographic variation of COVID-19 cases and on deaths, across both municipalities and neighborhoods. Our findings show that factors associated with diffusion matter more than place-based factors in the geographic incidence of COVID-19 in Sweden. The most significant factor of all is proximity to places with higher levels of infections. COVID-19 is also higher in places that were hit earliest in the outbreak. Of place-based factors, the geographic variation in COVID-19 is most significantly related to the presence of high-risk nursing homes, and only modestly associated with factors like density, population size, income and other socioeconomic characteristics of places.