Neighbourhood socioeconomic status and physical access to amenities and greenness are factors that have been associated with mental, physical and perceived health. However, associations between long-time exposure to these circumstances and changes in perceived health in the middle-age population have remained a relatively underexamined area. This study aimed to examine the association between residential environmental history and changes in perceived health in the Northern Finland Birth Cohort 1966 (N = 5973) population encompassing the two latest data collections at 31 and 46 years of age. Longitudinal time-varying geographical data on the residential environment's economic dependency ratio, population density, distance to local services and presence of green areas were derived from various spatial registers and linked to cohort members' exact residential history. According to a multivariable logistic regression analysis, having a residential history in municipalities with higher-than-average (poor) economic dependency ratios was associated with higher odds of poor perceived health changing to good. Among men, living farther than 2 km away from local services was associated with a higher risk of change from good perceived health to poor, and living farther than 300 m away from green areas was associated with a lower risk of change from good perceived health to poor. The residential environments's urban/rural context may be one factor contributing to the findings. The results point to the importance of considering local residential area characteristics and residence duration in certain areas as potential determinants of health. Finally, having long-term residential history in areas with poor access to services and amenities has the potential to undermine health during one's lifetime.