Brain networks are continuously modified throughout development, yet this plasticity can also make functional networks vulnerable to early life stress. Little is currently known about the effect of early life stress on the functional organization of the brain. The current study investigated the association between environmental stressors and network topology using data from the Adolescent Brain Cognitive DevelopmentSM (ABCD®) Study. Hierarchical modeling identified a general factor of environmental stress, representing the common variance across multiple stressors, as well as four subfactors including familial dynamics, interpersonal support, neighborhood SES deprivation, and urbanicity. Functional network topology metrics were obtained using graph theory at rest and during tasks of reward processing, inhibition, and affective working memory. The general factor of environmental stress was associated with less specialization of networks, represented by lower modularity at rest. Local metrics indicated that general environmental stress was also associated with less efficiency in the subcortical-cerebellar and visual networks while showing greater efficiency in the default mode network at rest. Subfactors of environmental stress were associated with differences in specialization and efficiency in select networks. The current study illustrates that a wide range of stressors in a child’s environment are associated with differences in brain network topology.