Differences in looking at the eyes of others are one of the earliest behavioral markers for social difficulties in neurodevelopmental disabilities, including autism. However, it is unknown how early visuo-social experiences relate to the maturation of infant brain networks that process visual social stimuli. We investigated functional connectivity (FC) within the ventral visual object pathway as a contributing neural system. Densely sampled, longitudinal eye-tracking and resting state fMRI (rs-fMRI) data were collected from infant rhesus macaques, an important model of human social development, from birth through 6 months of age. Mean trajectories were fit for both datasets and individual trajectories from subjects with both eye-tracking and rs-fMRI data were used to test for brain-behavior relationships. Exploratory findings showed infants with greater increases in FC between left V1 to V3 visual areas have an earlier increase in eye-looking before 2 months. This relationship was moderated by social status such that infants with low social status had a stronger association between left V1 to V3 connectivity and eye-looking than high status infants. Results indicated that maturation of the visual object pathway may provide an important neural substrate supporting adaptive transitions in social visual attention during infancy.