Amyloid beta (Aβ) follows a sigmoidal time function with varying accumulation rates. We studied how the position on this function, reflected by different Aβ accumulation phases, influences APOE ɛ4’s association with Aβ and cognitive decline in 503 participants without dementia using Aβ-PET imaging over 5.3-years. First, Aβ load and accumulation were analyzed irrespective of phases using linear mixed regression. Generally, ɛ4 carriers displayed a higher Aβ load. Moreover, Aβ normal (Aβ-) ɛ4 carriers demonstrated higher accumulation. Next, we categorized accumulation phases as “decrease”, “stable”, or “increase” based on trajectory shapes. After excluding the Aβ-/decrease participants from the initial regression, the difference in accumulation attributable to genotype among Aβ- individuals was no longer significant. Further analysis revealed that in increase phases, Aβ accumulation was higher among noncarriers, indicating a genotype-related timeline shift. Finally, cognitive decline was analyzed across phases and was already evident in the Aβ-/increase phase. Our results encourage early interventions for ɛ4 carriers and imply that monitoring accumulating Aβ- individuals might help identify those at risk for cognitive decline.