Prior familiarity with a face seems to substantively change the way we encode and recognize later instances of that face. We report five experiments that examine the effects of varying levels of prior familiarity and conceptual knowledge on face recognition memory. All experiments employed a 3-phase procedure, in which faces were familiarized in varying ways and to varying extents prior to study and test. Across experiments, increased prior familiarity led to a simultaneous increase in both correct and false identification rates, either when familiarity was gained through passive exposures or conceptual processing. Discriminability, on the other hand, was enhanced by prior familiarity only when the level of familiarity was high and when it involved conceptual processing (Experiments 1–3). Familiarity engendered by passive exposure affected response bias equivalently to more active orienting tasks, but it reduced discriminability in a standard Old/New recognition test (Experiment 4) and did not lead to an enhancement in discriminability in a lineup identification task (Experiment 5). Familiarity engendered by trait evaluations (Experiments 1–3) or name learning (Experiments 2–5) increased discriminability and yielded a more liberal response bias. These results suggest that the benefits of prior familiarity on discriminability in recognition memory are determined by the presence of prior conceptual knowledge. The implications of this work for eyewitness identification situations in which the suspect is known or familiar to the witness are discussed.