Hybrid seed inviability is a common reproductive barrier in angiosperms. Recent work suggests that the rapid evolution of hybrid seed inviability may, in part, be due to conflict between maternal and paternal optima for resource allocation to developing offspring (i.e., parental conflict). However, parental conflict requires that paternally derived resource-acquiring alleles impose a maternal cost. I test this requirement using three closely related species in the Mimulus guttatus species complex that exhibit significant hybrid seed inviability and differ in their inferred histories of parental conflict. I show that the presence of hybrid seeds significantly affects conspecific seed size for almost all crosses, such that conspecific seeds are smaller after developing with hybrids sired by fathers with a stronger history of conflict and are larger after developing with hybrids sired by fathers with a weaker history of conflict. This work demonstrates a potential maternal cost of paternally derived alleles and also has implications for species fitness in secondary contact.