Ken A. Thompson1,2, Yaniv Brandvain3, Jenn M. Coughlan4, Kira E. Delmore5, Hannah Justen5, Catherine R. Linnen6, Daniel Ortiz-Barrientos7, Catherine A. Rushworth8, Hilde Schneemann9, Molly Schumer1,10,11 and Rike Stelkens12 1Department of Biology, Stanford University, Stanford, California 94305, USA 2Department of Plant Biology, Carnegie Institution for Science, Stanford, California 94305, USA 3Department of Plant and Microbial Biology, University of Minnesota - Twin Cities, St Paul, Minnesota 55108, USA 4Department of Ecology & Evolutionary Biology, Yale University, New Haven, Connecticut 06511, USA 5Department of Biology, Texas A&M University, College Station, Texas 77843, USA 6Department of Biology, University of Kentucky, Lexington, Kentucky 40506, USA 7School of Biological Sciences, The University of Queensland, Centre of Excellence for Plant Success in Nature and Agriculture, St Lucia, Queensland 4072, Australia 8Department of Biology and Ecology Center, Utah State University, Logan, Utah 84322, USA 9Department of Genetics, University of Cambridge, Cambridge CB2 3EH, United Kingdom 10Centro de Investigaciones Científicas de las Huastecas “Aguazarca,” A.C., Calnali 43240, Mexico 11Hanna H. Gray Fellow, Howard Hughes Medical Institute, Chevy Chase, Maryland 20815, USA 12Division of Population Genetics, Department of Zoology, Stockholm University, 106 91 Stockholm, Sweden Correspondence: kthomp1063gmail.com

Ecologically mediated selection against hybrids, caused by hybrid phenotypes fitting poorly into available niches, is typically viewed as distinct from selection caused by epistatic Dobzhansky–Muller hybrid incompatibilities. Here, we show how selection against transgressive phenotypes in hybrids manifests as incompatibility. After outlining our logic, we summarize current approaches for studying ecology-based selection on hybrids. We then quantitatively review QTL-mapping studies and find traits differing between parent taxa are typically polygenic. Next, we describe how verbal models of selection on hybrids translate to phenotypic and genetic fitness landscapes, highlighting emerging approaches for detecting polygenic incompatibilities. Finally, in a synthesis of published data, we report that trait transgression—and thus possibly extrinsic hybrid incompatibility in hybrids—escalates with the phenotypic divergence between parents. We discuss conceptual implications and conclude that studying the ecological basis of hybrid incompatibility will facilitate new discoveries about mechanisms of speciation.