Frequency-dependent selection is the term given to an evolutionary process by which the fitness of a phenotype depends on its frequency relative to other phenotypes in a given population.
Frequency-dependent selection is usually the result of interactions between species (predation, parasitism, or competition), or between genotypes within species (usually competitive or symbiotic), and has been especially frequently discussed with relation to anti-predator adaptations. Frequency-dependent selection can lead to polymorphic equilibria, which result from interactions among genotypes within species, in the same way that multi-species equilibria require interactions between species in competition (e.g. where αij parameters in Lotka-Volterra competition equations are non-zero).
The first explicit statement of frequency-dependent selection appears to have been by Edward Bagnall Poulton in 1884, on the way that predators could maintain color polymorphisms in their prey.
Perhaps the best known early modern statement of the principle is Bryan Clarke's 1962 paper on apostatic selection (a synonym of negative frequency-dependent selection). Clarke discussed predator attacks on polymorphic British snails, citing Luuk Tinbergen's classic work on searching images as support that predators such as birds tended to specialize on common forms of palatable species. Clarke later argued that frequency-dependent balancing selection could explain molecular polymorphisms (often in the absence of heterosis) in opposition to the neutral theory of molecular evolution.
Another example is plant self-incompatibility alleles. When two plants share the same incompatibility allele, they are unable to mate. Thus, a plant with a new (and therefore, rare) allele has more success at mating, and its allele spreads quickly through the population.