Inbreeding depression

Inbreeding depression is the reduced biological fitness in a given population as a result of inbreeding, or breeding of related individuals. Population biological fitness refers to an organism's ability to survive and perpetuate its genetic material. Inbreeding depression is often the result of a population bottleneck. In general, the higher the genetic variation or gene pool within a breeding population, the less likely it is to suffer from inbreeding depression.

Inbreeding depression seems to be present in most groups of organisms, but varies across mating systems. Hermaphroditic species often exhibit lower degrees of inbreeding depression than outcrossing species, as repeated generations of selfing is thought to purge deleterious alleles from populations. For example, the outcrossing nematode (roundworm) Caenorhabditis remanei has been demonstrated to suffer severely from inbreeding depression, unlike its hermaphroditic relative C. elegans, which experiences outbreeding depression.

Inbreeding (i.e., breeding between closely related individuals) results in more recessive, deleterious traits manifesting themselves, because the genomes of pair-mates are more similar. Recessive traits can only occur in an offspring if present in both parents' genomes. The more genetically similar the parents are, the more often recessive traits appear in their offspring. Consequently, the more closely related the breeding pair is, the more homozygous, deleterious genes the offspring may have, resulting in very unfit individuals. For alleles that confer an advantage in the heterozygous and/or homozygous-dominant state, the fitness of the homozygous-recessive state may even be zero (meaning sterile or unviable offspring).

...
Wikipedia