Although variations of genes (also known as alleles) can be selected for because they help or hinder an organism, other mutations can have no effect. When the allele itself is not responsible for the change in its frequency in a population, genetic drift is acting on the allele. the frequency of these genes can change drastically over time, especially with the smallest populations.
In the largest populations, the allele frequency of each gene stays relatively stable. This happens because the genes are not affecting fitness, and thus do not have a natural selection pressure against or for the allele. In the smallest populations, the frequency of these genes can fluctuate greatly. Some become fixed within the population, while others disappear. These chance events which lead to changes in frequency are called genetic drift.