A twister roller coaster is the generic name given to any roller coaster layout which tends to twist or interweave its track within itself several times. It is essentially the opposite of an Out and Back roller coaster, which is often a much more simplistic layout. Twister roller coasters often have the illusion of having small or tight clearances due to the track usually travelling through several support structures. This is known as a head chopper effect.
Twister roller coasters were unheard of before the 1920s. John Miller is credited with inventing upstop wheels and secure lapbar restraints, both which led roller coaster designers to create wilder and twistier layouts.
A good example of the difference between an out and back design and twister design is layouts of Apollo's Chariot and Raging Bull, two Bolliger & Mabillard designed hypercoaster roller coasters that debuted in 1999. Apollo's Chariot uses a traditional out and back layout while Raging Bull is a twister.