All four members of Pylon were art students at the University of Georgia in Athens. Guitarist Randall Bewley and bass guitarist Michael Lachowski  began playing music and attempting to form a band in 1978. They practiced in a studio in downtown Athens which Lachowski rented from Curtis Crowe, upstairs from the current location of the Grill on College Avenue. The room was lit by a single 40-watt light bulb, so the band referred to it as the "40 Watt Club." Crowe soon joined the band as a drummer. On February 14, 1979, the three held auditions for a singer. Vanessa Briscoe (now Vanessa Briscoe Hay) was chosen.

Pylon played their first live show at a party on March 9, 1979. Members of The B-52's attended Pylon's third show. Fred Schneider and Kate Pierson of the B-52's were particularly supportive and helped promote Pylon's music in New York City. Soon Pylon opened for Gang of Four in New York. Pylon's debut single, "Cool," appeared on DB Records  in early 1980, and received many positive critical reviews. From 1980 to 1983 Pylon toured parts of the U.S., Canada, and the UK, playing with groups such as the B-52's, Gang of Four, Mission of Burma, Love Tractor, R.E.M., the Talking Heads, and U2. Pylon was the opening act for the first part of U2's first U.S. tour. Crazy/M-Train charted at #61 for Club play singles in 1982 according to Billboard Magazine entry in All Music Guide. At that time the members of Pylon were beginning to feel that playing together was becoming less fun and more like a business. They decided to break up the band in 1983.

With some help from the members of R.E.M., Pylon's reputation as one of the great underground bands of the new wave era was solidified in 1987. When Rolling Stone named R.E.M. "America's Best Band" in November 1987, R.E.M. drummer Bill Berry said, "We're not the best rock 'n' roll band in America." Pylon, he said, was the best.

With the encouragement of R.E.M. and others, Pylon reunited in 1989 to open for the final leg of R.E.M.'s "Green" tour. They recorded another LP in 1990, played several more shows including the South by Southwest Festival  in 1990 and 1991, and broke up again in 1991. Pylon came out of retirement in 2004, and played their first set in more than a decade in August 2004 at the then-fledgling Little King's Shuffle Club. Pylon also played a New Year's Eve show that year at the 40 Watt Club, and headlined the first night of the Athens music festival, AthFest  in June 2005. In 2006, they recorded a so far untelevised segment for "Pancake Mountain". On April 4, 2007, they performed an impromptu show at Little King's to announce the reissue of "Gyrate" on DFA Records, called "Gyrate Plus", which was released on October 16, 2007. They performed several key shows in 2008 including the Part Time Punks Festival in LA, a show taped for Georgia Public Radio in Athens, GA, and at the Revolve Film Festival in Winston-Salem, NC, with Mitch Easter.

On February 25, 2009, guitarist Randall Bewley died, two days after suffering a heart attack while driving his van.

Their second album, Chomp was re-released with extra tracks on October 20, 2009 on DFA Records.