A native of Lenoir, North Carolina, Walker was signed by the Chicago Cubs as an amateur free agent in 1944. He made his Major League Baseball debut with the Cubs on April 20, 1948, and appeared in his final game on June 15, 1958. As a Brookyn Dodger, he was behind the plate when Bobby Thomson hit the "Shot Heard 'Round the World" on October 3, 1951, substituting for Hall of Famer Roy Campanella, who was nursing an injury.

After retiring as an active player in June 1958, Walker served out the season as a Los Angeles Dodgers coach. He then was a minor league manager in the Dodgers' and New York Yankees' farm systems from 1959 through 1964. After 1965 he was a pitching coach for the Washington Senators, New York Mets, and Atlanta Braves, working closely with managers Gil Hodges, Yogi Berra and Joe Torre, among others. Walker was the Mets pitching coach for 14 seasons from 1968 through 1981 including the 1969 World Champion "Amazin'" Mets. As the Mets' pitching coach, he supervised Hall of Famers Tom Seaver and Nolan Ryan. Walker scouted for the Braves and St. Louis Cardinals after his coaching career ended.