Born in Long Beach, California, Duffy attended Lakewood High School in Lakewood, California, where he acquired the nickname "Duffman." At Lakewood High School, Duffy was a two-year letterwinner and led the team to a 23–12 record and a league championship in 2008. He was a two-time all-state selection and a two-time all-league pick also. As a junior, Duffy achieved a batting average of .409 and 28 runs batted in. He hit .380 his senior year with 34 runs batted in and 3 home runs.

Duffy played college baseball at Long Beach State University from 2010 to 2012. During his 2010 season at Long Beach State, Duffy played in 30 games, starting 20 of them; averaged .244, had 86 at-bats, 8 runs, 21 hits, and 7 RBI. In the 2011 season, Duffy played in 55 games, starting 54 of them; averaged .266, had 214 at-bats, 22 runs, 57 hits, and 31 RBI. In those two seasons at Long Beach State, Duffy played in 85 games, starting 74 of them; averaged .260, had 300 at-bats, 30 runs, 78 hits, and 38 RBI. In 501 college at-bats, Matt Duffy hit .253 and had an on-base percentage of .305. Over those three seasons, Duffy recorded zero home runs, sixteen doubles, and one triple. He was kept on the team because of his defensive abilities.

Between his sophomore and junior seasons, Duffy played in the Cape Cod League with the Orleans Firebirds, where he improved his swing by standing further back in the batter's box and incorporating a leg kick. He still works with Firebirds' hitting coach Benny Craig in the offseason. Craig urged Duffy to read Harvey Dorfman's The Mental Keys to Hitting, which Duffy says he has read at least 10 times, referring back to it whenever he is in a slump.

...
Wikipedia