The 40-year-old made a brilliant start, winning the opening two frames, but quickly saw his advantage evaporate as breaks of 74, 56 and 94 from Selby saw the pendulum swing.
However, Carter embarked on a tidy run of his own with knocks of 82, 63 and 68 sending him into the quarter-finals.
‘Stone cold killer’ – Selby wipes out moth
'It's a mess' – Hearn admits Masters won't have fans at Alexandra Palace
Earlier, Joe Perry beat Ding Junhui 6-3 in the opening match of the tournament.
Coming into the event on the back of his UK Championship success in December, Ding was among the favourites for the tournament he won back in 2011.
Ding hit the only century of the match with a 135 clearance in the second frame and three times drew level with Perry to leave the scores level at 3-3.
However, breaks of 93 and 83 brought Perry within a frame of success and the 'Gentleman' went on to take the ninth frame to progress to the second round.
‘Out in style!’ – Carter escapes snooker with lucky pot
First round fixtures and results
- Ding Junhui 3-6 Joe Perry
- Mark Selby 4-6 Ali Carter
- Judd Trump v Shaun Murphy
- John Higgins v Barry Hawkins
- Neil Robertson v Ding Junhui or Stephen Maguire (UK Championship runner-up)
- Mark Allen v David Gilbert
- Kyren Wilson v Jack Lisowski
- Mark Williams v Stuart Bingham
O'Sullivan begins Masters campaign against Ding, Trump to face Gilbert
WST reveals plans for 1,000 fans per session at Masters