Despite producing two century breaks, the five-time world champion never led in the match and was eventually knocked out 5-4 as Joyce set up a last-16 meeting with home favourite Ding Junhui.
Joyce made a 137 clearance in the seventh frame to move 4-3 ahead after O'Sullivan shot a break of 132, but the Rocket came back to level at four apiece to set up a decider, which Joyce then took with 62 points to 34.
"It’s right up there with my best wins," said Joyce to worldsnooker.com.
"I beat Judd Trump at the UK Championship when he was world number one a few years ago, so this is probably second to that one.
"I have beaten a few of the top players on TV, it’s on the back tables that I need to start winning and translating these kind of victories into long runs in tournaments.
"I had a good chance to win in the eighth frame tonight and messed it up. The crowd was tough because they wanted Ronnie to win. In the blink of an eye it was 4-4 and that unsettled me.
"Luckily for me Ronnie missed a red in the last frame and I managed to hold myself together to make a good break."
Chinese player Ding progressed in emphatic style, hitting successive centuries of 109 and 127 in the opening two frames on his way to a 5-0 whitewash of compatriot Zhou Yuelong.
O'Sullivan was not the only big-name casualty of the second round as two-time World Championship finalist Ali Carter surrendered a 3-2 advantage to lose 5-3 against world number 28 Michael White.
World champion Mark Selby ensured his place in round three in routine fashion, seeing off Martin O'Donnell 5-1, while last year's winner Judd Trump continued his title defence with a 5-0 victory over Scotland's Eden Sharav.
John Higgins will face fellow former world champion Mark Williams in the third round after ending his hoodoo against Mark Davis.
The veteran Scot, who lost all four of his previous meetings with Davis, triumphed 5-2, while Welshman Williams came from a frame down to beat Michael Holt by the same scoreline.
Shaun Murphy, runner-up at the tournament in 2008, recorded a 5-1 success over Gary Wilson and there were also victories for Rory McLeod, Tian Pengfei, Stuart Bingham, Stephen Maguire, Kyren Wilson, Hossein Vafaei Ayouri, Daniel Wells and Andrew Higginson.