The Australian took over at Leinster in 2013 after four seasons at England's Leicester Tigers, but struggled to match the success of predecessor Joe Schmidt who led the team to two European titles before taking charge of Ireland.
Leinster won the PRO12 in O'Connor's first season in charge and reached the semi-finals of this year's inaugural Champions Cup but one domestic league since February saw them miss out on a semi-final spot last month.
It was the first time they had finished outside the top four in their league in a decade and they secured their slot in next season's Champions Cup only in the last game of the season.
"I am disappointed and frustrated with our form in the PRO12 this season. We set high standards for ourselves and for many reasons, we did not achieve those highs throughout the league this campaign," O'Connor said in a statement.