It was a huge effort from Ulster who were forced to play most of the game with 14 men after losing Tom Court to a red card - their second sending off in three home games after Jared Payne's dismissal against Saracens last month - early in the first half.
Leinster, though, were down to 13 men at one point in that first half after Rob Kearney and Rhys Ruddock were shown yellow by English referee Luke Pearce.
Shane Jennings scored Leinster's only try of the night in the second half while Ian Madigan kicked four penalties and a conversion - though he missed three kicks at goal - while replacement Jimmy Gopperth nailed a penalty.
Man of the match Paddy Jackson scored all Ulster's points, on the night the new Ravenhill was officially opened, with five penalties and a try on an emotional occasion as skipper Johann Muller and Court played their last games at the ground.
Madigan opened the evening's scoring with a fifth minute penalty after Ulster's scrum was demolished on their own put-in and the Leinster out-half then added a second after 11 minutes when Nick Williams was pinged in a tackle.
Shortly after Paddy Jackson put the restart out on the full, Madigan had a third shot at goal from long range but hit the attempt wide but then came Court's red card.
But with their backs to the wall, it was Ulster who scored next after Jackson charged down a Madigan chip to score in the corner with Leinster then being reduced to 14 after Rob Kearney's high tackle on the Ulster out-half as he crossed the line.
Jackson failed to land the difficult conversion from his 23rd minute score, but Ulster, against the odds, were right back in the game and their cause was further helped when Rhys Ruddock was shown yellow three minutes later after Jackson was taken out in the air though the crowd bayed for a red after Payne's dismissal last month for his challenge on Alex Goode.
It was Leinster, now down to 13 men, who found themselves on the back foot and though they survived an attacking five-metre scrum from Ulster, they were pinged just after the half hour at the breakdown and Jackson kicked the home side into the lead at 8-6.
Their lead did not last though as Madigan kicked Leinster back into a single point lead with his 35th minute penalty just as Rob Kearney arrived back to bring the visitors back to 14 men on the park.
Even though Leinster were back to full strength at the end of the half, Ulster still held out and the half ended with the visitors leading 9-8.
Leinster had the first scoring opportunity of the second half but Madigan put another long range effort wide after 44 minutes and then he was wide again two minutes later with a much more straightforward effort.
But Ulster again conceded a kickable penalty but this time Madigan went for the corner and the net result was more heroic defending from Ulster as Dave Kearney was tackled into touch when a try looked certain.
Still, the visitors did manage to score when Madigan landed his fourth penalty on 52 minutes to take their lead to 12-8. The home side though then came back again with Jackson's second penalty narrowing Leinster's lead to a single point again after 56 minutes.
But after Tommy Bowe knocked on the restart, Leinster attacked down the right channel and O'Driscoll drew two defenders to off-load inside to Shane Jennings who scored at the posts, allowing Madigan to convert and put the visitors 19-11 in front.
Typically, though, Ulster responded and, after replacement Darren Cave's clean break, they won a penalty near the sticks and Jackson slotted the kick to take his side back within bonus point territory.
It didn't last though and, after conceding another scrum penalty, Leinster replacement Jimmy Gopperth nailed his first penalty to put his side 22-14 ahead with 11 minutes remaining.
The see-saw nature of the scoring continued and after Leo Cullen was penalised, Jackson did the needful two minutes later to close the gap to five points which then became two with Jackson's fifth successful penalty which followed in the 77th minute.