Oli struck a minute into the second half, to bring his side level before grabbing the winner with a fine individual goal, just before the hour mark.
Stimson included three of his seven new signings in his line-up, but they were soon exposed by Danny Wilson's more settled 11.
Richie Humphries' dangerous long throw was allowed to bounce in the area and James Brown reacted fastest to turn the ball home and give his side a 25th-minute lead.
Having stuck the knife in, Hartlepool came close to twisting it moments later when Richie Barker met Ali Gibb's measured cross with a firm header, only to produce a stunning save from keeper Simon Royce.
Nervous debutant Adam Bygrave then delivered a heart-stopping moment on 31 minutes, sending a back-pass towards a vacant goal, before Royce scampered back to prevent disaster.
Stimson's men left the field to a chorus of boos at half-time and it appeared to have the desired effect as they drew level within a minute of the restart.
Leroy Griffiths' high, looping ball forward appeared to get caught in the wind and the Hartlepool defence hesitated, allowing Oli to sneak in and poke home past bemused keeper Jan Budtz.
After Gibb saw a close-range effort brilliantly blocked by Craig Armstrong, Oli notched his second of the afternoon, shuffling his way past two defenders before calmly slotting home at the near post.
With eight minutes remaining, hero Oil was replaced by 15-year-old Luke Freeman, who became the youngest player to represent Gillingham in a League game.
It appeared his day might be ruined when Brown let fly from 20 yards in stoppage time but the shot flashed over, allowing the Gills to celebrate a much-needed win.