The Luton Town loanee was left dreadfully exposed in the 17th minute when Leon Best was allowed to run through unchallenged to put Yeovil ahead.
And despite pulling off a string of saves, he was beaten again on the hour when Chris Cohen fired in an unstoppable angled shot when a corner was only partially cleared.
Gillingham were playing their third game in less than a week and that opening goal was to prove a killer. It left them looking tired and tame for much of a match which Yeovil dominated.
Cohen hit a long ball down the middle and for some inexplicable reason the Gills defence stood still while Best - in his second match on loan from Southampton - raced clear to hit a low drive past the helpless Brill.
Gillingham did their best to get back into the game but their only real chance of the first half came just before the interval. Gary Mulligan got in a decent shot but home goalkeeper Steve Mildenhall saved well.
Early in the second period Ben Chorley cleared off the line from Arron Davies as Yeovil looked for a second, while at the other end Clint Easton set up Mulligan whose close-range effort rebounded off Mildenhall's feet.
The second Yeovil goal on the hour virtually killed the game. Davies had a shot well saved by Brill at the expense of a corner and from the subsequent flag-kick Yeovil skipper Terry Skiverton had a header cleared off the line.
Deep in stoppage time Gillingham almost got on the scoresheet when Yeovil conceded a free-kick 19 yards out in a central position.
Easton, the most impressive Gillingham player on the day, hit it firm to the goalkeeper's left but it rebounded to safety off the inside of the post.