Battling Wrexham, who had won only once in their last seven league matches, lasted until the 71st minute when Ryan Valentine handled under no pressure.
Referee Paul Taylor waved away Wrexham's protests and Roberts sent brave goalkeeper Mike Ingham the wrong way with the spot kick.
Carey, who took over from sacked Denis Smith last week, gave young striker Marc Williams a rare chance and left out influential midfield player Darren Ferguson.
Carey was rewarded for his enterprise with a goal in the 8th minute when Simon Spender centred from the right and young Williams rose to score with a header from 15 yards.
But Swindon stormed back to equalise just a minute later when Roberts set up a shooting chance for French winger Sofiane Zaaboub who scored with a left foot shot, his first for the club since signing a contract for the club on Thursday.
Ingham dived to his left to keep out a free kick from Jack Smith late in the first half and Lee Peacock almost scored at full stretch after Ricky Shakes had set him up.
Ingham pushed away a six yard header from Peacock in the 48th minute and then deflected a goal-bound shot from Zaaboub with his feet.
Shakes twice hit a post with inswinging corners from the right as Swindon began to take control.
After Roberts' penalty, Swindon went close to adding a third but Shaun Pejic cleared a shot from the impressive Shakes on the line.
Wrexham offered little threat as an attacking force in the second half although Carey used two of his substitutes with Josh Johnson and Gareth Evans getting their chance to turn the match Wrexham's way.
Referee Taylor booked Paul Mitchell, the former Swindon player on loan to Wrexham from MK Dons for a first half foul.
Taylor also showed yellow cards to Spender and to Lee Roche and Swindon's Curtis Weston for a 55th minute flare up.