Erling Haaland headed the opening goal after missing a first-half penalty
Jayden Bogle thought he had earned the Blades a point with a drilled equaliser after 85 minutes, but Spain midfielder Rodri blasted into the top left corner three minutes later to give City the win.
Erling Haaland earlier missed a first-half penalty before heading the opener from Jack Grealish's cross just after the hour mark.
The win was Pep Guardiola's 200th Premier League win as City manager in just his 269th game, the quickest any boss has reached 200 wins in English top-flight history.
Although he was absent from the touchline, recovering from back surgery, Guardiola has been pulling the strings from afar in recent days.
This result also marked the first time City have won the first three games of a campaign since the Spaniard's first season as manager in 2016-17.
City, with assistant boss Juanma Lillo giving orders, deserved to take all three points after dominating possession and creating the better chances.
However, they very nearly paid for unusual profligacy in front of goal.
They had 79% of possession and 30 efforts at goal but were frustrated for much of the game by a stubborn United defence, with captain John Egan and goalkeeper Wes Foderingham impressing.
Haaland spurned a chance to put City ahead when his first-half penalty hit the inside of the right post and bounced to safety, though he made no mistake later when converting Grealish's inviting cross from the left.
Bramall Lane erupted when United capitalised on a mistake by former Blades defender Kyle Walker to equalise through Bogle, but City responded almost immediately.
Walker atoned for his error by breaking into space down the right and putting in a dangerous cross that hit substitute Phil Foden before landing in the path of Rodri, who fired in a venomous shot.
Foden had been dropped despite his fine performance in the win over Newcastle, but his late introduction immediately after Bogle's leveller gave City much-needed impetus.
This is just the fourth time Sheffield United have lost their first three league games in a season, but they can take heart from a defiant defensive display that very nearly earned them a point.
Even after falling behind they did not fold and began to create openings, with Gus Hamer's shot well blocked by Ruben Dias and substitute Oli McBurnie heading just wide from the corner.
After Rodri put City back in front, Yasser Larouci lashed a late effort over from a tricky angle as the home side threatened a second equaliser, with Anel Ahmedhodzic squandering a close-range chance.
McBurnie and fellow substitute Bogle were making their first appearances of the season as United's injury list begins to ease, though Ben Osborn was forced off with what looked like a groin strain in the first half.
There was a pre-match boost for Paul Heckingbottom's side, with England Under-21 striker Cameron Archer paraded on the pitch after completing a reported £18m move from Aston Villa.
-- Courtesy of BBC Sport