Graziano, 62, had been under fire for more than a year over poor conditions at city public housing complexes and allegations that maintenance workers there demanded sex in return for repairs.
Pugh — along with other leading mayoral candidates — pledged during the campaign to remove Graziano.
She also pledged to separate the city and federal functions of the agency known as Baltimore Housing.
"I've been very clear," Pugh said Tuesday. "I thought it was important that the city go in a different direction."
The new mayor said Graziano offered her his resignation and she accepted it. His last day will be Jan. 6. Graziano will receive about $116,000 in severance pay for unused leave, Pugh said.
To residents living in poor conditions in public housing, Pugh said: "We're hearing them and we're going to help them."
Graziano made about $220,000 a year leading Baltimore Housing, which includes both the federal Housing Authority of Baltimore City and the city's Department of Housing and Community Development. The housing authority operates 11,000 public housing units in the city, while the city wing of the agency focuses on enforcing housing code violations and tearing down vacant properties.
In a statement, Graziano called his 16 years in office "an honor" and wished Pugh "all the best."
"I am proud of the significant progress we have made in preserving and creating affordable housing for our residents and building stronger neighborhoods," he said. "I thank all who have contributed to these efforts.
"Clearly, many challenges remain, which I am confident the Housing Authority of Baltimore City and the Department of Housing & Community Development will continue to address."
Pugh said Deputy Housing Commissioner Michael Braverman will serve as interim commissioner while she conducts a national search for a permanent replacement. She said she might seek two — or even three — people to replace Graziano. The new hires would lead the separate housing departments Pugh plans to create.
Pugh's removal of Graziano was applauded by several observers who said it will allow the agency to have a fresh start.
City Councilman John Bullock, the new chairman of the council's housing committee, said he was encouraged by the choice of Braverman as interim commissioner.
"I want to make sure we have a housing commissioner who is being responsive to the concerns of those in public housing, but also responsive to the concerns of the mayor and the City Council," Bullock said. "Having someone like Michael Braverman is a good move. He's very well regarded."
Dayvon Love, co-founder of the activist group Leaders of a Beautiful Struggle, said he was pleased Graziano will soon be gone. He said he was concerned about Graziano's embrace of a federal program to privatize many public housing complexes in Baltimore.
"He presided over a housing policy that seemed to be going in the direction of gentrification," Love said. "I'd like to see a housing commissioner that specifically has in mind the dangers of gentrification and empowers local residents."
Graziano, who was hired by former Mayor Martin O'Malley in 2000, told The Baltimore Sun during this year's mayoral campaign that he understood his days were numbered.
"A new mayor has a right to pick her own housing commissioner," Graziano said during a wide-ranging interview last fall. "I understand that. ... I have a position right now and I've taken an oath of office. And I have duties to perform until the mayor asks me to no longer perform those duties."
Graziano is the longest-serving agency head in Baltimore's government. He has worked for three mayors.
Graziano has been the subject of several public controversies. Only weeks into the job in 2000, he was arrested at a Fells Point bar after a drunken tirade laced with anti-gay slurs. He apologized, and O'Malley sent him to get alcohol treatment.
Later, he drew criticism for refusing to pay millions of dollars in settlements and court judgments in cases alleging that children were poisoned by lead paint in public housing. Graziano has said he was following orders from the federal government and paid the bills as soon as he was allowed.
Six years ago, Stephanie Rawlings-Blake's transition team declared Graziano had a "vision deficit." Rawlings-Blake, citing various complaints, told him to "hit the reset button." He later won her over and she became one of his biggest defenders.
"Under very challenging budget circumstances, Paul Graziano has transformed the housing department into an innovative department that has been recognized internationally for its work," the former mayor said.
But criticism of Graziano intensified during 2015 and 2016. Residents complained about poor conditions in the city's public-housing complexes, and some in West Baltimore went days without heat or water.
At the same time, a lawsuit alleged that some maintenance workers refused to make repairs at apartments unless tenants engaged in sex acts.
Calls for Graziano to be fired came from student protesters at City Hall and became a social media hashtag — #firegraziano.
Graziano has defended his performance in office, arguing that problems at the agency were caused by lack of funding, not of caring. He says he took disciplinary action against the maintenance workers accused of wrongdoing. The housing authority settled the class-action lawsuit for up to $8 million.
"I've dedicated my life to serving the low-income residents of public housing," he told The Sun last year. "I wouldn't be doing it for two-thirds of my life if I didn't care."
Over time, Graziano said federal funding for capital improvements in Baltimore's public housing dropped from $42 million in 1993 to $15 million today. Because about half that money goes to paying down debt, the agency has less than $7 million a year to spend on building improvements, he added.
Carol Ott, director of the advocacy group Housing Policy Watch, was among those calling for Graziano's firing.
On Tuesday, she hailed Pugh's decision.
"I'm thrilled to know that our new mayor has kept her campaign promise," Ott said. "This has been a long time coming."
She said she'd like to see a new housing commissioner get tougher on "negligent property owners" who violate the city's code and those who don't get rid of chipping lead paint on their properties. She said she sees hope in the choice of Braverman as interim commissioner.
"I have found Michael to be competent, innovative and forward-thinking," Ott said. "I think Michael has an opportunity to bring in a new fresh voice to an agency that has grown stagnant."
Graziano was the only high-profile staff member that Pugh pledged to remove during the campaign. She has kept most other agency heads, including Baltimore Police Commissioner Kevin Davis and Health Commissioner Leana Wen.
In her first press briefing after becoming mayor this month, Pugh said some of her first goals were to reorganize Baltimore's housing and economic development agencies and remove Graziano.
"He will not be the housing commissioner, period," she told reporters.