President Trump’s chief strategist Steve Bannon has become increasingly isolated inside the White House following the ascension of John Kelly as chief of staff, sources inside the White House and outside advisers tell Fox News.

Bannon, the flame-throwing former executive chairman of Breitbart News, has long been a target of mainstream Republican ire – and so far has survived even as top Trump lieutenants like Sean Spicer and Reince Priebus have resigned.

But the pressure is building.

One source with direct knowledge of White House staffing told Fox News they expect Bannon will be the next one ousted, pointing to the icy nature of his relationship with Trump.

Trump briefly addressed the speculation during a wide-ranging Q&A with reporters at Trump Tower on Tuesday afternoon, while leaving the door open as to whether Bannon would stay.

“I like Mr. Bannon, he’s a friend of mine,” Trump said, while downplaying his impact in the 2016 campaign. “I like him. He’s a good man. He’s not a racist … but we’ll see what happens with Mr. Bannon.”

He added that the press treats Bannon unfairly.

(AP)

Bannon has been the subject of a series of reports in recent days about his future. Axios reported that Trump has told “close associates” he believes Bannon is behind leaks about White House colleagues.

White House sources also told Fox News they think Bannon is the main source behind many negatives stories about his political rivals at 1600 Pennsylvania Avenue, including Priebus; Trump son-in-law Jared Kushner; Trump daughter Ivanka; and more recently National Security Adviser H.R. McMaster. And some have blamed him for the tumult inside the administration over these last few months, calling him a "leaker."

"President Trump is just waiting for him to resign," one former administration official said.

The New York Times also reported that Trump has sent Bannon into a sort of “internal exile” and has not personally met with him for over a week. The same report said Trump did speak with Bannon over the weekend as he tried to formulate his response to the white supremacist rally in Charlottesville, Va., where a counter-protester was killed by a car ramming into the crowd.

It’s unclear exactly what Bannon advised – but Trump drew intense bipartisan criticism for initially blaming “many sides” for Saturday’s violence. Trump on Monday issued a new statement specifically condemning white supremacists and neo-Nazis.

Critics who dislike Bannon anyway have pointed to the problems with the administration’s response in renewing their calls for Bannon to go.

House Minority leader Nancy Pelosi, D-Calif., said in a statement that Trump’s initial response was a “direct reflection of the fact that his chief strategist, Steve Bannon, is an alt-right white supremacist sympathizer and a shameless enforcer of those un-American beliefs.” She called on Trump to fire Bannon.

Bannon has denied any such charges of racism, and Trump is unlikely to listen to Pelosi.

"I think [Pelosi's statement is the] best thing that could have happened for [Bannon]. It could have saved his job," one White House official told Fox News.

Still, Trump has shown he listens to Kelly, who reportedly pressed for Anthony Scaramucci’s ouster as communications director as soon as he became chief of staff.

Scaramucci was fired after 11 days on the job. The Times reported Bannon “is in trouble” with Kelly as well.

A letter to Trump sent Tuesday from the heads of the Congressional Black Caucus, Congressional Hispanic Caucus, Congressional Progressive Caucus and Congressional Asian Pacific American Caucus also called for the removal of Bannon – as well as advisers Stephen Miller and Sebastian Gorka.

Bannon has not yet responded to a request from Fox News for comment.

For his part, Scaramucci has blasted Bannon in media interviews over the weekend and on Monday night.

This comes as another top Trump official, McMaster, tries to fend off attacks against his character. Axios reported that “outside forces” plan to allege a drinking problem, though it's unclear what evidence, if any, they may have. Sen. John McCain, R-Ariz., and others have jumped to his defense.

McCain accused “alt-right” critics of “impugning the character of a good man and outstanding soldier who has served honorably in uniform and sacrificed more for our country than any of his detractors ever have.”

“Such smear tactics should not be tolerated and deserve an emphatic response,” he said in a statement.

Fox News’ Judson Berger and Kristin Brown contributed to this report.