The board said Mr Altman had not been "consistently candid with his communications", hindering its ability to exercise its responsibilities.
The maker of the ChatGPT bot, OpenAI is behind a wave of excitement about artificial intelligence.
Mr Altman helped launch the firm.
The 38-year-old had also become a spokesman of sorts for the burgeoning industry, appearing before Congress this year to testify about new rules for artificial intelligence.
On social media, Mr Altman wrote that he had loved his time at the company.
"It was transformative for me personally, and hopefully the world a little bit. Most of all I loved working with such talented people," he wrote. "Will have more to say about what's next later."
In a statement the board said it was grateful for Mr Altman's contributions but that members believed new leadership was necessary.
"The board no longer has confidence in his ability to continue leading OpenAI," the company said, citing "a deliberative review process by the board, which concluded that he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities."
The announcement of his departure sent shockwaves through the sector.
On social media, former Google boss Eric Schmidt called Mr Altman "a hero of mine" and said that he had "changed our collective world forever".
"I can't wait to see what he does next. I, and billions of people, will benefit from his future work- it's going to be simply incredible," he wrote.
OpenAI president and co-founder Greg Brockman announced he had quit his role at the company following Mr Altman's ousting.
In a statement posted X, formerly known as Twitter, Mr Brockman said: "I'm super proud of what we've all built together since starting in my apartment eight years ago.
"We've been through tough and great times together, accomplishing so much despite all the reasons it should have been impossible. But based on today's news, I quit."
He said he would continue to "believe in the mission of creating safe AGI that benefits all of humanity".
OpenAI started in 2015 as a non-profit.
It restructured in 2019 and is now backed by Microsoft, which has invested billions.
Just weeks ago, OpenAI was reportedly in talks to sell shares in the company to investors at a price that would value it at more than $80bn (£64bn).
The company said its board members - who include an OpenAI chief scientist, the head of popular question and answer app Quora, and an AI researcher affiliated with Georgetown University - did not have shares in the firm and that their fundamental governance responsibility was to "advance OpenAI's mission and preserve the principles of its Charter".
The company said chief technology officer, Mira Murati, would take over as interim chief, effective immediately, while the board searches for a permanent replacement.
ChatGPT is known for its ability to respond to prompts from users with human-like text, images and videos.
Hundreds of millions of people have tried it out, and many are now regularly using it to help them do their jobs and study - to consternation in some cases, like teachers facing essays written by the bot and people worried for their jobs.
The company has also faced legal action from writers who say the bot developed its abilities by harvesting their work, in violation of copyright law.
Billionaire Elon Musk, who with Mr Altman was one of the founding co-chairs of OpenAI, has also criticised it for straying from its non-profit roots.
-- Courtesy of BBC News