Geraldo Rivera said that a “toxic relationship” with a co-host of the Fox News program The Five played a role in his departure from the news network after 23 years last month.

Geraldo Rivera attends Sean Hannity & Friends celebrate the publication of "The Geraldo Show: A Memoir"

At the time of his Fox News exit, the veteran journalist initially suggested that he left The Five at his own will, but has now said that he was “fired” from The Five and decided to quit the network altogether.

“I had a very toxic relationship with another of the cast members,” Rivera, 80, said during an appearance on ABC’s The View Thursday.

Joy Behar and Geraldo Rivera on 'The View'.Geraldo Rivera with Joy Behar on ‘The View.’.ABC

The Five is a five-person panel talk show. Four of the voices are conservative, while the fifth is liberal. Rivera appeared on the show as a rotating liberal personality — along with Jessica Tarlov and Harold Ford Jr. — and hosted with regular conservative voices Jesse Watters, Dana Perino, Jeanine Pirro and Greg Gutfeld.

Rivera notably had a number of on-air clashes with Gutfeld. In April, Rivera told Gutfeld to “stop pointing at me” when they argued over electric vehicles, per AP. Meanwhile, last year Rivera called Gutfeld “an arrogant punk” during a discussion about abortion.

Last month in an Associated Press interview, Rivera cited “growing tension” at The Five as a reason for vacating his co-host seat. “It has been a rocky ride but it has also been an exhilarating adventure that spanned quite a few years,” he told the outlet. “I hope it’s not my last adventure.”

The Five‘The Five’ with then-member Geraldo Rivera.Fox News

When asked by View co-host Joy Behar which of the cast members he had an issue with, Rivera, who joined Fox News in 2001, didn’t directly identify the person and replied with a laugh: “I may get there for you.”

He continued: “So I got a call from two of the female executives and they said, ‘You’re off The Five, but there are plenty of other things you could do.’ I had a year and a half left on my contract and I said, ‘Well, stop it. If you fired me from the No.1 show, then I’m going to quit.’ And that’s basically what happened.

“But in the interim, as in the days that led up to my last day of work, they were so kind to me. They really were so nice, remembering all the things we did together over the 20-odd years. Remember I went there to be a war correspondent after the 9/11 attacks.”

Rivera later said: “I thought that it was very unfair that I was not judged objectively in our disputes, but rather he was always favored…I was suspended, you know, three times my appearances, I had two to three appearances scheduled weekly, then bi-weekly, then monthly, then they kind of disappeared. They were canceled in the last day, right before I was supposed to go on. So I was a really ticked-off guy.”

Behar noted that since Rivera used male pronouns to describe the unnamed person, he can’t be referencing his former Fox News colleague Pirro.

“I love Jeanine,” Rivera replied. “She was a district attorney [in Westchester County, New York] and I put her on TV. But now I’m not mad at anybody now, I’m a free person. I have no idea what I’m gonna do next.”

Geraldo RiveraGeraldo Rivera.Roy Rochlin/Getty

Sunny Hostin, Behar’s colleague on The View, earlier asked Rivera why the news channel couldn’t create a new show for him.

“They could have, I guess. My ideology does not fit Fox. They always kind of squeezed me in. I came as a war correspondent. I did 11 assignments in Iraq, 11 assignments in Afghanistan in the Mideast, Somalia, you know, every hot spot on the globe…the reason I went to Fox was gone. I shouldn’t have left Fox then in 2011. But I didn’t.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer.

On June 30, Rivera made an appearance on Fox and Friends and said he was celebrating his 80th birthday on the Fourth of July. In his final segment on the network, Rivera said at the time that he had “such affection for the people in this building.”

“I really, I feel very emotional and deeply moved,” he added, after a montage of his work on Fox was shown to audiences. “I will always remember this morning. It is something that I didn’t expect, this celebration is beyond the beyond.”