Bobby Brown has admitted to striking his late ex-wife Whitney Houston once, but denies being “a violent man.”

In a 20/20 interview with Robin Roberts, Brown discussed an incident from his forthcoming memoir, Every Little Step, to be released on June 13.

He stated that he hit Houston while he was “attempting to stay sober.”

Brown insisted that accusations of him being a “woman beater” or consistently “violent toward her” are “simply untrue.”

“I have never been a violent man towards a woman ever,” Brown said.

Brown, 47, was married to Houston, who passed away in 2012 at age 48, for 15 years.

In 2003, Brown faced misdemeanor battery charges related to alleged domestic violence. However, Houston previously claimed in a Redbook interview, “I do the hitting.”

Their tumultuous relationship, which ended in 2006 when Houston filed for divorce, was marked by a pattern of drug abuse.

Brown and Houston would often seclude themselves in one part of their home, using drugs while nannies looked after their daughter, Bobbi Kristina Brown, just a few rooms away.

Brown maintained that he always tried to prevent his daughter from witnessing him or his wife in such a state.

“It’s difficult when you’re doing it every day, it’s really tough,” he added.

In his book, as reported by 20/20, Brown listed cocaine, alcohol, and crack-laced joints as his preferred drugs.

“The last years of our marriage were awful,” he recalled. “Both of us trying to get clean or one of us trying to get clean… it was dreadful.”

Concerning Bobbi Kristina, Brown confessed, “We could have done better. We should have done better.”

Now drug-free, Brown dismissed longstanding rumors that he introduced Houston to drug abuse.

“I didn’t initiate her,” Brown said. “I accept my role and I take it hard for my involvement, but we all have our own minds and some of us are stronger than others.”

Brown also spoke about the death of Bobbi Kristina, who passed away in July 2015, six months after being found unresponsive in a bathtub in her Georgia residence.

He accused Nick Gordon of being involved in the deaths of both his ex-wife and their daughter, a claim that Gordon’s attorney has denied in an exclusive statement to PEOPLE.

Brown also disclosed that his daughter was scheduled to move to Los Angeles to live nearer to him just two days before the incident.

“If I could get those two days back… my daughter would be here. All I want is those two days back,” Brown said through tears.

Although he has not visited Houston and Bobbi Kristina’s graves, Brown stated, “They’re with me, they’re here.”

A close relative of Brown told PEOPLE ahead of the interview that, despite his “many serious mistakes,” he “is dedicating the rest of his life to making good choices and guiding his family in the right direction.”

In another segment of the interview, Brown recounted asking Houston’s father for her hand in marriage.

“I definitely asked him before… he wanted to see how much money I had in my bank account,” Brown said.

“I had significantly more money, more cash than she did.”

Brown is now married to Alicia Etheredge, and they are expecting their third child together.