When Prince Harry was in his twenties, he began to fantasize about having a simple life with a wife and child, and wanted to leave the royal family.

On the Armchair Expert podcast, hosted by Dax Shepard and Monica Padman, the Duke of Sussex discussed his desire to quit royal service — as well as the damaging effects fame had on his late mother, Princess Diana.

He explained to Dax: “It’s the job right? Grin and bear it, get on with it.

“In my early 20s, it was a case of, I don’t want this job. I don’t want to be here, I don’t want to be doing this, look what it did to my mum.

“How am I ever going to settle down, have a wife and a family when I know, that it’s going to happen again?”

Harry continued, “Because I know, I’ve seen behind the curtain. I’ve seen the business model, I know how the operation runs and how it works, and I don’t want to be a part of this.”

The hosts were interested in learning more about the 36-year-old’s upbringing in such an unusual setting as the British royal family. Padman asked if he’d feel like he’d been “in a cage” for much of his childhood, and Harry said that he’d struggled with his duty as a royal and his urge to live a more normal life from an early age.

He also spoke about how Meghan encouraged him to get support. “It was a conversation that I had with my now wife. And she saw it straight away, she could tell that I was hurting and that some of the stuff that was out of my control was making me really angry,” he said. “It would make my blood boil.”

“Suddenly there was like a bubble that burst,” Prince Harry, 36, said after beginning therapy.

“I plucked my head out of the sand, gave it a good shake off,” he said after a struggle with his mental health that led him to get assistance. “And I was like, ‘Okay, you’re in this position of privilege, stop complaining or stop thinking as though you want something different, make this different, because you can’t get out.”

Harry went on to say, “So how are you going to do this differently? How are you going to make your mom proud? How are you going to use this platform to really affect change and be able to give people that confidence to be able to change their own lives?”

Princess Diana, Harry’s mother, died in a car accident in Paris in 1997 at the age of 36 while attempting to escape aggressive paparazzi. The British royal expressed his concern that something similar would occur again in his household.

Although Harry remembers being furious, he claims that his rage did not express itself in any sort of outburst. Instead, he simply internalized a lot of the things that bothered him.

“I’ve never screamed, I’ve never shouted. For me the best way to let out aggression is boxing,” he explained. “But for me, prior to meeting Meghan, it was very much a case, certainly connected to the media, that anger and frustration of, ‘this is so unjust.’”

Harry clarified that he had to learn to cope with something he couldn’t do, which he also struggles with when it comes to paparazzi and media scrutiny of his family.

“The three major times that I felt helpless, one as a kid in the back of a car while my mom was being chased by paparazzi, two was in Afghanistan in an Apache helicopter and the third one was with my wife,” Harry revealed. “Those are the moments in my life where, yeah, feeling helpless hurts. It really hurts. And that’s when you think to yourself, ‘s—-, I have the privilege, I have the platform, I have the influence and I even I can’t fix this. I can’t change this. And when you start getting in your head about it, that’s when it starts taking a toll.”

Prince Harry said he went on to found the Invictus Games, a paralympic-style sport for disabled servicemen and vets.

“Once I started to see the progress and the impact, I saw like, ‘Wow, healing other people heals me,’ and I think that’s where the compassion comes in for all of us, which is, once you’ve suffered, you don’t want anybody else to suffer,” he said. “As a human, that’s what we’re supposed to do, right, compassion, there’s an element of selfishness there, and I don’t think there’s anything wrong with that. I think if you helping other people gets you the fix that you want or that you need, happy days!”

Other topics explored in the 90-minute interview included Harry’s privilege as a British royal growing up and how it influenced his experience in a special way that wasn’t actually negative.

“Believe me, all of us have seen suffering. I, luckily, because it’s been part of my own growth, have spent many many years traveling around the world seeing other people suffer and being able to have that empathy for them, the ability to put myself in their shoes,” Harry explained. “That was the education that I had. So the weird thing is that, yeah, I was born into this privilege, but the privilege also gave me the most unbelievable… front row seat and education. My education is not in school. My education is about meeting people across the commonwealth.”

He said, “Everywhere I go, I ask questions, everywhere I go I try to listen. I don’t want to come in and say, ‘This is what I think, these are what my solutions are.’ I already know they’re already looking at me like ‘You’re a prince, you live in a palace, where’s your crown and where’s your cape?'”

Prince Harry also spoke of three occasions in his life that he felt “completely helpless:”: in a car with his mother Princess Diana while being pursued by reporters, in a helicopter when working in Afghanistan, and when his wife Meghan Markle was suffering.

“Those are the moments in my life where feeling helpless hurts. It really hurts,” he said. “And that’s when you think to yourself, ‘S—, I’ve got the privilege, I’ve got the platform, I’ve got the influence and even I can’t fix this. I can’t change this. And when you start getting in your head about it, that’s when it starts sort of taking a toll.”