John Oliver on why his new HBO show "terrifies" him

Despite the years of performing on "The Daily Show with Jon Stewart," John Oliver was still nervous when he first walked onto the set of "Last Week Tonight with John Oliver."

"There was something about seeing a desk with your name on the front of it, which definitely terrifies you. It becomes very real when your name is on the desk. At that point you'd better deliver," Oliver said Wednesday on "CBS This Morning."

With his wit, timing and accent, Oliver became famous as the "senior British correspondent" on "The Daily Show," which he joined in 2006. He also filled in on the anchor desk when Jon Stewart was away in the summer of 2013.

"There was no reason to leave. I was happy there. Everything was my dream job. The only reason to leave was that I kind of felt like I had to," he said.

Stewart helped him through the whole decision, he said, as the offer from HBO was an opportunity he couldn't turn down.

"It's prime real estate, Sunday night on HBO. It's like walking down Park Avenue and seen that someone's built a tent on an empty lot, and, oh -- you've wasted that," he said.

Now with five episodes under his belt, Oliver has more decisions to make.

"I think at the moment, we're trying to see what the scope of the possibility is with the show. We haven't really gotten to a format yet, a clear structure of everything. At the moment we're quite erratic."

Among the trials for the show are deep dive stories.

"So last week we did the FCC and net neutrality, which doesn't scream, 'Oh, that sounds hilarious,' but we tried."

The viewers' response crashed the FCC's website.

"We merely pointed people to [the FCC's] website and told them why they should be angry about it," Oliver said. "And they went, in droves."

Oliver also talked about his early years where he was influenced by the works of his favorite standup comedian, Richard Pryor.

"Comedy is subjective, so it's very hard to be completely sure about something, but most people, comedians, would say that Richard Pryor's the greatest and then other people also try. I knew all of his albums off by heart. I used to listen to his albums to go to sleep, so he was my bedtime story, Richard Pryor. He is without doubt the best. You won't find many comedians push back on that. There's still no one like him."

He also spoke about his wife, a combat medic and an Iraq War veteran.

"She's obviously done more of worth and value in her life than I have, so I tend to just shut up," he said, joking about life at home.