Former Treasury Secretary Timothy Geithner knew his job "would be terrible"

Six years after the U.S. financial meltdown, former Treasury Secretary Timothy Geithner finally sees the economy getting better and healing from the scars of the financial crisis.

"I think there's room for optimism," he told CBS News' Bob Schieffer on "Face the Nation."

Geithner reflects on the turmoil and his time as Treasury Secretary in a new book, "Stress Test." Coming into the job, he said, he knew "it would be terrible" and tried to talk the president out of asking him to take on the job.

"I had been already deeply involved in the design of the rescue, the deeply unpopular rescue. And I thought he would be tarred by that and hard for him to separate himself from that. So, I tried to talk him out of it. But I knew it would be hard," Geithner said.

Part of the reason for the political backlash against legislation that came out of the financial crisis, such as the Troubled Asset Relief Program (TARP) stemmed from the fact that actions necessary to stem the damage from the economic collapse seem deeply unfair to those they are designed to protect.

"It looks like you're giving aid to the arsonist," Geithner noted. "Inherent in successful economic rescues are things that are deeply unpopular. And, of course, I knew that, felt that was unavoidable, lamentable."

Geithner struggled with his own difficulty with public speaking, which he believes was "very damaging" for his role. Despite the fact that he was hired to fix the crisis, not run the communications, he believes perception is very important in governing.

His overall experience in the cabinet was a positive one, however, and he praised President Obama for his willingness to make unpopular decisions that he said put policy ahead of politics.

"It would have been easy for him to sit back and say, 'I'm going to let it burn itself out. It's something I inherited. I'm not responsible for it.' And that would have been devastating for the country," Geithner said. "So, my experience was he was excellent in crisis, good at making decisions, very tough on the rest of us, very tough on all of us to make sure he was getting all the options. And I never felt I was in a position where they put a political constraint on what we were doing, and definitely never tried to make us more optimistic than we should have been."

Asked by Schieffer to respond to lawmakers like Sen. Elizabeth Warren, D-Mass., who say the system is rigged in favor of banks and the wealthy, Geithner acknowledged that the country still has challenges to overcome.

"How well you do in life today, how good your education is, how good your health care is, depends too much on the color of your skin and how rich your parent is. And that's something that should worry all Americans," he said. "We need to find a way as a country to rediscover what has been the great strength of this political system, which is the ability to find pragmatic consensus, room for compromise, on things that can make people's lives better. I think we'll rediscover that capacity. But we've seem to have lost it for the moment."

Geithner also weighed in on the scandals plaguing the Veterans Administration (VA), including hospitals that tried to cover up the long wait times for veterans. In a separate interview on "Face the Nation" Sunday, White House Chief of Staff Denis McDonough said the president is "madder than hell" about the scandal but will stick by VA Secretary Eric Shinseki to diagnose and correct the problems.

" Well, I think the men and women who have served our country, we owe them everything. The American people owe them everything and our government owes them everything. And it should be a relentless effort to try to get them better care and better service," Geithner said. "I believe this president is going to put his best people on it...the only advice you can give them is make sure you are putting as much onto that problem as you can. Whatever you think it needs, you want to do two or three times as much as people think it needs."

Obama has sent Deputy Chief of Staff Rob Nabors, a trusted advisor, to work with Shinseki on the VA's review of patient access to care.