The resilient Senator Scott Brown

Scott Brown has defied the odds all his life. He's the Republican who came out of nowhere last year and won the Senate seat Ted Kennedy held in Massachusetts for 47 years.

He's an anti-tax, fiscal conservative whose election in blue-state Massachusetts energized and helped launched the Tea Party movement. As the 41st Republican senator, everyone thought he'd be the "no" vote that blocked nearly everything President Obama and the Democrats proposed. But he has turned out to be unpredictably independent and beholden to no one.

Now Senator Brown has written a memoir called Against All Odds about the hardships of his early years. As the child of seven broken marriages, his childhood was marked by neglect, violence and trauma.

Segment: Scott Brown
Extra: Scott Brown on sex abuse
Extra: Rescuing Mom
Extra: Reelection worries?

"Do you think that you have innate resilience?" correspondent Lesley Stahl asked.

"I think as a result of what I've gone through it makes me tougher, absolutely," Sen. Brown replied.

"Things roll off your back more easily because of your life?" Stahl asked.

"Yup," Brown said. "When I'm getting the crap beat out of me outside in the political spectrum, I'm like: 'Pssss, this is nothin'. Bring it. Let's go. Next!'"

Brown has needed that bring-it-on toughness in Washington, where he is showing everyone he is his own man, with a voting record that has his fellow Republicans in a dither.

"I look at each and every bill on its merit regardless of party affiliation," Brown told Stahl. "Party plays no bearing whatsoever at all, in any of the votes that I take, period. So to say, 'Well, the party wanted this,' I don't care what the party did.'"

You don't hear that kind of talk every day. While Brown votes with the Republicans against tax increases, he has broken ranks on other major issues, like a jobs measure and ending the military's "Don't Ask, Don't Tell" policy.

"Whichever way you vote, you're very likely to tick off the Tea Party or Massachusetts. You're right caught in this horrible vise," Stahl pointed out.

"Well, I don't know if I'm ticking off the people in Massachusetts. I think I'm doing exactly what they wanted. They sent somebody down there who they wanted to be an independent voter and thinker. And if I'm upsetting people on the left and on the right, then I must be doing my job," Brown said.

"I have a history of working across party lines and I'll try to do the same thing here," he added.

When he worked across party lines and became a decisive vote in passing the bill to expand government oversight of Wall Street, the Tea Partiers called him "Benedict Brown" and said they would support another Republican when he runs again.

"He's in the tradition that we should have in our politics where you stand for things but you can differ reasonably," Democratic Congressman Barney Frank told Stahl.

"You compromise, the dreaded word," Stahl remarked.

"Absolutely," Frank said.

Frank negotiated with Brown to win his vote on the financial reform bill.

"The Wall Street Journal op-ed page went crazy attacking him. The Tea Partiers said he was a betrayer," Stahl said. "Everything came down on his head. He voted for this bill in the end."

"Frankly, my guess is that, in Massachusetts, that's the kind of attack that politicians welcome. Because you had a fairly shrill group attacking you, on something that you did that was very popular and you don't mind getting that attention," Frank said.

Produced by Karen SughrueIt's true that Brown's independence is paying off back home: polls rank him the most popular politician in the state.

And in Washington, he's a fascination: at 51, a youngster in Senate terms, he's probably the only member who's a regular in triathlons and half ironmans. He may look a little like the actor Richard Gere, but his dismal childhood was more like David Copperfield's.

"How many moves did you make, how many different houses when you were growing up?" Stahl asked.

"Within the first 18 years, it was 17 different moves, 12 separate houses, and that doesn't count the times I ran away," he replied.

Back in his hometown of Wakefield, Mass., he tells us about how he suffered as a little boy from physical and psychological abuse at the hands of his mother's many husbands.

His first stepfather was Dan Sullivan.

"He was kind of a mean cuss," Stahl remarked.

"Yeah, he was a truck driver. Hot tempered. You know, I remember him being a pretty good drinker," Brown said.

In the book, Brown tells what happened when he woke Dan up from a hangover. He "plowed into (me) with those massive knuckles until I was shaking, sobbing, snot pouring out of my nose."

"And then he starts beating up your mother. There's an incredible scene in the book, you're 6 years old, and you jumped to your mother's defense?" Stahl asked.

"Yeah. I was in bed, and I heard her screaming. And I remember him hitting her...just pounding on her. And I remember just grabbing his leg, and just biting it right in the inside of his thigh," Brown remembered. "He was just pounding because I wouldn't let go. And was trying to shake me off, and everything."

But when Brown was 10, he experienced another kind of abuse at the hands of a camp counselor.

"You tell us that you were actually sexually abused. More than once," Stahl remarked.

"Fortunately nothing was ever fully consummated, so to speak. But it was, certainly back then, very traumatic," he said.

"You say it wasn't consummated. But you knew...he touched you," Stahl said.

"Yeah, he touched me," Brown said.

"And he made you touch him," Stahl said. "And then you thought you had escaped, and he came for you again. ...More than once."

"As predators do," Brown said. "He said, 'If you tell anybody I'll kill 'ya. You know, I will make sure that no one believes you.' And that's the biggest thing. When people find people like me at that young, vulnerable age, who are basically lost, the thing that they have over you is they make you believe that no one will believe you," Brown said.

He never reported it, and told no one, including his own mother.

"My mom will read about it for the first time. My wife (too)... I haven't told anybody. That's what happens when you're a victim. You're embarrassed. You're hurt," he said.

What about his parents? He says his father Bruce had all but abandoned him, moving to another town, raising a new family, rarely visiting and paying such little child support, that Brown threatened to take him to court. His mother Judith, a waitress, struggled to pay the bills, going on and off welfare. He writes that she drank and they battled. When she threatened to break his sports trophies, Brown would run away for days.

He told Stahl he started getting in trouble and was playing with matches.

"That's the garage I almost burned down," he told Stahl during a drive through his hometown.

He seemed to want to show Stahl the "low points" of his childhood.

"Well, this is, used to be a record store, I stole some records there," he told Stahl. "And then this is (where) Park Snow used to be, and this is where I stole the suit. And this is the, used to be the old A&P," Brown said. "Stole food from there."

His crime spree ended when he was caught walking out of a mall with "Black Sabbath" records in his overalls, and arrested. He was 13. Lucky for him, a sympathetic judge let him off.

"You came perilously close to being a juvenile delinquent," Stahl said.

"I did, I know. I think about it still, to this day, every single day, I think about it," Brown said.

Brown says he never sought professional help for the traumas in his life; sports have always been his therapy.

As a teenager, he would break into the gym at Wakefield High School to shoot hoops deep into the night, literally working out his anger. He became co-captain of the varsity team and admits to being a bit of a hot dog on the court.

With the help of his coaches, he got a scholarship to Tufts University. After college, he got his big break: he won a good-looker contest and became a centerfold in Cosmopolitan magazine.

That launched his modeling career, which is how he paid for law school. His success as a lawyer made him wealthy for the first time in his life.

Stahl was curious about how his deprived childhood has shaped him as a politician and asked about his votes against extending unemployment benefits and jobs programs for low-income teens.

"Your mother was on welfare. You even benefited, you tell us in the book...from one of those kinds of programs, CETA," Stahl pointed out. "And yet you vote against those kinds of programs. I'm just trying to get at the mindset."

"I see what you're trying to, you know, infer somehow that I'm not, you know, focusing on people that are unfortunate," Brown replied.

"When you make those votes, are you thinking, 'My God, this is what I...,'" Stahl said.

"I think about each and every vote. It's not...," Brown said.

"But those particular...," Stahl said.

"Every vote. I think about how it affects people each and every vote," Brown replied.

"But I'm asking, do you think of your own childhood?" Stahl asked.

"No, not really. No. In the back of my mind...," Brown replied.

"Really?" Stahl asked.

"Yeah. These are votes that are real. They're today. And it's 45 years later. And in this day and age, with the deficit running so high, we have to find a way to use the money that's there in the system. It's critical," he said.

How his childhood influences his politics is one thing. How it impacts his family is another. Unlike his mother and father - each have been married four times - Brown has been married only once: for 24 years, to TV reporter Gail Huff.

"Scott is an opposite of everything he went through," Huff told Stahl. "You know, he went through all these marriages, and he said, 'I want to have a good marriage.' He went through a difficult childhood. He said, 'I want to be a good parent.' All these experiences ended up having a reverse and opposite effect on him."

She says he never missed a basketball game or horse show when their daughters were growing up. Ayla, now 22, is an aspiring country singer and Arianna, two years younger, is studying veterinary medicine. In his book, Brown laments the loss of his family's privacy since he became a senator.

"But now with this book, don't you think it's gonna be twice as much?" Stahl asked.

"Really? See I don't think that," Brown said.

"It's awfully intimate," Stahl pointed out. "It's like going on Oprah and spilling it all out there."

"Well, it's who I am. I can't hide from who I am. It's what happened," Brown said.

When Stahl said he didn't have to tell it, Brown said, "Yeah. But then it's like half truths. I like to just get it out there."

Brown says writing his book helped him reconcile with his parents, Judith and Bruce.

His anger is behind him, he says, or most of it, as we found when we asked him about the house where he lived in high school, and where he fought hand-to-hand battles to protect his mother from yet another violent stepfather named Larry.

"I heard that not many years ago Larry's house, that house, came up for sale, and you actually thought about buying it," Stahl said, as she and the senator stood outside the house.

"I actually called the realtor and went in and took the tour and relived kinda where everything was and put it, to make sure I wasn't kinda dreaming, and as I left, I said, 'Man, I wish I had the money. I'd just buy this thing and burn it down,'" Brown remembered.

"So you wanted buy it to burn it down?" Stahl asked.

"That's it. Yeah. It's a beautiful house. The people that live there, I'm sure, love it. But, you know, if the walls could talk," he said.

"Right. Well, they'll talk now," Stahl pointed out.

"Yeah," Brown agreed.