Two Cain backers in N.H. decamp for Gingrich

Two New Hampshire state representatives who were backing Herman Cain for president have decided to pull their support from Cain in the wake of claims that he was involved in an extended extramarital affair.

As WMUR first reported, state Reps. William Panek and Sam Cataldo will now back Newt Gingrich for president. 

Panek told CBS News he had been on the fence concerning his support for Cain for the last couple weeks because of Cain's political rhetoric, and that the allegations of the longtime affair was the last straw.

CBS News could not immediately reach Cataldo, but the Gingrich campaign confirmed that he had switched his allegiance to the former House speaker.

Panek said he initially "didn't take [the] allegations at face value," but that once the woman presented evidence of the affair "I pretty much just had had enough." (Atlanta's FOX 5 News reported that accuser Ginger White showed reporters cell phone bills that showed 61 calls or text messages - including in the early-morning and evening hours - from what she said was Cain's private cell phone number. Cain called the station after it texted the phone number for confirmation. He has denied affair, saying he and White were simply friends.)

Woman alleges 13-year affair with Herman Cain

Panek said he contacted the head of Cain's New Hampshire team to inform him of his decision, and that Cain's New Hampshire director said he "completely understood the decision," adding that the situation "didn't make us look good either."

"It was bad enough when it was 1 or 2 women," Panek told CBS News, referring to previous sexual harassment allegations against Cain. He said that any affair is "between him and his wife" but that "the handling of it does bother me." Panek added that "you don't want to say [Cain is] lying but...It makes us look bad too because we're supporting him."

Cain has denied the affair as well as previous sexual harassment allegations. Gingrich has acknowledged an extramarital affair with his current wife while married to his second wife.

The news follows the Cain campaign's acknowledgement that it is "reassessing" Cain's candidacy and would complete the assessment "over the next several days." Cain reportedly said on a conference call Tuesday morning that he needs to decide if the affair allegation creates "too much of a cloud, in some people's minds, as to whether or not they would be able to support us going forth."

Herman Cain "reassessing" candidacy after affair claim

On Monday, White came forward alleging the long-term "very inappropriate" relationship with the former Godfather's Pizza CEO. Cain said he knew White but denied the affair. A statement from Cain's lawyer said White's claims of a consensual affair between two adults is not a legitimate news story.

Cain slipped in the polls following late-October accusations of sexual misconduct by multiple women while he was head of the National Restaurant Association, accusations he has strongly denied. From the middle of October through the first week of November, Cain was polling in the 20-30 percent range, putting him atop polls of the GOP presidential contenders. Recent polls show him sliding to support in the mid-teens, behind Mitt Romney and Newt Gingrich.

WMUR reported that two additional New Hampshire representatives who had backed Cain "have concerns" about the candidate but want more information about the affair story.  It also noted that a sign on the door to Cain's locked New Hampshire office Tuesday afternoon read "out to lunch."

With reporting by John Dickerson and Caroline Horn.

Full CBS News coverage: Herman Cain