Will Charlie Rangel follow Eric Cantor out the door?

For all of the public's dissatisfaction with Congress this year, voters have so far ousted just two incumbents -- 91-year-old Rep. Ralph Hall, R-Texas, and House Majority Leader Eric Cantor, R-Va.

On Tuesday, the House of Representatives may face more upheaval.

Eighty-four-year-old Rep. Charles Rangel, D-N.Y., first elected to Congress in 1970, is up against three primary challengers -- including state Sen. Adriano Espaillat, who fell just about 1,000 votes short of defeating Rangel in the 2012 primary and is once again a serious threat.

As in Cantor's case, the polling ahead of the primary appears to be on the incumbent's side. And like Cantor, Rangel enjoys high-ranking status in Washington. The circumstances in New York's 13th district are, of course, different than those in Virginia's 7th: unlike Cantor, Rangel has fended off serious challenges before, in spite of his record of ethics problems. Additionally, Rangel's district, which was once predominantly African-American, has gone through dramatic demographic changes.

• Thad Cochran promises to deliver, but is Mississippi interested?
• Eric Cantor: "No regrets" after stunning re-election defeat

Still, Rangel's fate ultimately hangs on the factor that decided Cantor's: turnout.

"Any time you're dealing with a primary, it's always a turnout game, how do you expand your base," New York Democratic strategist Evan Stavisky told CBS News. With nothing else on the ballot Tuesday, turning out supporters will be all the more challenging. "Certainly there's some challenges that each candidate faces, but we'll see who wins with their ground game."

And in New York City, rallying one's base often means mobilizing support from specific ethnic groups.

"[Former] Mayor [David] Dinkins called New York City the 'gorgeous mosaic," said Stavisky, who is not representing Rangel or Espaillat. "Each one of those tiles is trying to crowd out the other in order to win on Election Day."

A NY1/Siena College poll of likely Democratic voters released last Thursday illustrates the demographic dynamic that has contributed to Rangel's past success as well as his current challenges. The poll gives him a 13-point lead over Espaillat, thanks in large part to his strong hold on African-American voters in the district.

Among African-American respondents, Rangel holds a commanding lead of 70 points over Espaillat. Among Latinos, however, he lags 53 percent to 29 percent. Those numbers wouldn't have posed a problem for Rangel in the 1990s. In recent years, however, demographic changes have increased the Hispanic presence in his district. In 2013, newly-drawn political boundaries created a district merging Rangel's traditional northern Manhattan constituency with parts of western Bronx, giving the 13th district a 55 percent Hispanic - mostly Dominican - majority.

The above map breaks down the 13th district into the community districts it contains. The one with the lowest Hispanic share of the population (22 percent) is Central Harlem, a historically black neighborhood. Yet as the map shows, Central Harlem has by far had the most growth in the share of Hispanics (42.6 percent) from 2000 to 2010. The highest Hispanic population shares are found in Washington Heights (71 percent) and the Bronx (63 percent to 67 percent).

Bruce Gyory, a political consultant in New York who's not working for any campaign, told CBS News that there's clear "breakthrough motivation" for Dominican voters who'd like to make Espaillat the first Dominican-born member of Congress. As when New York's voters elected Al Smith to be the state's first Irish-Catholic governor, or when New York City elected Dinkins as its first African-American mayor, race and ethnicity is once again a major factor, Gyory said.

The Siena poll suggests that Rangel's overwhelming lead among black voters more than compensates for his disadvantage among Hispanics, and 89 percent of respondents to the survey said the race or ethnicity of their representative makes no difference to them.

Still, Gyory pointed out that pollsters have a long track record of underestimating minority voter turnout. For instance, the likely voter sample in a 2010 Siena poll said minority voters would account 15 percent of the turnout in New York's gubernatorial race. However, exit polls showed it turned out to be 29 percent. In the 2012 presidential election, Gallup predicted nonwhite voters would account for 20 percent of the turnout, but exit polls showed minorities made up 28 percent.

That means in the 13th district, where minorities make up 80 percent of the electorate, the polls could be significantly off. Gyory also emphasized how close Espaillat came to defeating Rangel in 2012.

By suggesting Rangel holds a double-digit lead, "we're rejecting two-year-old muscle memory, when it was a very close race," Gyory said. "With all the demographics continuing to move in favor of Espaillat, all the real factors point to a close race."

Gyory also questioned whether the Siena poll is accurately tracking white voters in the district, particularly since the survey was conducted before the New York Times endorsed Espaillat. While endorsements may not always matter, it could help Espaillat boost his name recognition among this portion of the district.

"These are precisely the kinds of voters who look to the Times -- they tend to be socially liberal but they're not necessarily active in the local block association," he said. "They look to the Times for a sort of 'Good Housekeeping' seal of approval,"

Rangel, meanwhile, is using one of his own big-name endorsements -- from former President Bill Clinton - to encourage voters to come out on Tuesday.

In a robocall the Rangel campaign recently released, Mr. Clinton says, "Charlie is the only candidate with a proven track record of delivering results on the economy. He's a fighter for good jobs, successful small businesses and working families. He has been for 40 years now."

Andrew Guess contributed to this report.