NEW YORK (CNN) -

He says it's his last fight. And it's been his hardest.

After 43 years in Congress, New York Rep. Charlie Rangel said the campaign for his 23rd term is his final one.

The 84-year-old "Lion of Harlem" wants two more years in Washington, but on Tuesday, voters will decide if his time is up.

He's a founding member of the Congressional Black Caucus, and the former chair of the powerful Ways and Means Committee.

But the legend of Harlem politics is on shaky ground.

Rangel says he's ready this time

He's been bruised before. In the 2012 Democratic primary, victory came by a thin margin. Rangel beat state Sen. Adriano Espaillat by fewer than 1,100 votes.

The congressman said he didn't put up much of a fight.

"I didn't have a campaign last time. When he told me he was running I was in the hospital in Columbia Presbyterian with a viral infection in my spine."

This time, Rangel said he's ready.

"Well, I don't have a walker. I don't have a spinal injury."

Rangel has also had more time to recover from a humiliating censure in 2010 from the U.S. House of Representatives following a series of ethics violations.

Gearing up for Round 2

Since 2012, Espaillat has been gearing up for Round 2. This time, the underdog said he's counting on an upset and making comparisons to one of the biggest ones ever in boxing:

"Sonny Liston was a big, bad bear when he got into the ring, but (Muhammad) Ali was faster, smarter than him," Espaillat said. "He was able to draw circles around him, and he shook up the world and he changed boxing."

If voters want a change now, it may be a measure of how much the district has changed since Rangel went to Washington in 1971.

Harlem's rich African-American history now seems distant

Neal Schumacher knows the streets of Harlem as well as anyone. He gives walking tours, and he often points out the fact that the neighborhood is, "...not the Harlem that I grew up in, but a Harlem I embrace."

A falling crime rate has accelerated real estate development in the neighborhood, but affordable housing remains among the most pressing issues for many with roots in the community.

"People are concerned about gentrification, about rising housing costs, about some of the mom and pop shops closing. About being able to afford to live in the community where you are now," said Schumacher.

Harlem has long been known for its rich African-American history. But to Harlem historian Jacob Morris, that past feels more distant today.

"In the 20th century Harlem was incontrovertibly...the cultural capital of Black America. ...Is it still? I would say now it's the custodian of that great history."

Both say racial and ethnic politics not part of race

Today, the demographics are different. So is the district itself.

Part of the Bronx was added to Harlem's district before the 2012 primary. The 13th Congressional District now has a Hispanic majority.