(CNN) -

The rare moments Christos Sourovelis can take a break from running his own painting business, he can be found toiling away on his family's dream house in the suburbs of Philadelphia.

"I'm a working guy. I work every day, six days a week, even seven if I have to," Sourovelis says. One day this past March, without warning, the government took his house away, even though he and his wife, Markella, have never been charged with a crime or accused of any wrongdoing.

"I was so upset thinking somebody's going to take my house for nothing. That makes me crazy," Sourovelis says, shaking his head.

The nightmare began when police showed up at the house and arrested their 22-year-old son, Yianni, on drug charges -- $40 worth of heroin. Authorities say he was selling drugs out of the home. The Sourvelises say they had no knowledge of any involvement their son might have had with drugs.

A month-and-a-half later police came back -- this time to seize their house, forcing the Sourvelises and their children out on the street that day. Authorities came with the electric company in tow to turn off the power and even began locking the doors with screws, the Sourvelises say. Authorities won't comment on the exact circumstances because of pending litigation regarding the case.

Police and prosecutors came armed with a lawsuit against the house itself. It was being forfeited and transferred to the custody of the Philadelphia District Attorney. Authorities said the house was tied to illegal drugs and therefore subject to civil forfeiture.

In two years, nearly 500 families in Philadelphia had their homes or cars taken away by city officials, according to records from Pennsylvania's attorney general.

Authorities use a civil forfeiture law that allows them to seize people's property when that property is connected to the sale of illegal drugs.

CNN legal analyst and consumer attorney, Brian Kabateck, says the law is intended to protect the public. "It discourages crime and it takes the ill-gotten gains away from the bad people."

But not all people who have their property taken away are charged with a crime. Unlike criminal forfeiture, the civil law allows authorities to seize property without the owner ever being convicted or even charged.

In North Carolina property can be forfeited only if the property owner is actually convicted of a crime. This is not so in other states.

Civil liberties attorneys with the Institute for Justice, who recently filed a class action lawsuit against Philadelphia authorities for abusing the law, say, "Civil forfeiture is something that is an assault upon fundamental notions of private property ownership and due process."

But Kabateck disagrees, "It's a good law. It works. That doesn't mean that it doesn't sometimes have issues that need to be corrected. The system constantly has to change."

In Pennsylvania, the City of Brotherly Love is far and away the most aggressive in the state when it comes to people's property. Over a four-year period, Allegheny County, the second largest county in Pennsylvania, filed about 200 petitions for civil forfeiture. Philadelphia filed nearly 7,000 petitions in one year alone, according to the class action lawsuit, in which the Sourvelises are plaintiffs, along with other Philadelphia citizens.

Philadelphia officials seized more than 1,000 houses, about 3,300 vehicles and $44 million in cash, totaling $64 million in civil forfeitures over a 10-year period, according to the lawsuit.

The very authorities taking the property appear to be profiting from it, according to Pennsylvania state records. The Pennsylvania Attorney General's office says about $7 million went straight to the salaries for the Philadelphia District Attorney's office and the police department in just three years. In that same time period, records show the D.A.'s office spent no money on community-based drug and crime-fighting programs, according to the Philadelphia AG's office.

The Philadelphia District Attorney's office told CNN it seizes property only as a last resort, and added that it is limited in what it can currently say because of the pending litigation.

"In most cases the Public Nuisance Task Force doesn't pursue forfeiture because the underlying issue with the real estate is resolved when a settlement agreement is reached with the property owner in which he or she agrees to take reasonable efforts to prevent future narcotics dealing from the property."

The DA's office also says it works directly with citizens, the police, government agencies, and community groups in an effort to abate or close drug properties.

Civil forfeiture can be used on the federal or state level. Only eight states -- Maine, Maryland, Missouri, Indiana, Vermont, North Carolina, Ohio and North Dakota -- require seized funds be placed in a neutral account. Other states allow law enforcement to directly profit from the civil forfeitures or put proceeds into a special crime fighting fund.

In some states, like Pennsylvania, the burden is on the property owner to prove their innocence. The Sourvelises say they had to go to a courtroom and fight to get their home back where, instead of facing a judge, they faced a prosecutor from the DA's office.

There was no courtroom or judge, Christos Sourvelis says. "There's just one guy telling us to sign these papers. That's it."

After eight days of sleeping on a family member's couch, the Sourvelises were let back into their house, but only on the guarantee they would ban their son from the house -- a heartbreaking decision, they say. (Their son pleaded no contest to the drug charges.)

Still fighting the city to resolve their case and stay in their home permanently, Markella Sourovelis says: "To me I'm home, but I feel violated at this point. I'm doing things in my house, but I worry is it always going to be my house? Are they going to take it one day like that?"

The Philadelphia District Attorney's office told CNN it strictly follows the state law in an effort to crack down on drug abuse.