"They should not be partisan, or single out the tea party," Holman said. "But they should look for political groups, and they're not. They never prevented any of them from registering as nonprofit social welfare organizations."
And now that the issue has become a political hot potato, they're not likely to, he said.
"This controversy means they will never enforce the tax code," he said. "We are going to see the IRS hide."
Who knew what, and when?
Lois Lerner, the head of the Cincinnati-based IRS office that oversees tax-exempt groups, did not respond to requests for comment from CNN on Monday. Lerner is the IRS official who publicly admitted Friday that agents used the keywords "tea party" and "patriot" to flag applications for further review -- but she stressed that was done as a "shortcut" for picking the applications to review, not out of "political bias."
Lerner said in a Friday conference call that she "did not feel comfortable answering" as to when senior IRS officials became aware of the situation. She said the IRS has implemented changes to prevent similar mistakes in the future but could not say that any IRS employees had been disciplined.
But a congressional source familiar with the inquiry understood that Lerner knew of the targeting in 2011 and sent letters to Congress this year without disclosing the extent of her knowledge. And Republicans are questioning why Miller -- who the IRS said was informed of the problem in May 2012 -- didn't raise it when lawmakers asked questions about the issue the following June and July.
"An apology is not sufficient," Camp said. "There have been enough excuses and delays, and now it is time for the IRS to answer to the American people."
The IRS did not respond to questions about why Miller didn't inform Congress nor to questions about when its previous commissioner, Doug Shulman, found out. Shulman told a March 2012 Ways and Means hearing that his agency did not target conservative groups for political reasons.
"I can give you assurances. We pride ourselves in being a nonpolitical, nonpartisan organization," said Shulman, who was appointed by President George W. Bush. "There is absolutely no targeting. This is the kind of back and forth that happens when people apply for 501(c)(4) status."
The IRS has told Congress that Shulman did not know about the targeting of conservative groups when he made that appearance, said Antonia Ferrier, a spokeswoman for Hatch.