Do you own a second home in a resort area that you use personally or occasionally rent out? You could be in for a rude surprise at tax time, if the state where the home is located insists you’re a legal resident and must pay income taxes. In some cases, that may trigger hundreds of thousands of dollars of extra tax liability.

Several states, including Arizona, California, Hawaii, and New York, have been challenging the assertions of homeowners who claim to live out of state. A state may maintain that the owners actually reside within its borders, even if they consider the dwelling to be a vacation home. With other cash-strapped states taking notice of these tactics, the trend may well continue and expand.

In one high-profile case, John J. Barker, a Wall Street investment manager, and his wife argued that they were Connecticut residents who made only sporadic visits to a second home near the exclusive Hamptons area of New York. During the tax years that a New York state audit brought into question—2002 through 2004—the Barkers said the home was used by other family members. But the couple spent more than half of each of those years in New York, and the state has assessed a tax bill of more $1 million for the three years and tacked on penalties of around $220,000. The Barkers have appealed the ruling but may eventually have to pay at least part of the bill.

In another recent case, media personality Martha Stewart, too, argued that her legal residence was in Connecticut and her place in the Hamptons was just a second home. But New York prevailed and Stewart had to pay almost $222,000 in back taxes.

The Empire State has a reputation for being one of the toughest states for audits, and it often targets residents of states such as Florida, Nevada, and Texas that have no income taxes. But farmers in the heartland and jet setters up and down both coasts face similar problems. In the worst-case scenario, you could even be hit by taxes from two states on a single property.

You may be able to avoid trouble if you clearly establish your residency in a single state of your choosing. Because residency laws vary from state to state, you’ll have to research the method for documenting your legal domicile. At the very least, obtain a driver’s license, register to vote, and file your income tax returns in your home state. Other things that generally help evidence where you live include memberships, where you receive you mail and maintain your bank accounts, and where you register your vehicles. And be sure to keep detailed records of your whereabouts, including a summary of frequent-flier accounts, credit card receipts documenting trips between homes, and phone records that you could use to bolster a claim of residency (or of non-residency).