The wealthiest residents of the Hamptons aren’t in the Hamptons in early May. For the most part, they’re in New York City. But there are still traffic jams at rush hour and people in the houses, preparing for the people who own the houses.

The southeastern end of Long Island has two populations: The residents, who are rarely in residence, and the workers, who are usually there but never feel at home. Both move through the same streets, the same houses, the same rooms — and yet they rarely encounter one another. The people paid to wash the large glass windows in the beachfront mansions are also paid to be gone before the arrival of the people who enjoy the clear views of the ocean. The people hired to clean the homes are the bookends to someone else’s summer weekends.

In Southampton, during the week, day laborers often wait for work in a grass field next to a 7-Eleven. On the weekends, all that remains are the patches of dirt where their feet have worn through the grass. As the buses of bright young things roll east from the city, the main road out of the Hamptons is clogged with pickup trucks and vans dressed in the liveries of landscaping services, construction firms, cleaning firms. Locals call it the “trade parade.”

Many of the residents, who come to look at one another, tend to look through the workers. They do it because they can and, more than that, because they need to. Some of the Hispanic immigrants who make this place run do not have the right to live in the country where they labor; the systematic dependence on workers without legal permission to be here requires the systematic pretense that they don’t exist.

It may seem odd to suggest there’s anything that the residents of this place can’t afford, but the fabulous Hamptons is a discount wonderland. As in other rich neighborhoods across America, almost everyone is getting a deal on one of the most expensive luxury goods: Human labor. The nannies and dog walkers, the house cleaners and gardeners, the delivery men and dishwashers — many of them are people from other places, willing to do jobs that Americans won’t take or to work for wages that Americans won’t abide. Often both. It’s a relationship of mutual dependence, and mutual unease.

Even workers who become citizens — or the second generation, born at Stony Brook Southampton Hospital — still feel like outsiders. One of the workers I met told me that he came from Peru two decades ago. He learned to build mansions, became an American citizen and built a home in the Hamptons for his own family. A few years ago, however, he sold his home because he could not afford the property taxes. Sometimes, he says, he feels like a ghost.