Competition for housing is soaring, affordability is weakening, and the U.S. housing recovery is grinding to a crawl—and, in large part, the baby boom generation is to blame.

No, they’re not doing anything immoral or illegal; in fact, they’re not doing anything at all, and that is precisely the problem. They’re not moving.

Baby boomers, the enormous group born between the years 1946 and 1964, are staying in their big suburban homes far longer than previous generations did at this age, and that is having repercussions down the housing supply line. If baby boomers don’t downsize out of big suburban homes, younger buyers eager to upsize, especially in this improving economy, can’t find a home to buy.

As the competition from young millennials heats up at the entry level, the logjam at the top of the market only heats that competition and pushes prices ever higher and out of reach.

So, why don’t boomers move? Some if it is because their empty nesta have filled back up again with children moving back home after college, but in large part it is a result of the recession and the increasingly high cost of housing. It is simply cheaper for them to stay put, and so they do.

Baby boomers currently own about 3.6 million unoccupied rooms, according to Trulia, a real estate website that looked at the 100 largest housing markets to find boomers living in homes with at least two bedrooms more than the number of occupants.

Laraine Camera Goldberg is one of them. She and her husband bought a five-bedroom house in North Potomac, Maryland, nearly 40 years ago and raised their sons there. The neighborhood was and still is perfect for raising a family, with quiet streets and good schools. Now Goldberg’s sons are grown and gone, and she and her husband are divorced. She lives alone in the 3,000-square-foot home that boasts a huge yard and pool. She could sell the home easily, as there is very high demand in her price range, but there is one problem.

“I don’t know where to go from here because everything is so expensive. I’m stuck,” Goldberg said. “It’s actually less expensive to stay here and metro into the city than move into the city. So I’m stuck, but I’m in a neighborhood by myself that really I don’t belong in.”

Like so many other active baby boomers, Goldberg wants to downsize to a home closer to the heart of a metropolitan area. Prices in downtown Washington, D.C., are hitting new record highs, and even in the close-in suburbs both rents and prices are cost-prohibitive.

The story is similar in other cities as well. Baby boomers who choose to downsize to apartments are facing huge costs. Just 23 percent of boomers rent, the smallest of any generation, but of those who do, nearly half pay more than 30 percent of their income on housing, according to Abodo, a rental listing company that analyzed data from Harvard’s Joint Center for Housing. That is a higher share than both millennials and Gen Xers.

Goldberg considered moving to an ‘active-adult’ community, where the homes are more affordable, but those tend to be far from urban centers.

“I don’t like the traffic, and there is not a close metro. I mean why do that for people my age?” she said.

Goldberg’s decision not to sell means families looking to move into her neighborhood and raise their children in a big home have one less listing to consider. Multiply that by thousands of other boomers like Goldberg, and the supply situation becomes severe.

“I think the baby boomers are really struggling with the decision to downsize because there’s not really great options for them to downsize to,” said Brooke Heiberger, a real estate agent with Coldwell Banker in Bethesda, Maryland. “And for those who are willing to stay in the suburbs there’s just not a style of home or viable option to allow them to age in place, so it’s just easier to stay where they are in absence of being able to make a good decision.”

About a third of boomers said they intend to age in place, according to recent survey by Freddie Mac. That has contributed to the boom in home remodeling, but it continues to shrink the supply of homes for sale which is at a record low nationwide. Homebuilders are increasing production only very slowly, due to high costs of land, labor and material. Those costs have pushed the price of a newly built home to its highest on record. While builders are focusing more on the active adult brand, they need to keep those prices low and therefore cannot build in the urban areas most baby boomers prefer.

Goldberg continues to look for options in urban cores outside the city, but prices there continue to rise. For now, she stays put and has a lot of house guests. She also travels to see her new grandchild on the West Coast, leaving the big house empty for weeks at a time.