I dont think that is true broadly and definitely not in the US. This article says only one abandoned Atlanta Olympic structure exists today and that was because a shady college abandoned it. Rest are in use. This article says 85% of structures are in use around the world and Athens was the worst about it.It’s more than that. It is investing in facilities that have absolutely no use outside of the Olympics
Swimming Stadiums
Bicycle Velodrome
Canoe/Kayak course
These facilities are abandoned as soon as the Olympics are over