Lima 2019 increased its total quota by 600+ and their budget is about half of Toronto 2015. Ie. You do not need to have a cap on athletes, but other frivolous stuff.
I experienced both a Pan Am Games and Winter Olympics on the ground. Obviously the Winter Olympics are bigger globally, but do you really need two separate hockey venues? They could have merged it into one and had the competition across two weeks. Branding of venues was way over the top, if you cut that budget you would save so much money.
The Summer Olympics are a totally different animal. Why is it necessary to have 7! large stadiums for soccer. All of which will need staff, volunteers, security, branding etc. Even with 4 or 5, you can have a competition where the field has enough time to recover.
I can go on and on...