To be honest, athletics and swimming both exceed their quotas by hundereds at every Games & just athletics has 1900 athletes quota allocated, which is definitely a number that could be cut (and don't get me wrong, I'm an athletics fans, but 1900 really is huge number).
In team sports if only 1 player per team would be removed, it would already make a difference (as it happened with waterpolo, but the IOC caved to FINA and their request).
I'm not saying reducing quotas is necessary, but if the IOC is going to push forward with it, they will make it like this, now with any radical decisions.