Exactly this. For almost all of these federations, the Olympics are by far the most important event. Many of them are in the eyes of the public basically inexistant for the rest of the time and it's their one chance to get into the limelight for four years. Even the big ones like Athletics would probably sacrifice their World Championships for the Olympics.
The only exceptions to this rule might be men's football, men's basketball, men's golf and tennis. And for golf or tennis wouldn't be much change, because they basically have the same sports calendar year after year. And men's football already made a big change by postponing the European Championships, which will lead to the delayment of other events anyway (even the women's European Championships, so that they aren't overshadowed by the men).