A 'swimming detractor' not convinced here. Swimming olympic program almost double up from 1964 to 1968. Coincidentally, it's also at that time that television (mainly american) gained importance and coincidentally again it also allowed the US to secure their top position at the medals table with more than 50 medals at the pool that year (lol, we can all imagine how people wearing red stuff thought about that at the time).
It's also interesting how the first olympic program was draw. Swimming was merely one section of the so called nautic sports (Part E) alongside rowing and yachting (sailing). Doesn't seem that core to me.
IMO, swimming as this second major olympic sport alongside athletics is a very artificial thing and only feels ok if you come from one these very few countries that can rack up medals at this sport every 4 years.
Athletics is the only king.