Annoyingly, I think that might be a small part of it. The inability to get all the major sports on the program is a larger issue, as is general promotion.
It being a newer event maybe hurts it a little as well. It doesn’t quite have the prestige of the Pan American Games, Asian Games, or even Commonwealth Games.