For some athletes, the biggest prize in their sport is not an option, so the olympics became their by default-objective. A cyclist like Tom Dumoulin is not going to win a GT anytime soon (even, if we've seen worse in cycling ) , but he makes a great deal of winning the olympics.
Likewise, a young polish rugby players is never going to win the 15 a side rugby world cup (it'm crushing your dream here I know ), but he might have a shot at participating to the Olympics and that objective might become his main goal.
It doesn't make the olympics the more important event of the lot, but simply the biggest event one athlete could successfully win/particpate to.