For me, these are the bottom five sports at the moment...
Golf - The IGF/IOC has had trouble attracting the best players. That said, golf is globally popular. You can find golf courses all around the world. Venue can also host other sports (Athletics, Equestrian, or even Cross Country Skiing as demonstrated by Pyeongchang 2018).
Sailing - World Sailing is struggling to stay afloat financially, and the sport presents a lot of broadcast challenges. That said, it’s a good sport for universality. Plus, sailing as both a sport and skill/trade has been around for a long time, so it’s a natural for inclusion on the Olympic program.
Modern Pentathlon - It’s likely the least popular sport on the Olympic program due to poor coverage, confusing competition formats, and exclusivity in who can even participate/practice the sport. That said, it’s a historic sport. It’s the one sport created specifically for the Olympic Games, and so it makes a good novelty.
Boxing - The sport has had well documented issues with governance and bout judging. That said, this another good sport for universality. It also is one of the most historic and commonly practiced sports.
Weightlifting - Another sport that suffers from poor governance. It’s also experienced rampant doping. It’s another historic sport that’s also good for universality purposes.
That’s my look/opinion at the sports I see towards the bottom of the program. However, I think the current Olympic program is right about where it needs to be. I’d only drop sports on the basis of governance issues, rampant doping, cost effectiveness, decline global popularity, etc.