I don't think it's in the IOC's place to demand which nations are allowed to be part of a sports federations. These are separate entities. If a sport wants to allow all 50 United States states to compete separately then let them. My only concern is that the qualification process must limit the amount of athletes that attempt to qualify if other nations are also limited. Which for the most part they do for Great Britain (see curling or rugby sevens).
For territories, the ones that do compete were rather lucky as they essentially got grandfathered in. To me, there's no reason why Aruba gets to compete, but not Curacao. For nations with limited recognition things get a bit grey. Where do you draw the line? Palestine, Kosovo and Taiwan (via loophole) are all recognized by a large portion of the world. Western Sahara also has a large chunk, but not enough sporting recognition.