No, it is not fair. If you already must have a system where 3/4 teams advance, then at least add semifinals and finals after the round robin stage.
Any system where outcomes of the meaningless matches (meaningless for at least one of the teams) can have more impact than outcomes of the direct match-ups is flawed.
Besides, if you check recent championships results, you'll see that Puerto Rico has been on-par with Brazil for a while now. Also, they qualified for the 2018 Worlds, while Brazil didn't.