They can be separate.
For men’s and the women they can nominate 5, but if they have individuals qualified they must be included in the five. A travelling reserve can be included (but not counted)
For the mixed relay, they can only nominate 2 men and 2 women (no requirement to include qualified individuals) and there’s a travelling reserve for each gender (again, not counted as a quota)
So the max quotas would be 14 (5 in men’s, 5 in women’s, 4 in mixed). In practice it would be far less as there would be crossover (and individual qualifiers would bring the number down too)