I think for us the problem was that we invested a lot for athletes to be in Athens and still had some of those leftover for Beijing, but by the time London came around so many of them had retired. Plus we had an economic crisis which began between Beijing and London so that obviously didn't help either. However it seemed like for some sports there was no vision after Athens. It was like we focused on Athens so much that we neglected the future after Athens. I'm not sure how much Brazil focused on the future after Rio. China and the UK seemed to do a much better job than Greece in that regard.
We also very well could've had 128 instead of 103 in London. We had two shock non-qualifications (men's basketball where we were ranked 4th in the world and women's water polo where we won the 2011 World Championships), so those two sports obviously played a role in our massive drop in team size from Beijing to London (we qualified both of those teams in Beijing).
And please don't say that we will have 73 athletes in Paris, I refuse to believe it
To be honest though I think in Paris we will go back to 100+. We have a very young team going to Tokyo so I expect many to return, plus we have a lot of good young athletes/teenagers in sports like athletics, swimming, rowing and a few others which gave us hope. Plus having France host might make it easier for us in some team sports, especially our men's and women's basketball team as there will be one more European spot.