If you have too few individuals, then the population is not stable meaning it collapses and the species dies off (due to members not able to reproduce at the proper rate/amount).
If you have too many individuals, then the resources are over-consumed and it's not a sustainable model. In other words, there are too many individuals and there isn't enough food, water, etc.
The key is to have a balance so you don't go too low or too high with the population.