It's not a big enough impact though,compared to the impact of monster scaling.
Also,with too few, you can end up with mostly just one mob type over and over depending on how the encounters are set up.
(Which is why when you solo some quests, you end up with nearly all one type of monster, when you are a certain level)