My son has always been the baby of his class. He has a very late summer birthday. Most of the students in the class below him are actually older than he is. However, if you did not know his birthdate, you would never know. In fact, I have four kids and they all have summer birthdays and they all are the youngest in their classes. So far, I have not seen a problem with it. In fact, there are some advantages (like Jess being able to wrestle kids fed's as a Junior).

Age is just a number. There is an advantage with a Senior (whether 17 or 19) over a Freshman and even a Junior over a Sophomore. However, I don't think there is that much of a difference between an 18 year old and a 19 year old (or a 14 year old and a 15 year old). As far as 20, I believe that you can not be 20 years old and compete in sports in High School without a waiver. I am not 100% sure though.