Fever is not harmful or dangerous and since it is not very high (over 106 or 107 degrees F is considered to be very high) then it is unlikely to cause brain damage or some other problems. Also you should know that a child's temperature can vary during the day and can normally reach a high point of 100.2 degrees F in the evening so it is very important to know when you were checking your child’s temperature.
Fever is not a disease but usually it is a symptom that can accompany many childhood illnesses and infections so the only way to be sure that nothing is wrong with your son is to call your pediatrician and make appointment to see him.
Since there are no other symptoms your son is probably just fine but you are a mother and your job is to be worried over every single thing.