Is it normal to have smaller testicles

Hi I am 32 years old male having moderate sized testicles. I got checked by the doctor for testosterone levels. It was found to be above normal. He even checked my balls and penis and declared to be normal. I also have very good sperm count and having 1.5 year baby girl. Due to small testicles when i sit my pants go inside and i feel very embarrassing due it. Some of them even confirm me as hermaphrodite. I think some one can understand my problem i cannot even explain to them who behave as illiterates.This is affecting my social life too. With out any subject regarding this they straight away consider me as hermaphrodite or shemale. Is there any solution to increase the size of the balls?