A colt can be castrated at any age as long as both testicles are descended into the scrotum. The testicles are usually in the scrotum at birth; however, it is not unusual for one or both to remain in the body cavity until ten months of age. Occasionally the testicles are not completely descended until the twelfth month of age. Many farms will castrate during either the winter or spring months prior to the colt's first year of age. Insects are less of a problem at this time of year, and cooler weather should help prevent swelling.

Is there any truth in what I have heard that gelding a colt before the age of 1 1/2 to 2 years of age will stunt it's growth??

This is not true.  Testosterone produced by the testicals goes to produce muscle mass only in certain areas and produces more of a cresty neck. The horse's skeletal maturity is not determined by the testosterone production.  Usually colts will be castrated around weaning, this could be anywhere from 4 to 6 months.  Some veterinarians feel that the younger they are the easier they heal.  You didn't mentinon your colts age, however, if they testicals are descended already I would say you are safe to castrate him.

Good luck and I hope this helps.