This appears to be a problem caused by a larvae group of skeletonizer caterpillars. The following is some general info that I hope is useful.

On small trees, the best nonchemical control for these gregarious caterpillar species is to search the foliage for egg masses and tips of branches for colonies of larvae, then prune and destroy the infested leaves or twigs. This should be done early in the summer when populations are usually low since the initial infestations are found primarily on the lower branches.

A control technique that may be helpful is to till the ground thoroughly (if appropriate) in the immediate vicinity of the trunk in the fall. This will destroy some of the overwintering pupae.

For larger trees where it is impractical to search the foliage, chemical control may be required. Chemical applications should be made as soon as groups of larvae appear. Some insecticides that can provide control for these colony feeders are certain formulations of insecticidal soap, neem and Bacillus thuringiensis (Bt).

Hope this helps!