That looks like an insect gall. It won't harm the tree you just need to clean up the fallen leaves as you are doing.  In order to control it next year, you would need to ID the insect and spray in early spring targeting the particular insect.  An insecticide will kill beneficial insects as well, though, so since it is not harming the tree, I do not recommend controlling this problem.

I asked our IPM specialist and this is his reply:

I would say it most likely is an insect. It is also an insect that does not warrant treatment. Treatments for gall forming arthropods (insects in particular) require application during early leaf formation (bud break usually) or shortly thereafter. If it is a contact insecticide they are usually broad spectrum and negatively impact the beneficial arthropod community. Work with other gall forming insects has found that treatments with these broad spectrum insecticides provide no more control than the natural enemies provide on their own (i.e., no net benefit, and most likely loss because the natural enemies are not around for other possible pests it may target). Systemic products may provide some control, but I haven't heard anyone using them for this purpose and it may not be on the insecticide label. Additionally, the systemic products are usually organophosphates (older compounds and issues associated with those) or neonicotinoids (possible impacts on pollinators if applied near flowering plants). Since the galls really do not impact tree health, it really is not worth the time, effort, or money needed for treating the tree for this pest. A possible option (although not likely very effective) would be to rake leaves up from underneath the tree in the fall. I don't think this will have much impact because I think that when the arthropod matures it leaves the gall and lays eggs for next year on those buds.