Obviously you have a hole chewing pest for starters.  But there may be a disease going on as well.  It is common to have both.  Trees that are stressed for whatever reason are much more likely to be attacked by pests and disease.  Not enough water, too much water, severe winter weather, and other weather related conditions are all some of many possibilities that can stress the tree, making it more vulnerable to attack.  Keeping the tree healthy is the very best defense in problem avoidance.

Does not address the current situation however.  For that it seems a professional evaluation is the best approach.  An evaluation by a certified arborist is generally not expensive and you could then decide how to go forward based on professional information. You can find one or more certified arborists in your zip code at www.treesaregood.com  

A positive ID of everything going on in the plant is necessary before any treatment can begin to be considered.  He/she will look at much more than leaves.  They want to see where the tree is growing, how the tree is growing, what is growing around it, etc.  Usually they will look at everything else in your landscape while they are there as well.

Good luck!