You may have an invasion of walnut husk flies. These flies lay their eggs on the developing walnut husk, and the maggots crawl inside and begin feeding. It seems that most of the information about this fly is coming from California, due to the fact that large nut farms are common there. However, this fly is native to central and eastern US.

There is also an insect called the walnut curculio and another called the Navel orangeworms which might be found in walnuts, but they leave webbing in the nut. Do your walnuts have webbing inside? Do the walnuts drop prematurely? If so the pest could be the walnut curculio which is a small weevil about 1/5 inch long. The larvae feed in developing nuts on black walnut causing nuts to drop prematurely, referred to as “June drop" of walnuts. The larvae tunnel into developing nuts and within a few weeks infested nuts drop from the tree. On the ground, larvae remain in the nuts for about 2 weeks after which they chew an exit hole in the husk and enter the soil, where they pupate into adults. The adults then emerge in mid to late summer and spend the remainder of the year feeding on leaves until the leaves drop. Then the adults move into the soil where they spend the winter. So, sanitation is one of the main keys to reduce the moth infestation. Remove the infested nuts that fall on the ground before the adults can move back into the soil. You may find the following websites of interest, they recommend different types of control along with various insecticides:

As with all insecticides, be sure to read and follow the label instructions. Hope this was helpful.