Thank you for contacting the UVM Extension Master Gardener Helpline, Christine.  The photos are a great help. 

This appears to be a problem with the plant itself, possibly the roots.  From the photos, we don't see any signs of insect damage.  I have a couple of suggestions to help you determine the source of the problem:

You didn't mention which of the two main fothergilla species you have.  Dwarf varities (such as Mt Airy) are slightly less tolerant of cold temperatures than the larger varieties, with a USDA Hardiness Zone of 5-8.  The larger variety is included in the Zone 4-8 category.   Washington County may be slightly too cold, if this is the case.

I can't tell how much sun the plant is getting.   Both large and small varieties prefer a sunny location.   Smaller varieties may tolerate a bit of shade, but in both cases, the showier spring flowers will appear on plants with more sun.

These plants require well drained, slightly acidic soil.   A soil test will determine the soil PH.  If soil is not acidic, the addition of a fertilizer which increases the acidity (usually advertised for plants such as holly or blueberries) will help.

The plant may fail if the roots are circled (sometimes these roots are circled tightly in the pot, and if it is planted that way, or the burlap or cage is not removed), roots will continue to circle and not reach out for water and nutrients.

Planting too deep may also kill the plant.  Always make sure the root flare is at ground level.

In your summer photo, it looks like there is bark mulch on top of the roots.  Bark mulch should never actually touch the bark of a bush or a tree.  This may cause the bark to rot, affecting the flow of water to the bush.

Look for signs of animals chewing on the base of the plant.  Rabbits or voles may eat the bark and if they eat completely around the tree (girdling the tree), the tree may die because the water will not flow up through the bark.

I would dig down to take a look at the root system to see if any of these problems exist.

These plants will spread through root suckers, so this may be a way to continue having some fothergilla around.

I am attaching links to information on the care of fothergilla from various universities that you may find helpful.  

Fothergilla—A Four Season Shrub to Consider for Your Garden (psu.edu)

https://hort.extension.wisc.edu/articles/fothergilla/

https://hgic.clemson.edu/factsheet/fothergilla/

I hope you find this information helpful.  Good luck!