We do not have enough information from your photos to make an accurate assessment on the decline of your holly.  Looks like possible winter damage.  The branches look bare and may be dead.  In general, dead wood can be pruned out any time. Scratch the branches with your fingernail and look for green tissue. If you see, it the branch is viable. In general hollies start dropping their old foliage around May and new growth should emerge.  Foliage can be slow to develop.  Prune dead wood and monitor your shrub throughout the growing season.  If you notice additional symptoms, send more photos of the whole shrub, around the base of the shrub, and affected foliage. You may be dealing with a possible root problem, soil compaction, drought, etc. 
 See our website for more information on winter damage. http://extension.umd.edu/hgic/winter-damage-trees-and-shrubs 
mh