This looks like vole damage on the roots of your nandina. Voles are a type of meadow mouse that feeds on the roots of trees, shrubs, and herbaceous plants. They are difficult to control and there are no easy answers. Use mouse snap traps baited with apple slices or peanut butter to reduce numbers. Reduce habitat by keeping weedy and grassy areas mowed and keep mulch about one inch in depth to reduce protective cover. See our publication for more information https://extension.umd.edu/hgic/voles
mh