Here's a closer view of the caterpillar.

The pellets are indeed just scat - termed "frass" for insects. These are a very common native insect called the Orange-Striped Oakworm. They are in the same moth family as our much-adored giant silkmoths (Luna, Polyphemus, and so on) and, while they feed gregariously and therefore cause more prominent defoliation, rarely cause serious damage to trees.

You do not need to prune out damage, as the twigs are still alive and will be able to leaf-out normally next year. On a small, young tree, if defoliation is affecting a high proportion of the canopy (perhaps more than a third or half), you can try to remove the caterpillars physically - either by hand, if you can reach them, or with a strong jet of water from a hose. Birds and other animals will be happy to consume them wherever they land. (They won't eat just any plant they land on, so you don't need to worry about spreading the problem around. They strongly prefer oaks, and eat little else.) Pesticide sprays should be a last resort and are rarely recommended for this insect; we don't suggest using them here. These caterpillars look nearly full-grown and will be ceasing feeding soon, when they will then wander-off and disperse to pupate elsewhere.

Trees can afford to lose foliage a bit prematurely in late summer since they have spent all season photosynthesizing and leaf shed is relatively imminent anyway. They also produce more leaves than they need, so losing some at any point during the growing season is of little consequence to them long-term.

Oaks are a vital component of our ecosystem, and feeding native insects like this is a crucial role they play in maintaining our biodiversity. Birds, wasps (both larger predators and tiny parasites), insect-specific pathogens, and an array of other natural control mechanisms work together to keep outbreaks minimized from year to year and populations in check. As this young oak ages, insect presences like this will be less and less noticeable. Here's a blog post by one of our retired entomologists on this particular insect: http://bugoftheweek.com/blog/2019/8/26/trails-of-despair-for-wild-silk-moth-caterpillars-orangestriped-oakworm-anisota-senatoria-and-greenstriped-mapleworm-dryocampa-rubicunda?rq=oakworm

Miri