Thank you for your question.  Rather than being a worm, this appears to be a llia underwing caterpillar, which are often found in oak trees.  You'll note that it looks like a lichen, which gives it good camouflage.   The species is also known as the Wife  or the Beloved Underwing.   The caterpillars feed on leaves of woody plants, and the moths are largely nocturnal, so you won't see the adult stage unless you're out late at night.

I've attached a photo of a known llia underwing caterpillar, a photo of the moth it will become, and a photo of some strikingly similar lichen (which you should also have plenty of in your oak trees!)

Hope this is helpful.