Tobacco bud worms! I spent a summer squishing buds in a window box of petunias as an “preventative” to kill the critters inside and it worked, but I didn’t plant petunias the following year. This year the petunias on my deck didn’t develop any “worms” until a couple of weeks ago, when I noticed the black frass (poop). On my to-do list for today is to clean out the plants, as you say, it’s late in the season anyhow.
Here is the WSU Hortsense pest page, though they start the list as a problem of roses. There are chemical controls, but we encourage cultural options whenever possible.
Understanding the life cycle is helpful too. They pupate in the soil, and the Colorado Extension pest page has good photos and descriptions. I have wondered if the early ones are from leftover soil or purchased with the petunias, and these later season ones flew in from neighbors, as there are often two generations.
I hope this helps and whatever you grow next year blooms well.