Hello and thank you for your question. You have a couple of non-toxic options that will help you. One is to dig out and hand pull every weed and be on the lookout for more to emerge. The other is to cover the weeds with dark plastic to "cook" them in the heat of summer, and then to more easily dig them out. No herbicide will be specific enough not to kill your periwinkle, and the residue might be harmful to any other desirable plants you try to put in their place.

I'm sorry that I don't have an easier option for you, but if you're diligent you'll defeat these weeds in time.
Jeanne