The least harmful to your environment and your soil would be to roto-till the weedy area (to the right of your periwinkle in the picture) and the slope below the periwinkle and then rake away the weeds that have been tilled up.  You would then be able to plant more periwinkle in the newly-tilled area. You would then need to be diligent in keeping the weeds out until the periwinkle fills in the bare spots.

The roots of the weeds on your slope are helping to prevent erosion of the soil downward. Once you remove the weeds, you will likely need to add some kind of cover to prevent this erosion, such as mulch, some shrubs, or additional groundcover. Mulch will also help prevent more weeds from growing!

Another option is to use an herbicide to kill the weeds. I see both broad-leaf and grassy weeds in your picture. If you chose to use a chemical herbicide, your best option is probably to use a full-spectrum spray weed killer such as Roundup (other brands available at your garden center) because it will kill both types of weeds. (If you use a "grass killer" you will only eradicate the grassy weeds; and if you use a "broad-leaf weed killer" you will not kill the grassy weeds.)

You are probably aware that Roundup has been associated with certain types of cancer; however, it is still on the market and considered safe if used properly according to directions on the label and with adequate personal protection. With that caution in mind, I leave that decision to you.

Weeds growing in the midst of the groundcover will be more difficult to eradicate because using an herbicide in that area would also kill the periwinkle. You will likely need to hand pull those.
One additional option is to get rid of it all (other than where you have all periwinkle) and start over.  Also not an attractive option, but there is no easy option. Preventing weeds from getting started is always easier than eliminating them once they've grown large and are self-seeding!

Good luck!