These can be a nuisance and I certainly understand your concerns. They tend to choose a particular type of site for their nesting and your yard must just have the right combination of soil type, sun exposure, and other characteristics that they like. I usually just recommend learning to live with them in your yard. They are only actively nesting for about 3 weeks of the year and you will not notice them for the remaining 49 weeks. You could try encouraging the grass to grow more vigorously in the area. They tend to choose areas of exposed soil because they need the sun to warm the developing larvae in the nest. I would recommend starting with a soil test from MSU Extension. You can pick up a soil mailer from your local MSU Extension office or order one from the bookstore here: https://shop.msu.edu/products/bulletin-e3154
I have attached an extension bulletin about establishing a lawn from seed to give you some ideas about how to fill in those patchy areas. After Labor Day is a good time to try to reseed your lawn.