Hi Courtney,

I would recommend leaving them and they will die naturally during the summer. Bumble bees, unlike honey bees, do not make perennial nests, but rather they grow in the summer, make a batch of new queens (which disperse) and then the colony dies. Depending on when the colony started it may die by early August or late September. After the bees stop flying, then plug the hole. The bees will not cause structural damage, and will only be aggressive if you directly disturb the nest. This strategy will be the least disruptive and expensive option.