There are many  tips on what to do to prevent wasps from building nests in bird houses.  The most consistent  suggestion is to put a substance on nest-box ceilings. A number of substances have been recommended including wax, petroleum jelly, rabbit fur, and others,  Based on the ecology of paper wasps, soap is probably the most effective option. Here’s why:

Paper-wasp nests are made of wood pulp and saliva. The first cells are attached to a substrate by a slim stalk, or pedicel. Wasps may be deterred by the chemical composition of the soap itself, but it’s more likely that the initial wetness of the pedicel interacts with the soap, preventing the stalk from adhering.

A layer of soap on the ceiling seems most likely to do the trick, as paper wasps prefer attaching their nests to horizontal surfaces.  extending the soap layer just a short distance down the inside walls of the box could increase the effectiveness of this method.

Use an unscented bar of soap with minimal ingredients.

See this factsheet for more information on bluebird boxes:

https://hgic.clemson.edu/factsheet/providing-for-bluebirds-guidance-for-bluebird-nest-box-establishment-in-south-carolina/