I cannot say for sure that is a mud dauber nest.  I would first spray it with a wasp/hornet spray.  If nothing comes out, use a long board or stick to touch the nest and see if anything comes out.  If so leave the building until they calm down.  Then spray again.  You might us a fogger insect spray.  If nothing comes out, start to remove it but still watch for insects.  I would put a tarp or plastic on the floor to catch the dirt.

Good luck,

Marvin