I do not know what this insect is.  I can tell you that it is important to find out the source ( how they are entering your home or their nest).  That is the first step in eliminating the insect.  You said you first noticed them on the windowsill.  Check for an opening there.  If they are small can you vacuum them and then empty your bag or container in trash outside?  You can also visit pestid.msu.edu to see how to ask them for identification.