Hello,

This is not an insect nest or insect problem. This is a decaying rubber stopper. The  sun dries out the rubber, cracking and breaking it down. Try poking a knife into the grey mass and see if breaks apart.
Sliding glass doors have rubber stoppers in the track to keep them from banging the end when sliding open. They should be a screw in the center of the circle, holding the stopper in place. There are replacement stoppers sold for when they break down as yours has.

Regards,
Carol Quish