I think this is a case of "it depends on the weed".

Any weeds that are flowering, going to seed, or that spread by rhizomes or stolons ("runners"), should be removed from the garden. Some weeds can still go to seed even if they are flowering when pulled, like garlic mustard. Rhizomes and stolons may not die and re-root into your soil, defeating your efforts. 

If your weeds have pests or diseases, you would also want to remove them. If, for example, you had a weed in the nightshade family (Solanaceae), a pest or disease from that plant could possibly move to a tomato or potato plant because they are closely related. 

Leaving and soft stems from weeds can act as a mulch in a garden and will eventually break down. They should decompose more quickly than wood mulch. I would be careful not to let them build up more than 3-4 inches. Be aware that the organisms breaking down the organic materials in the weeds/mulch can tie up nitrogen and make it less accessible to your plants. This isn't typically a problem with well established trees, shrubs, or perennials, but you might need to give annuals or vegetables boosts of nitrogen fertilizer throughout the season if you mulch around them.