It sounds like you might have animals -- squirrels, raccoons, and/or deer -- feeding on the fruits. Unfortunately, there is not an easy solution. Generally, you first have to identify the animal (this page can help: http://icwdm.org/handbook/index.aspx) and then choose appropriate methods for exclusion, habitat modification, use of repellants or traps. Refer to our fact sheet on dealing with nuisance wildlife. https://extension.umd.edu/sites/extension.umd.edu/files/_docs/programs/hgic/HGIC_Pubs/wildlife/HG90%...

This page has good information on how to deal with squirrels, specifically.
https://extension.psu.edu/tree-squirrels

Christa