Oh that is so frustrating, those are nicely shaped plums.  Thanks for including the photo of the leaves, the little holes gave me the idea to look up "shot hole blight" as a potential disease, as well as brown rot as you wondered.  

Here is an Extension publication from WSU:
https://hortsense.cahnrs.wsu.edu/fact-sheet/plum-prune-fresh-shothole-coryneum-blight/

And one from Utah State Extension:
https://extension.usu.edu/planthealth/research/coryneum-or-shothole-blight

And University of California:
https://ipm.ucanr.edu/PMG/GARDEN/FRUIT/DISEASE/shothole.html

Unfortunately, diseases like this are hard to control during the growing season, as most control efforts are based on spraying during the dormant season with copper-based fungicides.  Make sure to read all the information on orchard management and sanitation in the articles.  There are some recommendations for approved pesticides (in Washington) for home orchardists in the WSU article.  

Good luck!  Cindy Brown, OSU Sherman County Extension