The damage to the leaves is probably caused by the elm leaf beetle. The larva chew on the underside of the leaf and produce skeletonizing injury to the leaf. CSU has a fact sheet devoted to the elm leaf beetle. Here is a link to the fact sheet:

https://extension.colostate.edu/topic-areas/insects/elm-leaf-beetles-5-521-2/

This particular elm tree may be a siberian elm which is a hardy, fast growing tree with brittle wood that is susceptible to breakage. The elm leaf beetle doesn't affect the tree trunk or limbs. 

Other than identify the leaf damage there is nothing more that we can do to help you. You may want to talk to your neighbor about trimming the tree since it was last trimmed 15 years ago. You may want to research the feasibility and legality of  trimming that part of the tree that encroaches on your property.