Thank you for your question and picture. Sadly, it looks like deer chewed on your river birch since it appears to be at the same height as they feed. Another common animal who feed on small trees and shrubs are rabbits. Damage like this usually happens in the winter where food is scarce but can occur at any time. I can't quite see how tall your fence is, but it should be buried a few inches into the ground to deter rabbits. If your damage is on all sides of your tree, it may die. See how it survives the winter - it may come back, and you can trim off the dead areas. Also, cover the lower areas with a tree guard, which can be seen in this article, along with other information:

Winterizing plants: A good fence is the best defense against animal damage | UMN Extension