The growths you see on the branches of your azalea are called lichens and are harmless in and of themselves.  They do not cause a plant or limb to die, but they do tend to appear more often and in greater numbers on dead or dying limbs.  They can also appear on healthy trees and shrubs.  It is not necessary to remove them or to remove any healthy, living limbs.  If the limb is obviously dead, remove the entire limb.  You can always give a limb the 'thumbnail' test: simply scrape off a small bit of bark.  If the tissue under the bark is green, the limb is alive.  If it is brown, it is dead.  If the older azalea has smaller leaves and thinner foliage, you may want to apply some azalea food with an acidifier.  Also, make sure the plant receives supplemental irrigation during times of drought.