I searched Flour Nutrition Different Extension looking for evidence based information from the Extension Service related to your question.  I find adding Extension to the search to be helpful. Sometimes websites ending in .com are less reliable.
If your total grain/flour intake is within typical consumption patterns it should not matter that you are using different flours.  However, I do want to bring to your attention that flour (wheat) is fortified with Folic Acid which is a very important vitamin in the prevention of certain birth defects when consumed by women prior to, and during, pregnancy.