Adam Clarke on Exodus 28:1

Aaron - and his sons - The priesthood was to be restrained to this family because the public worship was to be confined to one place; and previously to this the eldest in every family officiated as priest, there being no settled place of worship. It has been very properly observed that, if Moses had not acted by the Divine appointment, he would not have passed by his own family, which continued in the condition of ordinary Levites, and established the priesthood, the only dignity in the nation, in the family of his brother Aaron. "The priests, however, had no power of a secular nature, nor does it appear from history that they ever arrived at any till the time of the Asmoneans or Maccabees." See Clarke's note on Exo 19:22.