Your eyes go to the darkest part of a "light" image and the lightest part of a "dark" image. If we are taking pictures outdoors and you are wearing light colored clothing, the eye will generally go to the clothing before it goes to the face. The light colored clothing then competes for that attention!
If you wear darker clothes, but short sleeve or sleeveless garments, the bare arms call for attention. Realize too, that if it is a family of five, there are five faces but TEN ARMS!
I know that "all white" is a popular choice, and is very lovely if it is done right, but it takes a little planning on your part as well as mine! It is very important not to let the detail of the white garments "wash out" in your photographs.
Solid colors are most appropriate for families where I am trying to convey a sense of unity. If everyone is wearing something different, it is like having 4 or 5 individual pictures. Not that everyone has to dress alike, it just helps to stay in the same color family!