Xbox one party, friends hear each other twice and my game sound?

Whenever I'm at the party with friends, my friends hear my game sound and therefore also themselves twice. It can't be because of the headset because I have examined it and the ps4 is not something like that and I also compared my audio settings with my friend and nothing came of it. I may overlook something.

Turn down the microphone level from your headset via sound settings

Nowhere on the xbox can you turn down your microphone level.