Sound through headphones for XBox One not working?

I have a little problem and I don't know how to do it.

I have an XBox One and you can connect headphones to the controller so that you don't hear the sound from the TV but through the headphones. But that is not the case. Does anyone know why that is? Or how can I adjust that?

I use my headphones that came with my phone when I bought it.

Look in the audio settings to see if the output is also via headphones.