I have an Xbox one and a normal xbox controller. However, if I stand still during the game, for example I go away for a moment, the camera actually has to stop, but it always goes up with me, even when I'm playing. What is the problem?
Guess that your right stick on the controller is drifting. Have you also tested with a second controller whether the same problem occurs here?
Broken controller, buy a new one!
Is your. Button broken?