My Xbox controller does not work or it only works if I plug the batteries into the Xbox one, otherwise I don't have to do anything and sometimes it goes nuts
Have you ever tried to re-sync it?
If that doesn't help, there may be a solution here. I've never had such problems before. Unfortunately, the controller does not have a proper reset button.