Tbf unless you’re really anal about it, you’re never going to have perfectly clean hands all the time. But if your controller is getting discoloured quickly and regularly, then you can probably tell where the problem lies regardless of its colour.
I have similar issue with mouse, but it takes about 3-4 years of regular use for it to get really noticeably discoloured or start rubbing off top coat, to a point where even cleaning with IPA doesn’t do anything. And that’s a black mouse.
Also worth noting, many hand creams are a big no-no imho. Some of that shit destroys your gear, worse than any dirt.
Most things in our regular day-to-day lives would leave residue, even water (maybe deionized wouldn’t?) just a matter of how much and how noticeable. About the soap, well if you smell it on your hands after wash, then it’s still there and you will rub it onto stuff. At least it’s just soap tho…
I don’t know if you can reasonably remove all contaminants off your skin, unless you plan to like glove up or destroy the skin by washing waayyyyy too often. But you know, good hygiene will keep things you touch nicer for longer so keep at it.