Sex hormones and stress hormones are linked in that if one is high, the other will be low. Stimulants like sugar, caffeine and nicotine all raise your stress hormones making you less likely to get in the mood.