Once it starts to look like it's matching your other clothing, it's a problem. The whole secret is not matching.

I hate anyone who wears a dress belt with jeans.