If that's what someone is worried about, then not talking to women doesn't do much to prevent that possibility. So either way, not talking to women is a stupid strategy.
I would actually be more suspicious of a person with that policy if they were accused. It's weird. Like they know they have a problem.
Maybe not but less chance for someone to say "I talked to Mark and he said this to me." Sure you can still say it but easier to rebut "I have never seen Mark with June."