I always put in on the married party because after all, they're the ones who took vows so if I sleep with a married man, it's on him, not me. I took no vows to be faithful. That said, I think it varies from person to person and marriage to marriage.
I can imagine circumstances where it is ok to sleep with someone who is married. That doesn't mean it's always okay, but it also doesn't mean it's always wrong.
When you're talking about things as complicated and complex as human relationships, absolutes won't get you very far.