Non-ADD spouse at wit's end

This is my second marriage and my husband's (who has ADD) 4th. When I fell in love with and married my husband he was taking Welbutrin. After we were married (3 years ago)  he took himself off Welbutrin and stopped taking Adderal. He said he's fine and doesn't need it..... And that his psychiatrist agrees (The 10 minute phone call once a month psychiatrist). I can go about a month being loving, supportive, patient, tip toeing around his defensiveness and sensitivity and being invisible before I get angry..... I get really, really angry and try to hurt him verbally. I hate myself for doing this. It's not like me. I raised two children (now adults) without raising my voice and now I scream like a lunatic. The anger comes from the fact that he's never wrong and I'm always wrong, he has no desire to have sex since he went off Welburtin (but blames me), that he is selfish and self-centered (never gives me anything or recognizes my gifts and contributions) and that he has diagnosed me with a borderline personality disorder because I get so angry with him and loose it. YIKES! (I actually went to a psychiatrist who assured me that I do not have a BPD) My  husband loves to be a victim. I don't understand it. He has lost many jobs and seems to have difficulty with authority figures.

Now here's the part that makes me nuts. He has no insight about his behavior... his life. The problem was with his mother, his 3 x-wives, the sibling, the friend the "boss", and now me. I went from being the love of his life to being "the problem". Why can't he see what everyone else sees or at least believe those who care for him. He adores his granddaughter, but his daughter will not leave her with him unless I'm there to supervise (she also won't let him drive with her daughter in the car). He denies these things.

Please help. I went from being a strong independent woman to almost believing if I could just never loose my temper everything will be okay.

Wit's End