How long do you take the verbal and mental abuse?

Wow, this is a new thing for me and I am really taking a risk.  I am married to a man who has ADHD.  He chooses not to seek medical management.  

We have a young daughter and have been married 5 years.  We met and married a year and a half after knowing each other.  Knowing what I know now, I married him during the hyper-focus stage.  I thought, stupidly, that I was really just that great!  That he loved me, not loved being in love and the "thrill of the chase." 

He has always been really bad with money.  To the point where "our" savings, which is really MY savings has been dwindled down by THOUSANDS of dollars.  I am to the point where I am going to start transferring money into a separate online savings under the guise that I do not want to be "tempted" to spend money.  I really am just tired of working, saving and them having him "borrow" from our savings and never putting it back.  When I ask about the money I am lashed out at as if I am the one with an issue.  Told that I only care about money.  That's not the case.  I just dislike him having all the latest toys while I work to pay bills and put food on the table and save what I can.  It's getting old.  Before we married I had a really good savings and managed my money very well.  Now, I am out of control and in a spot that if I say ANYTHING about money I am being "paranoid" or "insecure."

When I was pregnant, he decided one day that it would be good to try to find other women to flirt with to boost his aging self-esteem.  Long story short, his escapades ended after about 8 weeks.  During that time, he bought a secret pre-paid cell (which I found - his excuses were great!), went to lunch up to 70 miles away when I was at work to meet a woman I assumed he met online.  He went to lunch with supposedly the same woman at a very fancy and expensive restaurant (which he does not take me to) on the totally opposite side of town while I was at work.  I think that they were two different women, but I have no proof.  Had me take pictures of him which he used to send to his prospects. He sent flowers to another woman that was 12 years younger than him (who has since married and is expecting a child - can you say golddigger? Only there is no gold - except mine, which is gone).  Did odd grooming habits (get the drift without me saying) and on the same day I noticed him doing that, he just had to go to work at 9 pm at night and there was a charge for a hotel within walking distance of his office that same night.  I ended up finding out about his affairs, that he claimed were only emotional and he never "met" or did anything "physical" with the younger woman, the following day after his hotel charge (cannot say stay, as he did come home around 1 pm in the morning).  After me finding out, he contacted her and let her know I knew.  There is more to the story, but I do not want to go into great details.  Either way, the point is that he stopped contact and focused on me, the baby and the rest of the pregnancy.  His justification for the affair was that he needed to feel wanted and liked the "thrill of the chase."  He said that he never had problems meeting women and was feeling unattractive (blamed me) and that he was losing his hair.  He wanted to feel like he was desirable.  He cried like a baby after I found out and begged for forgiveness.  I gave it.  However, I know that if I EVER even talked to another man, he would leave me with no change of explanation or reconciliation.

The real issue now, and what I want opinions on, is how to deal with a man that when he gets mad, explodes and says hateful things (i.e. he should have never married me, he wants a divorce, etc.) then five minutes later is back to normal.  When I say I am still hurt, his response is, "Can't you just move on?"  The awful thing is that my parents were very abusive and never apologized for their actions.  He now is the same way.  He hates them for how they are, but doesn't see that he is abusive.  

Recently we have had his family move in with us temporarily.  Things have been hard, as I have cultural and personality differences.  I was raised differently and take offense to guests in my home telling me what I should watch on TV, taking over my living room and rearranging my things.  I voiced my concerns to my husband and he took liberty to go and tell them, though I said to him that I would just ignore things for the time being.  It ended up in him getting into a fight with his family when I was not home.  Blamed me.  Said that I am "insecure, rude and that I do not "open up" to people.  He then went into a rant about how he didn't know me well when we married and if he knew then what he knows now that he would not have married me.  Funny thing is, I was really happy when I met my husband.  I had money.  I had my own apartment.  My own car.  My own friends.  I gave all of that up to be with him, thinking that I had a man who loves me for me.  I didn't know I married someone who would lose complete interest in me after we married and would take my money, not honor his words, get mad at every little thing, misinterpret having a conversation for "complaining", get mad if he is told "no", spew hateful venom from his mouth and then justify it by saying that he was just "mad" and didn't mean it.  Yesterday, when he told me all of these things, I told him I was so confused because just two days ago he told me how me and my child were "number 1" to him and that I was his life-partner.  When I asked how he could go from that to "I never would have married you"?  His response was, "things change."  He just kept saying that in a very hateful tone.  I then went and confronted the family members and I had received a very distorted picture of what their conversation and fight was really about.  I thought they had said horrible things about me and they were both very shocked at what I told them was said.  Things ended find with them understanding that I prefer things a certain way, etc.  My husband did a complete 180 and was super sweet and calling me and texting me all day today.  

As I have said, I know about the ADHD and what I have written is just part of the story.  I could document daily the CRAP he pulls and the attitude that he has.  He is super nice to everyone but me.  I tend to think that it takes all he has to hold it together in front of others and I have become the whipping post because he knows I won't leave.  But, the question is, how long should I put up with this?  Is it fair to be verbally and emotionally abused by someone with ADHD and just chalk it up to ADHD?  At what point does the person start taking responsibility for their actions and lack of control?  Should I stay or should I just make plans to exit knowing that the next 40 years probably will be hell?  I love him and just want him to be the way he used to be.  But, sometimes I think that it was just an illusion.