Children affected by my husbands ADHD

I'm hoping I can get some advice from families with small children. We have a 5 year old and a 6 year old who have both experienced the explosive anger from my husband. If I ever go out for dinner with a friend and my husband puts the kids to bed, I come back to crying scared children because daddy got so angry over something small. Now my daughter shakes and get anxious over the thought of being alone at night with him (I figure it's only at night because that's when he has a couple beer and it negates the affects of the med's). When she communicates that she is scared of his anger he always turns to me and says "it's your fault for brainwashing the children that I'm a bad person". Then he always says to my daughter " I guess nobody likes daddy and I'll just leave the family". He has no ownership over his anger and puts it on everyone else. For now I just go out the odd time in the morning with a friend and always stay with the kids at night, but my kids just told me they think " daddy is mean to mommy". I feel so trapped because I could never risk losing custody of the children because I don't feel they are safe with their dad. My husband feels it's ok to get rough physically with the kids because that's how he was raised.  The kids say they love their daddy and they would be sad if he was out of their lives.  He's a good father in many other ways but he does treat me like he hates me and blames me for the wedge between him and the kids..unfairly.