The hardest thing about being married to an ADHDer

I have been married to my husband, John who has ADHD and possibly also Asperger's Syndrome for 3 years and we have a 2 year-old daughter together.  Though there are so many struggles being married to someone with ADHD, but I think that the hardest part of all is that most other people don't understand and can't relate.  It's hard when family members and friends turn their backs on you because they think you are just married to a loser, loafer, etc.  It adds so much hurt to someone who is already struggling when the people they love and try to find comfort in give them the cold shoulder.  At least that has been the biggest problem for me I think.  My husband has had numerous jobs since we've been together (working on getting #12 right now).  Others don't understand why he has problems keeping jobs so they just assume he is a loser who won't support his family and that I am a dependent wife who just doesn't want to leave when I should.  They don't get the fact that he tries so very hard to do right and to be like "other people", but it is an uphill battle for him.  Does it cause me hurt and frustration and anger?  Of course!  But I also remind myself that he is doing the best he can and he has to find a job that fits him with all of his unique qualities.  He is a good man with a heart of gold, but he has made some mistakes and used poor judgment at times.  My sister and her husband pretty much cut themselves off from us because they view my husband as an immature idiot who won't hold a job and they are mad at me for staying with him.  That hurts me so much because I have always been close with my sister, but this has put a strain on our relationship.  My parents have struggled with it too, but I think they are finally starting to see that ADHD is a REAL diagnosis.  My dad has come from the school of thought that people are overdiagnosed and overmedicated for things like this, but I can tell he is starting to soften up a bit.  Recently when he found out my husband lost his last job, he said "well we will just pray that he finds a good fit next time".  That almost brought tears to my eyes when in the past I would always hear "well maybe you need to think about leaving him", etc.  It's hard when everyone around you is saying you are stupid to stay with someone you love.  I just wonder if other people have the same problem of wanting to be able to talk about things with a friend of family member, but there is no one around who understands because they don't live with an ADHDer the way I do.