Any insight?

This really isn't about anger, but it is about frustration..and confusion.

Yesterday we were having a conversation...actually started when I asked his opinion on something on this site...the behavior of a ADD spouse and what he thought was going on. Somehow this led to us talking about 'the past' when we were stuck in the vicious cycle of blame...him blaming me 100%, me blaming him 100%. This led to a conversation that still has me confused and worried. So often he admits his fault in the problems we had, just as often apologizing and stating how happy he is that we're not stuck in that same old pattern anymore. I got a small hint that he was 'defensive' yesterday which was my red flag to just drop the subject, but I didn't. We were discussing his affair and he points out that the only reason for the affair was attention...attention he wasn't getting from me. As a rule, it really irritates me to hear him say that because I know that he knows that I intentionally withheld attention from him...I openly admitted it all along...BUT the reason I did so was because of his consistent disrespect of our marriage and because of just the horrible state of our marriage. I was equally as responsible, and equally as miserable, but I didn't cheat. This is SOO important to me because I need to know that he understands that my lack of affection towards him wasn't for no reason. I told him 1000 times that I wanted attention and affection too..not just sex...and I wanted a good be happy. It was a package deal. point...he said that he had "tried" to do what I wanted (i.e. stop the bad behavior) "for 4 or 5 days", saying it never got him any more attention or affection. I think shock took over at that point and I said "you don't fix years of a bad marriage in 4 or 5 has taken us 10 months to get where we are..when you say things like that, I get scared that you don't understand that there was a reason for me putting up walls" and he immediately got defensive and SOMEHOW interpreted my comments to mean that all of our progress was "fake". HUH??!! I immediately tried to explain..he told me to just drop it..didn't want to hear it...and I told him it was unfair for him to say exactly what he wanted, but as soon as I try and explain, he cuts me "I" am trying to start a fight. I literallly told him I felt like I was speaking French..he simply hears something completely opposite or unrelated to what I was trying to say.

I walked away...coming back later to tell him that I didn't want to fight with him over the past, that neither of us were like that anymore, and it didn't matter. But, the truth is that it scares the hell out of me that he would REALLY think that he "tried and tried" (would 'behave' for 4 or 5 days). It felt like him blaming me for the affair...even though I know he'd insist not. He takes full responsibility for it and I KNOW he feels a lot of shame and guilt and remorse for it. Truth of the matter is, he is a different person now and is getting help (both counseling and is considering medication) so should that be enough? Is it just that he got defensive and went back into that 'mode', but doesn't really mean or even believe it himself?

Our counselor says he depends on me to ground him...I am definitely his rock...and although I fully plan on giving him 110% of myself for as long as he gives me the same...I know that the amount of attention he gets from me is directly related to the way he treats me (MUCH better now) so if he starts behavior badly again and I eventually withdraw from him again, will another affair result? I cannot control I suppose I have no choice but to pray his changes are as real as mine...and that this is never something I have to worry about. We had a dynamic in our marriage that drove a huge wedge between us that has been removed about I don't think we'll ever have those problems again, if that helps. We RARELY ever fought before that 'wedge'.