Separating financial matters

I am fortunate that family members have helped me financially, especially in the years since my husband lost jobs and did not seek out new employment.  I currently am feeling secure enough to want to invest some of this money instead of having it in a very safe (read "almost zero interest") account.  My husband and I have some shared investments and so the logical thing would be to go to the broker who oversees those investments and ask him to oversee my separate money, as well.  However, I don't want to place him into an uncomfortable position, either personally (he is one of the few people outside the immediate family that my husband has any kind of relationship with) or professionally (in the not-unlikely situation my husband and I get a divorce).  What do you think?  Should I go to this guy or find someone else?

Thank you.