Just another thought - if you met a man who had walked away from his children and not looked back what would you think of him?
Your husband is doing his best in difficult circumstances and you should be proud of him. Jealousy will ruin your relationshipif you are not carefull. Councelling may be a good idea, best wishes.