If you go by Sanford's own words, of which you have now reminded me (I didn't realize he was in congress at the time) yes, but not for his infidelity as Sanford would have preferred. He left the country without informing his Lt. Governor, thereby leaving the state in a dangerous position should an emergency arisen. They would have wasted precious time trying to locate him.
I didn't realize he was another of the "holier-than-thou" Republicans from the Clinton era. Karma hasn't been gentle with them. Gingrich, Hyde, Larry Craig, Mark Foley, Ensign now Sanford.