I think they would have only wanted you to go if you wanted to be there. I don't think they meant the offer to be any sort of burden to you. They asked you to be polite. They will go about their weekend and probably not think twice about it. I highly doubt this would discourage them to ask you to attend things in the future. Considering you met them at Sunday school, I would say the chances of them holding a grudge and never inviting you again are slim to none.