Believe me, there is nothing that would be better for him than to hear those words come out of your mouth to him.
My father and I were at odds for a very very long time, It took me nearly 20 years to finally get over the things that put us apart, I didnt talk to him, I couldnt stand the sight of him and even though I was never rude to him, he knew how I felt and how angry ans pissed I was at him.
A few years ago, Staci convinced me to talk to him, to let him know how I felt deep down inside and to let him know that while the issues that went on with him and my mom affected me too, it was all in the past and how I wanted to at least make the best out of being his son. It took a lot from me, but I called him (he lives in Dallas)
To my surprise, the very next week he made the trip (he was in mexico) and dropped what he was doing and came to Houston so we could talk. I can honestly tell you that besides the biorth of my kids and my wedding day that was easily one fo the best weekends of my life, not only did I get to rekindle that bond we had when I was younger but I had this huge feeling of relief, that he understood what I felt, and that through our talks we became as close as ever.
Believe me it will be worth it, a relationship with your father is not one that you should push aside. you should cherish it now rather than lamenting the could have and would haves later.
good luck