People who've never been in the limelight don't realize the hassle of the simple things in life like going out with your family to a restaurant, seeing a movie or just taking a walk.
At least with actors and musicians it's generally positive reactions unless you do something stupid. Being a famous athlete or a professional or collegiate coach is a whole different story. Fans are very passionate and how they treat you changes with the wind depending on how you perform and the results you produce. You can be the nicest person in the world but if you start costing your team games due to your performance or poor coaching the gloves come off and you suddenly have a giant bullseye on your back.
Lots of people don't realize how it feels to be a standup individual and the nicest person in the world yet be booed, heckled and even have threats made against you just because you dropped a pass in the endzone or decided to go for it on 4th and short instead of kicking the long field goal only to not convert and miss the playoffs because of your call. At least with celebrities their performances in a movie or album don't cause adverse reactions from the fans.
Sports fans are very passionate and only care about what you've done for them lately. And being a part of a big school with a long and proud tradition like Alabama, USC, Auburn, Texas, Florida, Michigan, Oklahoma, Nebraska, LSU and so on and any pro team multiplies the intensity and spotlight by a factor of 10. No matter what your reputation and past record is, you're always on a short leash and people who love you one minute will turn on you in a heartbeat and despise you and scream for your job the next. Definitely a stressful profession.