I don't think it is racist either. As with most all social experiments, and yes, that show is definitely a social experiment, races and sexes are usually divided. Perhaps next certain minority groups will call the Census Bureau a racist organization because the data they collect uses a person's race as well as other information.
Of course, the only way that tribal divide in that manner could be deemed racist is if the challenges were geared to favor one race over another. And we all know that is not the case. There have been winners who were white, black, latino, male winners, female winners, homosexual winners. No sir, definitely not racist.