Think of medieval Europe. Did you imagine a bunch of white people tooling around in the dirt, nary a Black or brown person in sight? You've had the wool pulled over your eyes. People of color definitely existed in medieval Europe.
Historical Accuracy: People of Color Existed in Medieval Europe