Aversive handing of farm animals, and the animals’ resulting fear of people, can substantially reduce both production and the welfare of the animals, and increase the risk of injury to the handlers. Research has found that cows can distinguish between people who handle them aversively and those who handle them gently. The aim of this series of experiments was to examine the ability of dairy cows to discriminate between handlers based on the colour of the clothes worn. A total of 66 Danish Friesian cows were used. The average distance cows kept from the person was calculated from 12 scores during 1 min with the person standing in front of the cow. In experiment A, the distance cows kept, from two unfamiliar people wearing red or yellow overalls, was tested. There was no difference in the distance the cows kept from the two people and the two colours. In experiment B, the distance cows kept from an unfamiliar person wearing familiar coloured overalls (blue) was compared to when the same unfamiliar person was wearing red or yellow overalls. There was no effect due to the colour of the overalls. In experiment C, cows were handled repeatedly, by two handlers wearing overalls of the same colour. Each person handled six cows gently and six cows aversively. There was no significant difference in distance kept from the gentle and the aversive handler. In experiment D, cows were handled repeatedly by one person wearing either red or yellow overalls. The person wore one colour when handling the cows gently and the other colour when handling aversively. The cows kept a longer distance when the handler wore the colour worn during the aversive treatment than when the handler wore the colour worn during the gentle treatment. The cows also responded according to the colour of the clothes when worn by an unfamiliar person, although they kept a shorter distance from the unfamiliar person than to the handler. The results suggested that cows could learn to avoid or approach the same handler based on the colour of overalls the handler wore during the gentle or aversive treatment.