Using an experiment, I examine whether middle-level supervisors incorporate observable good and bad luck into discretionary evaluation decisions, and how this affects employee behavior. Although the controllability principle asserts supervisors should not incorporate observable luck, I find that supervisors incorporate observable luck partially because they find it fair to do so. Although supervisors correctly anticipate that employees find it fair to be rewarded for good luck more than they are punished for bad luck, I find no evidence that supervisors reward good luck more than they punish bad luck. Employees’ effort is lower when supervisors incorporate observable luck but only after employees learn how supervisors evaluate them through repeated interactions. My results suggest fairness concerns can diminish one of the main theoretical benefits of allowing discretionary evaluations. Specifically, fairness concerns can prevent supervisors from using all available non-contractible information to decrease the weight of luck in employees’ compensation.