The effect of the response scale direction on response behavior is a well-known phenomenon in survey research. While there are several approaches to explaining how such response order effects occur, the literature reports mixed evidence. Furthermore, different question formats seem to vary in their susceptibility to these effects. We therefore investigate the occurrence of response order effects in Agree/Disagree (A/D) and Item-Specific (IS) questions. We conducted an experiment among n = 930 students in which we varied the scale direction (decremental vs. incremental) within A/D and IS questions and asked respondents to evaluate the questionnaires. The results reveal response order effects within the A/D but not within the IS question format. Furthermore, respondent's evaluations suggest that completion of the IS questionnaires requires more consideration than the completion of the A/D questionnaires. Altogether, our findings indicate that IS questions are more robust against response order effects than A/D questions.