People with autism spectrum disorder (ASD) experience stigmatization rooted in negative attitudes or prejudice toward them due to social awkwardness. However, little is known about implicit attitudes toward ASD, especially differences in attitudes compared to those of more visible conditions; physical disabilities. In this study, we implemented implicit association tests (IATs) to assess implicit attitudes. Sixty-three university students participated in IATs and answered questionnaires that measured explicit attitudes, social desirability, knowledge about--and familiarity with--disorders. The results demonstrated that implicit attitude toward ASD was significantly less negative than toward physical disabilities. Regarding the discrepancy, not socially awkward behavior but appearance of people with ASD can be evaluated as 'in-group' members and lead to less negative attitude compared with physical disabilities.