Students with complex communication needs have increasingly been using non-dedicated communication systems, such as mobile devices, to support their communication needs. This in turn, has led to an increased used of augmentative and alternative communication apps. The main challenge currently faced is the lack of empirically validated apps and evaluation systems to assess the features of the apps. As a result, this study attempted to determine the reliability of an app evaluation tool that was grounded in the components of the feature matching model. The goal was also to identify if the app evaluation tool could be used to evaluate various types of augmentative and alternative communication apps. Participants evaluated apps across the dimensions of usability, output, and display. Results suggest that expert raters were more reliability than novice raters across the various types of apps. Practical implications and future research directions are discussed.