Whereas great advances have been made in the statistical sophistication of assessments in terms of evidence accumulation and task selection, relatively little statistical work has explored the possibility of applying statistical techniques to data for the purposes of determining appropriate domain understanding and to generate task-level scoring rules. Now that complex tasks are becoming increasingly prevalent, the inattention to item-level scoring is becoming more problematic. This study utilizes exploratory techniques to examine the differences between experts and novices in command usage and troubleshooting strategies in the field of computer networking. Participants were students and instructors of the Cisco Networking Academy Program as well as experts from the field of networking. Each participant was asked to perform troubleshooting tasks and a log of their actions was recorded. Log files containing all commands that participants entered while completing the troubleshooting tasks were analyzed using techniques of Statistical Natural Language Processing (SNLP). Results indicated that experts and novices differed in the types of commands that were used as well as in the sequence of those commands. Moreover, some patterns of examinee response that were found were entirely unexpected, leading to a rethinking of the appropriate conceptualization of the domain and the tasks. Previous assumptions about expert novice differences were shown to be faulty along with previously constructed scoring rules based on those assumptions. Comprehensive research in the application of statistical techniques to the understanding of domains and the validation of scoring rules are recommended.