In the five years following implementation of a chat reference service at James Madison University (JMU), the service proved very popular but was not closely assessed for quality of service. Using grounded theory and qualitative data analysis techniques, a comprehensive assessment effort was begun in earnest and is in progress. Preliminary results indicate several patterns and difficulties that will inform the staffing model and training for this service and will potentially be generalizable to other similar libraries. Other libraries may find high value in assessing the transcripts of their chat reference interactions to gauge specific trends in their individual libraries.