Different genres of writing entail various levels of syntactic and lexical complexity, and how this complexity influences the results of Automatic Writing Evaluation (AWE) programs like Grammarly in second language (L2) writing is unknown. This study explored the use of Grammarly in the L2 writing context by comparing error frequency, error types and writing complexity for university admission test essays, textbook-based descriptive essays, social network site (SNS) posts, and SNS comments. Several findings were revealed. Punctuation, grammar, vocabulary, and spelling mistakes were the most common error types across all corpora. In particular, determiner errors such as article usage were most frequent while differences existed across corpora related to incomplete sentences, run-on sentences, and spelling. Test-based and textbook-based compositions consisted of longer sentences with less lexical variation than SNS-based writing. SNS posts and comments produced greater noun, verb, and modifier variation. After applying Grammarly corrections, SNS posts resulted in the greatest clarity and this was attributed to shorter sentence length and simpler word-choice than textbook-based and test-based writing. Grammarly was more appropriate for local surface-level errors (e.g. articles, preposition, and verb-noun agreement) while instructors are needed for issues related to awkward wording and cohesion. Pedagogical implications in the English as a Foreign Language (EFL) context are given.