The functional specificity of the brain region known as the Visual Word Form Area (VWFA) was examined using fMRI. We explored whether this area serves a general role in processing symbolic stimuli, rather than being selective for the processing of words. Brain activity was measured during a visual 1-back task to English words, meaningful symbols (e.g., $, %), digits, words in an unfamiliar language (Hebrew), and geometric control stimuli. Mean activity in the functionally defined VWFA, as well as a pattern of whole-brain activity identified using a multivariate technique, did not differ for words and symbols, but was distinguished from that seen with other stimuli. However, functional connectivity analysis of this region identified a network of regions that was specific to words, including the left hippocampus, left lateral temporal, and left prefrontal cortex. Results support the hypothesis that activity in the VWFA plays a general role in processing abstract stimuli; however, the left VWFA is part of a unique network of brain regions active only during the word condition. These findings suggest that it is the neural "context" of the VWFA, i.e., the broader activity distributed in the brain that is correlated with VWFA, that is specific for visual word representation, not activity in this brain region per se.