The theoretical framework of this study is based on the prevalent debate of whether prosodic processing is influenced by higher level linguistic-specific circuits or reflects lower level encoding of physical properties. Using the dichotic listening technique, the study investigates the hemispheric processing of Japanese pitch accent by native Japanese listeners and two groups of nonnative listeners with no prior pitch accent experience but differing in their native language experience with linguistic pitch: native listeners of Mandarin (a tone language with higher linguistic functional use of pitch) and native listeners of English (a stress language with lower functional use of pitch). The overall results reveal that, for both native and nonnative listeners, the processing of Japanese pitch accent is less lateralized (compared to lexical tone processing, which has been found to be a left hemisphere property). However, detailed analysis with individual pitch accents across groups shows a right hemisphere preference for processing the high-accent-low (H*L) pattern, a left hemisphere preference for LH*, and no hemisphere dominance for LH, indicating a significant reliance on the acoustic cues. These patterns are particularly prominent with the English listeners who are least experienced with linguistic pitch. Together, the findings suggest an interplay of linguistic and acoustic aspects in the processing of Japanese pitch accent by native and nonnative listeners.