It is now widely accepted that hand gestures help people understand and learn language. Here, we provide an exception to this general rule--when phonetic demands are high, gesture actually hurts. Native English-speaking adults were instructed on the meaning of novel Japanese word pairs that were for non-native speakers phonetically hard (/ite/ vs. /itte/, which differ by only a geminate) or easy (/tate/ vs. /butta/, which differ by a geminate and also their segmental composition). The words were presented either with or without congruent iconic gestures, for example, "Ite means stay" (with a STAY gesture). After instruction, participants were given phonetic and vocabulary tests for the words they had learned. Although performance for the phonetic task was above chance for all conditions, gesture played different roles in the semantic task for easy and hard word pairs--it helped word learning for easy pairs, but it hurt for hard pairs. These results suggest that gesture and speech are semantically integrated during word learning, but only when phonetic demands are not too high. (Contains 2 figures and 3 footnotes.)