Affect detection from open-ended virtual improvisational contexts is a challenging task. To achieve this research goal, the authors developed an intelligent agent which was able to engage in virtual improvisation and perform sentence-level affect detection from user inputs. This affect detection development was efficient for the improvisational inputs with strong emotional indicators. However, it can also be fooled by the diversity of emotional expressions such as expressions with weak or no affect indicators or metaphorical affective inputs. Moreover, since the improvisation often involves multi-party conversations with several threads of discussions happening simultaneously, the previous development was unable to identify the different discussion contexts and the most intended audiences to inform affect detection. Therefore, in this paper, the authors employ latent semantic analysis to find the underlying semantic structures of the emotional expressions and identify topic themes and target audiences especially for those inputs without strong affect indicators to improve affect detection performance. They also discuss how such semantic interpretation of dialog contexts is used to identify metaphorical phenomena. Initial exploration on affect detection from gestures is also discussed to interpret users' experience of using the system and provide an extra channel to detect affect embedded in the virtual improvisation. Their work contributes to the journal themes on affect sensing from text, semantic-based dialogue processing and emotional gesture recognition.