Surveillance video recording is becoming ubiquitous in daily life for public areas such as supermarkets, banks, and airports. The rate at which surveillance video is being generated has accelerated demand for machine understanding to enable better content-based search capabilities. Analyzing human activity is one of the key tasks to understand and search surveillance videos. In this thesis, we perform a comprehensive study on analyzing human activities from short term to long term and from simple to complicated activities in surveillance video achieves. A general, efficient and robust human activity recognition framework is proposed. We extract local descriptors at salient points from videos to represent human activities. The local descriptor is called Motion SIFT (MoSIFT) which explicitly augments appearance features with motion information. A quantization and classification framework then applies the descriptors to recognize activities of interest in surveillance videos. We further propose constraint-based clustering, bigram models, and a soft-weighting scheme to improve the robustness and performance of the algorithm by exploring spatial and temporal relationships between local descriptors. Detection is another essential task of surveillance video analysis. The difficulty of detection lies in identifying the temporal position in a video. Therefore, we propose a sliding window approach to search candidate positions with cascade classification to reduce false positives. Finally, we perform a study to utilize automatic human activities analysis to improve geriatric health care. We explore the statistical patterns between a patient's daily activity and his/her clinical diagnosis. Our main contributions are an intelligent visual surveillance system based on efficient and robust activity analysis and a demonstration exploring long term human activity patterns though video analysis. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]