Recent rapid malware growth has exposed the limitations of traditional in-host malware-defense systems and motivated the development of secure virtualization-based solutions. By running vulnerable systems as virtual machines (VMs) and moving security software from inside VMs to the outside, the out-of-VM solutions securely isolate the anti-malware software from the vulnerable system. However, the external placement of the anti-malware tool introduces a number of limitations, including the well-known semantic gap problem. In this dissertation, we study the limitations in prior out-of-VM approaches and develop the "process out-grafting framework" in order to effectively address them. First, we address isolation and compatibility challenges in out-of-VM approaches for fine-grained process execution monitoring by developing two key techniques. The first key technique, "on-demand grafting," relocates a suspect process from inside a VM to run side-by-side with the out-of-VM security tool. This effectively removes the semantic gap and supports existing user-mode monitoring tools without any modification. The second key technique, "mode-sensitive split execution," forwards system calls back to the VM and enables continued execution of the out-grafted process without weakening the isolation of the monitoring tool. Our experiments with a prototype show that we can effectively use "process out-grafting" to natively support a number of existing tools without any modification. The evaluation results, including measurement with benchmark programs, show the effectiveness and practicality of our approach. Next, based on the fine-grained monitoring capability, we apply and extend process out-grafting to enable semantically-rich out-of-VM policy enforcement. Specifically, we demonstrate out-of-VM system call policy enforcement, which effectively restricts the behavior of an out-grafted process. Further, in order to facilitate the secure observation of a process that violates system policy, we develop the VMsnare component of our framework. In VMsnare, we have designed and developed our next two key techniques, "attack preservation" and "live analysis." With these two techniques, we effectively extract live malware processes from a production environment into a honeypot for flexible and extensible analysis. Our experiments with a prototype implementation demonstrate the effectiveness and practicality of our approach. Finally, in our framework, we facilitate the time-traveling forensic analysis of intrusions and derive valuable insight into attackers' techniques and motivation. Towards this, we have designed and developed the Timescope component of our framework, which leverages insights from previous VM-level deterministic record and replay systems and enables multi-faceted and extensible forensic analysis. We have further extended Timescope and developed a number of honeypot-specific forensic analysis modules. By repeatedly traveling back in time, multiple phases of analysis can be performed, either in parallel or sequentially. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]