Anomaly detection is a challenging problem that has been researched within a variety of application domains. In network intrusion detection, anomaly based techniques are particularly attractive because of their ability to identify previously unknown attacks without the need to be programmed with the specific signatures of every possible attack. There is a significant body of work in anomaly based intrusion detection applying statistical analysis, data-mining, and machine learning disciplines. However despite more than two decades of active research, there is a striking lack of anomaly based systems in commercial use today. Many of the currently proposed anomaly based systems do not adequately address a series of challenges making them unsuitable for operational deployment. In existing approaches, every step of the anomaly detection process requires expert manual intervention. This dependence makes developing practical systems extremely challenging. In this thesis, we integrate the strengths of machine learning and quality-of-service mitigation techniques for network anomaly detection, and build an operationally practical framework for anomaly-based network intrusion detection. We propose methods for self-adaptive, self-tuning, self-optimizing, and automatically responsive network anomaly detection. In specific, we propose and develop methods for adaptive input normalization adjusting scaling parameters online based on evolving values in observed traffic patterns, adaptive algorithms for flow-based network anomaly detection that respond to feedback to account for concept drift, and evolving methods for aggregated alert correlation that consolidate individual alarms into network events. We propose and design a model for dictating optimal performance in an anomaly detection system and reinforcement learning algorithms for automated tuning and optimization and a confidence forwarding model to support automated response. Furthermore, we develop a fair bandwidth sharing and delay differentiation mechanism for scalable automated response that insulates network resources from malicious traffic while minimizing collateral damage. We develop a prototype network anomaly detection system that integrates the proposed and developed techniques. We evaluate developed approaches using the 1999 Knowledge Discovery and Data-mining Cup and MAWI Lab datasets, but also we create a new dataset based on a combination of live network traces and controlled simulated data injects. Results demonstrate the effectiveness and capability of automated means. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]