The Internet Age has seen the emergence of richly annotated image data collections numbering in the billions of items. This work makes contributions in three primary areas which aid the management of this data: image representation, efficient retrieval, and annotation based on content and metadata. The contributions are as follows. First, Spatial Mining Pyramid (SPM) hierarchically mines configurations of discriminative local features in an extension of the bag-of-features representation. SPM can also be incorporated into a boosting framework for building classification models. Second, Compressed Vector Ordering (CVO) provides an improved indexing scheme for exact k-nearest neighbor image retrieval. CVO uses distance bounds from approximation files to order clustered retrieval in a way which requires fewer random and sequential disk accesses than other state-of-the-art approaches. Third, SpiritTagger employs a Bayesian framework for automatically annotating geo-referenced photos by simultaneously considering low level visual features and geographic tag frequency. Experiments show annotation performance improves in densely photographed areas and by appropriately filtering the tags. A real-time web demo of the annotation system provides tag suggestions on previously unseen photographs by mining over 1.4 million images. The trend towards increasingly larger collections of data online continues. With effective organization and modeling the scale of user contributed data can empower incredible new applications. This thesis provides flexible techniques for such applications via content-based mining and retrieval of image data. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]