Jeffrey Beall's blog listing of potential predatory journals and publishers, as well as his "Criteria for Determining Predatory Open-Access (OA) Publishers" are often looked at as tools to help researchers avoid publishing in predatory journals. While these "Criteria" have brought a greater awareness of OA predatory journals, these tools alone should not be used as the only source in determining the quality of a scholarly journal. Employing a three-person independent judgment-making panel, this study demonstrates the subjective nature of Beall's "Criteria" by applying his "Criteria" to both OA and non-OA Library and Information Science journals (LIS), to demonstrate that traditional peer-reviewed journals could be considered predatory. Many of these LIS journals are considered as top-tier publications in the field and used when evaluating researcher's publication history for promotion and tenure.