Psychometrician, AWS Training & Certification

The AWS Certification team is seeking a Psychometrician with experience working with criterion-referenced assessment programs to support a large global AWS Certification and Credentialing program. In this role, you will support all psychometric aspects of exam development and operation, including job analyses, standard setting, automated test assembly, item and test analyses, optimal item bank design, quality assurance, and project planning.

You will work closely with a team of psychometricians, subject matter experts, certification exam program managers, publishing, delivery, security, and product management teams to support ongoing analyses of exam and credential data. To be successful in this position, you must be highly motivated, creative, detail oriented, and a self-starter who is able to think big, execute, ensure high quality, yet stay focused on the details.

Key job responsibilities

• Conduct Job Task Analysis (JTA) workshops and post-JTA survey analyses to define the blueprint and test specifications for new certifications or updates to existing certifications
• Conduct standard setting studies to set the passing score for exams and credentials
• Run item analysis to evaluate quality and performance of exam items
• Use automated test assembly procedures to assemble forms or item pools
• Work with content development to track item bank trends and optimize the health of item banks
• Support the development of a cloud-based analytics and reporting system
• Partake in development and performance analysis of credentials
• Interpret and clearly communicate the results of analyses to stakeholders through written and oral reports
• Follow the accreditation standards set by ISO/IEC:2012 17024 and the National Council for Certifying Agencies (NCCA) as they relate to valid psychometric practices
• Contribute to the development and execution of the strategic goals regarding the AWS certification and credentialing program.
• Consult with leadership, internal staff, external consultants, and industry leaders regarding advancement of current offerings

Basic qualifications

• PhD
• At least 1 year post-doctoral direct experience working with high stakes certification or licensure programs
• Experience conducting job task analysis and standard setting