Applied Science Manager, Artificial General Intelligence , Frontier AI Assets: Assessments

Amazon • Boston, MA • 1h ago

Description

The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Science Manager with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to lead a team ensuring the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.

Key job responsibilities

As part of the AGI team, an Applied Science Manager will lead and mentor a team of Applied Scientists who develop comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. The manager will guide the team in designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that align with core scientist team developing Amazon Nova models. The Applied Science Manager will oversee expert-level manual audits, meta-audits to evaluate auditor performance, and provide coaching to uplift overall quality capabilities across the team. A critical aspect of this role involves managing the development and maintenance of LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Science Manager will also oversee the configuration of data collection workflows and ensure effective communication of quality feedback to stakeholders. The manager will have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.

The Applied Science Manager will be responsible for recruiting, hiring, and developing team members, conducting performance reviews, setting clear expectations and growth plans, and fostering a culture of scientific excellence and innovation. The manager will communicate with senior leadership, cross-functional technical teams, and customers to collect requirements, describe product features and technical designs, and articulate product strategy.

A day in the life

An Applied Science Manager with the AGI team will lead quality solution design, guide root cause analysis on data quality issues, drive research into new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. The manager will work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice. The manager will also conduct regular 1:1s with team members, provide mentorship and coaching, and ensure the team delivers high-impact results aligned with organizational goals.

Basic Qualifications

3+ years of scientists or machine learning engineers management experience

Knowledge of ML, NLP, Information Retrieval and Analytics

Preferred Qualifications

Experience building machine learning models or developing algorithms for business application

Experience building complex software systems, especially involving deep learning, machine learning and computer vision, that have been successfully delivered to customers

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you're applying in isn't listed, please contact your Recruiting Partner.

The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits .

USA, MA, Boston - 183,800.00 - 248,700.00 USD annually

USA, WA, BELLEVUE - 183,800.00 - 248,700.00 USD annually

Management & Administrative

Req ID: RQ228601 Type of Requisition: Regular Clearance Level Must Be Able to Obtain: Top Secret SCI + Polygraph Public Trust/Other Required: None Job Family: Cyber and IT Risk Management Skills: Information Security,Inf...

55min ago

Go to Apply

Information Systems Security Manager (ISSM) I

General Dynamics Information Technology • Bedford, MA • 57min ago

Management & Administrative

Req ID: RQ228596 Type of Requisition: Regular Clearance Level Must Be Able to Obtain: Top Secret SCI + Polygraph Public Trust/Other Required: None Job Family: Cyber and IT Risk Management Skills: Cybersecurity,Informatio...

57min ago

Go to Apply

Associate Customer Service Representative

Highmark Health • Boston, MA • 1h ago

Management & Administrative

Company : United Concordia Dental Job Description : JOB SUMMARY This Associate Customer Service Representative handles customer inquiries by telephone and/or email and/or in person from individual members, groups, provid...

1h ago

Go to Apply

Area Manager

Amazon • Revere, MA • 1h ago

Management & Administrative

Description External job description Our WW Operations network delivers millions of packages and smiles to Amazon customers every day. We are looking for motivated, customer-focused individuals who want to join our team...

1h ago

Go to Apply

Buying Analyst, Robotics Supply Chain - Procurement

Amazon • North Reading, MA • 3h ago

Management & Administrative

Description Is your passion inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work affects the bigger picture? Answer yes to any of these and you will fit rig...

3h ago

Go to Apply

Buying Analyst, Robotics Supply Chain - Procurement

Amazon • North Reading, MA • 3h ago

Management & Administrative

3h ago

Go to Apply

Subscribe to get new jobs sent directly to you
(no charge, cancel anytime)