Annotation Data Scientist, Evaluation Integrity (Siri)

Apple • Cambridge, MA • 2h ago

Weekly Hours: 40

Role Number: 200664186-1242

Summary

Play a part in the ongoing revolution in human-computer interaction. Siri is evolving - and the way we evaluate it has to evolve with it. Join the Evaluation Integrity team to help build the trusted quality signal behind every Siri release. Within the Siri evaluation organization, the Human Evaluation sub-team is responsible for answering the question: can we trust our evals? We do that by designing human-in-the-loop (HITL) annotation tasks that scrutinize every moving part of an agentic evaluation - the simulated user agent, the conversation it has with Siri, and the automated evaluators that grade the exchange. This role sits at the intersection of data science, human annotation engineering, and evaluation methodology, and is instrumental in turning human judgment into a rigorous, reproducible signal that directly informs pre-ship model and product decisions.

Description

As an Annotation Data Scientist on the Evaluation Integrity team, you will design and run HITL annotation projects that evaluate the quality and authenticity of agentic user personae, the validity of agent-to-agent conversations, and the reliability of LLM-as-judge and rule-based evaluators against Siri's product specifications. You will own annotation initiatives end-to-end; from rubric design and tooling, through annotator calibration, to data science analysis that turns annotator judgments into actionable signal for modeling, planning, and product teams.

Minimum Qualifications

Bachelor's or Master's degree in a quantitative or related field such as Data Science, Computer Science, Linguistics, Statistics, or Cognitive Science, or equivalent job-related experience.

3+ years of hands-on experience working with human-annotated datasets or human-in-the-loop evaluation methodologies for machine learning, natural language processing, or large language model systems.

3+ years of experience using Python for data processing, analysis, and prototyping, including experience with libraries such as pandas, Jupyter, and at least one data visualization library.

Experience designing, implementing, and communicating annotation schemas, rubrics, or ontologies for machine learning training or evaluation data.

Experience managing multiple concurrent dataset curation efforts, including scoping work, iterating on guidelines, coordinating with in-house or vendor annotators, and monitoring annotator performance metrics such as accuracy, throughput, and inter-annotator agreement.

Experience specifying or designing custom annotation tooling in collaboration with software engineers.

Preferred Qualifications

Experience evaluating LLM-powered or agentic systems, including familiarity with LLM-as-judge methodologies, rubric-based grading, or trajectory and tool-call evaluation.

Familiarity with statistical methods that address accuracy and variability in human annotation data, such as inter-annotator agreement, Cohen's or Fleiss' kappa, Krippendorff's alpha, or bootstrapping.

Data-querying experience with SQL, Spark, or similar, and comfort working with large, complex, real-world datasets.

Experience building pre-ship evaluation pipelines for conversational or assistant products.

Experience with prompt engineering, or with designing simulated user personae for agent evaluation.

Experience running annotation programs across multiple locales or at large scale.

Excellent written and verbal communication skills, with the ability to explain technical topics clearly to data scientists, engineers, annotators, and cross-functional partners.

Proven ability to collaborate effectively across functions and drive projects of varying sizes and scopes - knowing when to dive deep and when to delegate.

Apple

Go to Apply

Share this job

Similar Jobs

Data Scientist, Manufacturing Analytics

Ford Motor Company • Boston, MA • 2h ago

IT & Engineering

We made history and now we work to transform the future - for our customers, our communities and our families. You'll see your work on the road every day, helping people move freely and pursue their dreams. At Ford, you...

2h ago

Go to Apply

Senior Data Scientist - Fleet Analytics

The Hertz Corporation • Boston, MA • 2h ago

IT & Engineering

Senior Data Scientist The Company Hertz is one of the largest global mobility organizations and is currently undergoing a significant digital transformation. Hertz is committed to becoming an essential component of the m...

2h ago

Go to Apply

2026 Fulltime- Power and Analog Design Electrical Engineer II- Onsite

RTX Corporation • Marlborough, MA • 3h ago

IT & Engineering

Date Posted: 2026-05-19 Country: United States of America Location: US-MA-MARLBOROUGH-MA1 ~ 1001 Boston Post Rd ~ BLDG 1 Position Role Type: Onsite U.S. Citizen, U.S. Person, or Immigration Status Requirements: Active an...

3h ago

Go to Apply

2026 Full-time - Packaging Engineer I - Onsite (MA)

RTX Corporation • Andover, MA • 4h ago

IT & Engineering

Date Posted: 2026-05-19 Country: United States of America Location: US-MA-ANDOVER-AN1 ~ 350 Lowell St ~ AN1 ESSEX BLDG Position Role Type: Onsite U.S. Citizen, U.S. Person, or Immigration Status Requirements: The ability...

4h ago

Go to Apply

Principal Electrical Engineer

Medtronic • Billerica, MA • 5h ago

IT & Engineering

We anticipate the application window for this opening will close on - 5 Jun 2026 At Medtronic you can begin a life-long career of exploration and innovation, while helping champion healthcare access and equity for all. Y...

5h ago

Go to Apply

Sr. Software Engineer - Java

Validity • Boston, MA • 7h ago

IT & Engineering

About the Role Validity is looking for an experienced Sr. Software Engineer to work on our data management team. As a member of the data management team, you will work closely with engineers and other stakeholders to des...

7h ago

Go to Apply

Software Engineering Technical Leader

Cisco • Boston, MA • 7h ago

IT & Engineering

The application window is expected to close on: 06/26/2026 Job posting may be removed earlier if the position is filled or if a sufficient number of applications are received . Meet the Team: Join our team to design and...

7h ago

Go to Apply

Manufacturing Engineer

Fairbanks Morse Defense • Walpole, MA • 7h ago

IT & Engineering

Manufacturing Engineer - CNC Programming Level: Manufacturing Engineer / Senior / Technical Lead (based on experience and demonstrated capability) Department: Manufacturing Engineering Reports To: Manufacturing Engineeri...

7h ago

Go to Apply

Sr Manufacturing Engineer - 2nd Shift

RTX Corporation • Andover, MA • 8h ago

IT & Engineering

Date Posted: 2026-05-18 Country: United States of America Location: US-MA-ANDOVER-AN2 ~ 352 Lowell St ~ AN2 HAMPSHIRE BLDG Position Role Type: Onsite U.S. Citizen, U.S. Person, or Immigration Status Requirements: The abi...

8h ago

Go to Apply

Supply Technician

Veterans Affairs, Veterans Health Administration • Chelmsford, MA • 17h ago

IT & Engineering

Summary The position will be located at the VA Northeast (NE) Consolidated Mail Outpatient Pharmacy (CMOP), in Chelmsford, Massachusetts. The employee performs property accountability duties, receiving and inventory rela...

17h ago

Go to Apply

Subscribe to get new jobs sent directly to you
(no charge, cancel anytime)