Senior Software Engineer, AI Resiliency

NVIDIA • Redmond, WA • 2h ago

We are now looking for a Senior Software Engineer for AI Resiliency!

At NVIDIA, we are pushing the boundaries of what's possible in AI. We are currently seeking a Senior Software Engineer to lead the development of AI software resiliency for the most powerful AI supercomputers in the world. As a member of our AI Software Resiliency team, you will play a pivotal role in defining and implementing critical resiliency features for AI supercomputers at a scale of 100,000+ GPUs. Your expertise will be crucial in driving down cluster downtime towards zero, ensuring that our AI systems remain robust and reliable at all times.

What You'll Be Doing:

Develop AI Software Resiliency Features: Implement and optimize software features that improve AI system reliability at a massive scale, such as fast checkpoint-recovery, error detection, error isolation, and straggler/hang detection.

Hands-On Coding & Optimization: Contribute to large-scale distributed systems with high-quality, production-level C++ and Python code. Enhance performance for AI workloads running on thousands of GPUs.

Fault Tolerance & Debugging: Work on AI system error handling, implementing techniques to detect silent data corruption (SDC) and other failure scenarios. Assist in developing monitoring tools for proactive failure mitigation.

Collaborate Across Teams: Work closely with senior engineers, AI researchers, and hardware/software teams to integrate resiliency features into AI frameworks like PyTorch and JAX/XLA.

Testing & Automation: Develop and implement tests to ensure robustness, scalability, and efficiency of resiliency mechanisms. Contribute to CI/CD pipelines to automate validation of AI workloads.

Support Production Deployments: Assist in debugging and performance tuning large-scale AI workloads in cloud and HPC environments, ensuring seamless operation of AI training and inference workloads.

What We Need to See:

You've achieved a Bachelor's, Master's or PhD in Computer Science, Electrical Engineering, or a related field, or equivalent experience.

Proficiency in C++ and Python , with experience in writing efficient, high-performance code.

6+ years of relevant experience

Strong understanding of distributed systems concepts , parallel programming, and fault tolerance in large-scale computing environments.

Familiarity with AI frameworks such as PyTorch, JAX/XLA, TensorFlow, or similar.

Experience with debugging and profiling tools (e.g., gdb, perf, valgrind, NVIDIA Nsight).

Excellent problem-solving skills and ability to work in a fast-paced, highly collaborative environment.

Ways to Stand Out From the Crowd:

Hands-on experience in training models or working with model training teams .

Hands-on experience with CUDA, NCCL, or MPI for GPU-accelerated computing, especially at extreme-scale .

Knowledge of checkpointing strategies, error mitigation, or fault-tolerant computing in AI training.

Experience working with large-scale AI clusters, HPC environments, or cloud-based AI workloads .

Strong systems programming skills and experience with low-level performance tuning.

As part of the AI Resiliency team at NVIDIA, you'll work alongside world-class engineers solving some of the hardest challenges in AI infrastructure. You'll have the opportunity to contribute directly to making AI training and inference more reliable, scalable, and efficient. If you're passionate about AI, distributed systems, and high-performance computing, we want to hear from you!

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD.

You will also be eligible for equity and benefits (https://www.nvidia.com/en-us/benefits/) .

Applications for this job will be accepted at least until May 16, 2026.

This posting is for an existing vacancy.

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

NVIDIA

Go to Apply

Share this job

Similar Jobs

Sr. Software Engineer

Insight Global • Bellevue, WA • 2h ago

IT & Engineering

Job Description This role is an opportunity to join a global fintech insurance company as a software engineer supporting the retirement vertical for enterprise clients. The position focuses on building and enhancing cust...

2h ago

Go to Apply

Software Engineering IC4

Microsoft Corporation • Redmond, WA • 2h ago

IT & Engineering

Overview Now more than ever, we live in an age where AI and Cloud Services are empowering people around the world to achieve more. Behind these innovative services exists a critical infrastructure that is bringing them t...

2h ago

Go to Apply

Senior Software Engineer, Foundry Agents - CoreAI

Microsoft Corporation • Redmond, WA • 2h ago

IT & Engineering

Overview Microsoft Foundry is building the foundational platforms that power the next generation of intelligent agents and generative AI systems. Within CoreAI, the Foundry Agents organization is responsible for key owne...

2h ago

Go to Apply

Software Engineer, AI Security

Amazon • Seattle, WA • 2h ago

IT & Engineering

Description We are building a first-of-its-kind vulnerability auditing and review platform for LLM and foundational model integrated software - including components like foundational models, MCP servers, agents, and agen...

2h ago

Go to Apply

Software Engineer, Audio Embedded Device

Meta • Redmond, WA • 2h ago

IT & Engineering

Summary: At Meta, we're building products that improving the way people connect, work, and play. Our audio teams are at the forefront of this innovation, developing advanced technologies that redefine the way we experien...

2h ago

Go to Apply

Manufacturing Engineer (2026-067)

Astronics • Kirkland, WA • 3h ago

IT & Engineering

Manufacturing Engineer (2026-067) Astronics Subsidiary Astronics AES Location Redmond, WA Description Astronics Advanced Electronic Systems (AES) is seeking a Manufacturing Engineer to join our cohesive and diverse team...

3h ago

Go to Apply

Senior Specialist, Manufacturing Engineer

L3Harris • Redmond, WA • 3h ago

IT & Engineering

L3Harris is dedicated to recruiting and developing high-performing talent who are passionate about what they do. Our employees are unified in a shared dedication to our customers' mission and quest for professional growt...

3h ago

Go to Apply

Senior Data Scientist , Alexa AI Aurora

Amazon • Bellevue, WA • 20h ago

IT & Engineering

Description The Alexa AI AURORA organization is seeking a passionate, talented, and resourceful Senior Data Scientist to define and solve complex, ambiguous problems in state-of-the-art conversational AI. You will lead l...

20h ago

Go to Apply

Optical Engineer II

Bucher & Christian Consulting, Inc. dba BCforward (BCF) • Redmond, WA • 20h ago

IT & Engineering

Job Title: Optical Engineer II Location: Redmond, WA Duration: Contract - 9 months Pay Range: $65 Up to $68/hr Job ID: 373405 About BCforward BCforward is a leading global IT consulting and workforce solutions firm provi...

20h ago

Go to Apply

Mechanic - Component Electrical Shop

Aviation Technical Services • Everett, WA • 23h ago

IT & Engineering

$5,000 Sign-On Bonus Available Relocation Assistance Available We are Trusted Partners | Supporting Flight. Our more than 50 years of success is the result of our employees who have dedicated themselves to providing unma...

23h ago

Go to Apply

Subscribe to get new jobs sent directly to you
(no charge, cancel anytime)