Senior Software Engineer - Inference as a Service

NVIDIA • Santa Clara, CA • 8h ago

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It's a unique legacy of innovation that's fueled by great technology-and amazing people. Today, we're tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what's never been done before takes vision, innovation, and the world's best talent. As an NVIDIAN, you'll be immersed in a diverse, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world.

We are seeking a Senior Software Engineer to join our Software Infrastructure Team in Santa Clara, CA. This team is at the heart of the NVIDIA AI Factory initiative, building and maintaining the core infrastructure that powers our closed and open source AI models. In this role, you will be a key contributor to our Inference as a Service platform, developing the systems that manage GPU resources, ensure service stability, and deliver high-performance, low-latency inference at a massive scale.

What you'll be doing:

Contribute to the design and development of a scalable, robust, and reliable platform for serving AI models for inference as a service.

Develop and implement systems for dynamic GPU resource management, autoscaling, and efficient scheduling of inference workloads.

Build and maintain the core infrastructure, including load balancing and rate limiting, to ensure the stability and high availability of inference services.

Implement APIs for model deployment, monitoring, and management for a seamless user experience.

Collaborate with engineering teams to integrate deployment, monitoring, and performance telemetry into our CI/CD pipelines.

Build tools and frameworks for real-time observability, performance profiling, and debugging of inference services.

Work with architects to define and implement best practices for long-term platform evolution.

Contribute to NVIDIA's AI Factory initiative by building a foundational platform that supports model serving needs.

What we need to see:

BS, MS, or PhD in Computer Science, Electrical/Computer Engineering, Physics, Mathematics, other Engineering or related fields (or equivalent experience)

12+ years of software engineering experience with expertise in distributed systems or large-scale backend infrastructure.

Strong programming skills in Python, Go, or C++ with a track record of building production-grade, highly available systems.

Proven experience with container orchestration technologies like Kubernetes.

A strong understanding of system architecture for high-performance, low-latency API services.

Experience in designing, implementing, and optimizing systems for GPU resource management.

Familiarity with modern observability tools (e.g., DataDog, Prometheus, Grafana, OpenTelemetry).

Demonstrated experience with deployment strategies and CI/CD pipelines.

Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.

Ways to stand out from the crowd:

Experience with specialized inference serving frameworks

Open-source contributions to projects in the AI/ML, distributed systems, or infrastructure space.

Hands-on experience with performance optimization techniques for AI models, such as quantization or model compression.

Expertise in building platforms that support a wide variety of AI model architectures.

Strong understanding of the full lifecycle of an AI model, from training to deployment and serving.

NVIDIA is widely considered to be one of the technology world's most desirable employers. We have some of the most forward-thinking and passionate people on the planet working for us. If you're creative and autonomous, we want to hear from you!

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 200,000 USD - 322,000 USD for Level 5, and 248,000 USD - 391,000 USD for Level 6.

You will also be eligible for equity and benefits (https://www.nvidia.com/en-us/benefits/) .

Applications for this job will be accepted at least until August 26, 2025.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

NVIDIA

Go to Apply

Share this job

Similar Jobs

Product Marketing Manager - Microsoft Sentinel

Microsoft Corporation • Mountain View, CA • 7min ago

Management & Administrative

Microsoft's goal is to enable every individual and organization worldwide to achieve greater productivity. The company's culture emphasizes a growth mindset, striving for excellence, and motivating teams and leaders to p...

7min ago

Go to Apply

Product Marketing Manager, Apple Ads

Apple • Cupertino, CA • 9min ago

Management & Administrative

Product Marketing Manager, Apple Ads Cupertino, California, United States Marketing Summary Posted: Aug 22, 2025 Weekly Hours: 40 Role Number: 200600886-0836 At Apple, we focus deeply on our customers' experience. Apple...

9min ago

Go to Apply

Outside Sales Representative ( B2B ) / Territory Account Manager

System One • Fremont, CA • 14min ago

Management & Administrative

Job Title: Outside Sales Representative ( B2B ) / Territory Account Manager Location: Fremont, California ( East Bay area ) Type: Direct Hire For immediate consideration, please connect with me on LinkedIn at https://www...

14min ago

Go to Apply

Oracle Cloud Supply Chain Senior Consultant

Deloitte • San Jose, CA • 1h ago

Management & Administrative

Today's chief financial officers (CFOs) and financial executives are being asked to improve business performance and shareholder value, along with operational effectiveness and efficiency. Deloitte Oracle Cloud Supply Ch...

1h ago

Go to Apply

WW Ops BPR Sr.Project Manager, Supply Chain Solution Architect

Apple • Sunnyvale, CA • 1h ago

Management & Administrative

WW Ops BPR Sr.Project Manager, Supply Chain Solution Architect Sunnyvale, California, United States Operations and Supply Chain Summary Posted: Aug 22, 2025 Weekly Hours: 40 Role Number: 200608171-3956 Imagine what you c...

1h ago

Go to Apply

Data Entry Associate (Automotive)

Aston Carter • Newark, CA • 1h ago

Management & Administrative

Description Job Description: To perform vision-based data annotation for a machine learning project as well as verifying annotation quality of incoming data. This job requires good communication skills critical thinking...

1h ago

Go to Apply

Data Entry Associate

Aston Carter • Newark, CA • 1h ago

Management & Administrative

1h ago

Go to Apply

Technical Account Manager

Zoom • San Jose, CA • 7h ago

Management & Administrative

What you can expect You will specifically support our customers within Zoom's Premier Support and Technical Account Management team. This function discovers, develops and maintains strategic partnerships with select ente...

7h ago

Go to Apply

Commercial Sales Manager

AutoZone, Inc. • East Palo Alto, CA • 7h ago

Management & Administrative

The Commercial Sales Manager is responsible for driving the commercial sales within their AutoZone location. The CSM develops and maintains positive relationships with commercial customers by managing the acquisition and...

7h ago

Go to Apply

Administrative Assistant

Robert Half • Santa Clara, CA • 7h ago

Management & Administrative

Description About the Role: We are seeking a motivated and detail-oriented Administrative Assistant to join our team. In this role, you will provide vital support to ensure smooth day-to-day operations. You'll be respons...

7h ago

Go to Apply

Subscribe to get new jobs sent directly to you
(no charge, cancel anytime)

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.