If you love building real Artificial Intelligence (AI) systems that solve real customer problems and are passionate about measuring, improving, and scaling AI quality, this role is for you. The Applied AI team within Autodesk's Architecture, Engineering, and Construction (AEC) organization builds cloud-native AI solutions that make industry workflows smarter, faster, and more connected.
This role is primarily focused on AI evaluation and quality engineering for Generative AI systems. You will establish evaluation standards, build automated validation frameworks, and drive observability practices that ensure AI systems are reliable, trustworthy, and production-ready. In addition to AI evaluation, you will contribute to the design and development of AI-powered applications, including Retrieval-Augmented Generation (RAG) systems, agentic workflows, and Large Language Model (LLM)-based solutions deployed at scale.
Responsibilities
Lead AI evaluation and quality strategy for Generative AI and Machine Learning (ML)-powered systems
Design and implement scalable evaluation frameworks for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, agentic workflows, and traditional Machine Learning (ML) models
Fine-tune, evaluate, benchmark, and deploy Large Language Models (LLMs) in production environments
Implement evaluation and observability standards, including regression testing, monitoring, tracing, feedback loops, and quality measurement for AI systems
Define and track AI quality metrics, including accuracy, robustness, bias, drift, grounding, latency, safety, and hallucination rates
Build automated evaluation pipelines and integrate them into Continuous Integration and Continuous Deployment (CI/CD) workflows for continuous model validation
Don't want to miss the next one?
Subscribe to daily email alerts for roles matching your interests.
Evaluate and adopt AI testing frameworks, benchmark methodologies, and emerging evaluation tools
Build scalable Retrieval-Augmented Generation (RAG) pipelines using structured and unstructured enterprise data sources
Integrate AI agents with Autodesk platforms and enterprise services using Model Context Protocol (MCP)-based integrations or equivalent architectures
Partner with Product Managers, Architects, Software Engineers, Experience Designers, and Data Scientists to deliver AI-powered experiences
Ensure AI systems align with Responsible AI principles and enterprise governance requirements
Contribute to reusable AI platform capabilities, Software Development Kits (SDKs), orchestration patterns, and engineering best practices
Leverage AI coding assistants to accelerate feature delivery while maintaining rigorous validation through testing, Continuous Integration and Continuous Deployment (CI/CD), code reviews, and security best practices
Minimum Qualifications
Bachelor's or Master's degree in Computer Science, Machine Learning, Data Science, Engineering, or equivalent practical experience
5+ years of hands-on experience delivering Artificial Intelligence (AI) and Machine Learning (ML)-powered systems into production
Experience designing evaluation frameworks, automated testing solutions, benchmarking systems, or model validation pipelines
Strong understanding of Large Language Model (LLM) evaluation, model quality measurement, and AI observability concepts
Strong proficiency in Python with experience building automation frameworks and testing infrastructure
Experience deploying and operating AI applications in production, including monitoring, observability, and continuous improvement
Experience building cloud-native applications on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
Excellent technical communication and collaboration skills
Demonstrated experience using AI coding tools to accelerate the delivery of production systems
Preferred Qualifications
Hands-on experience with AI evaluation frameworks such as LangSmith, DeepEval, TruLens, Promptfoo, OpenAI Evals, or similar tools
Experience evaluating Retrieval-Augmented Generation (RAG) systems, agentic workflows, AI assistants, and prompt-based applications
Familiarity with Responsible AI, AI governance, model risk assessment, and AI observability platforms
Experience deploying cloud-native AI solutions on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
Familiarity with deep learning architectures such as Transformers and modern Machine Learning (ML) frameworks, including PyTorch, Lightning, Ray, or equivalent technologies
Experience with enterprise knowledge systems, semantic search, vector search, or contextual retrieval systems
Experience with emerging agentic AI frameworks such as LangGraph, CrewAI, OpenAI Software Development Kits (SDKs), Claude Software Development Kits (SDKs), Model Context Protocol (MCP), Agent-to-Agent (A2A), or equivalent technologies
Contributions to open-source AI projects, technical blogs, conference presentations, or published research are an advantage
The Ideal Candidate
Demonstrates a strong understanding of Artificial Intelligence (AI), Machine Learning (ML), and Generative AI concepts and applies industry best practices to solve complex technical challenges
Uses analytical thinking, sound judgment, and data-driven decision-making to evaluate model performance and recommend effective solutions
Connects technical implementation with broader product, customer, and business objectives
Applies creativity and innovation to improve AI quality, evaluation methodologies, and engineering practices
Leads projects or key workstreams while contributing to the successful delivery of broader initiatives
Takes ownership of improving engineering processes, quality standards, and operational excellence
Builds strong working relationships with senior technical leaders, peers, and cross-functional stakeholders
Works independently while seeking guidance on highly complex or ambiguous challenges when appropriate
Demonstrates excellent collaboration and communication skills when working across globally distributed teams
Shares knowledge, mentors less experienced engineers, and contributes to a culture of continuous learning
Maintains a passion for learning emerging AI technologies, frameworks, and best practices while continuously improving technical expertise
LI-SJ1
Learn More
About Autodesk
Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.
We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.
When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!
Salary transparency Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.
Belonging We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging
Are you an existing contractor or consultant with Autodesk?
Please search for open jobs and apply internally (not on this external site).
Similar roles you might like
More openings like this one — take a look before you go.
