AI decision systems are rapidly evolving. Modern platforms must combine deterministic logic, machine-learning models, advanced AI agents, and human expertise to make high-quality decisions at scale.
Evaluating these systems requires more than static benchmarks and traditional review queues. It requires platforms where automated evaluators, AI agents, production signals, and human experts work together continuously.
Microsoft Ads is looking for a hands-on Principal Software Engineer to build the next generation of agentic evaluation, audit, and human-review platforms.
You will create systems that help teams understand how decisions were made, identify where they fail, investigate supporting evidence, prioritize uncertain or high-impact cases, and turn review outcomes into measurable improvements.
The platform will connect:
production decisions → automated evaluation → agentic investigation → human judgment → system improvement
Responsibilities
Architect and build scalable platforms for agentic evaluation, production audits, and human-in-the-loop review.
Develop workflows in which AI agents gather evidence, analyze cases, compare decisions, and assist reviewers while preserving human oversight and accountability.
Build reviewer experiences that present the right evidence, explanations, uncertainty, and recommended actions at the right time.
Design distributed orchestration systems for long-running workflows involving models, agents, tools, automated checks, and human decisions.
Provide technical leadership across engineering, applied science, product, design, operations, and policy teams.
Required Qualifications
Don't want to miss the next one?
Subscribe to daily email alerts for roles matching your interests.
Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field, with 8+ years of relevant industry experience.
Strong production-coding skills in at least one of C++, C#, Java, Python, or TypeScript.
Demonstrated experience designing, building, and operating large-scale distributed services or software platforms.
Strong system-design skills across APIs, workflow orchestration, event-driven systems, storage, concurrency, fault tolerance, and service reliability.
Experience building AI evaluation, agent orchestration, human-in-the-loop, audit, experimentation, or LLMOps platforms.
Experience owning substantial systems from initial architecture through rollout, adoption, and production operations.
Preferred Qualifications
Background in advertising, trust and safety, fraud, responsible AI, search, recommendations, or another high-scale complex decision domain.
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
Similar roles you might like
More openings like this one — take a look before you go.
