Scan your resume against ATS criteria for this Problem Design Engineer role at Widgetfactory.
We create high-quality coding problems for training AI models — drawn from real production repositories, not synthetic puzzles. Your role is to ensure these problems are fair, clear, and technically sound.
| Remote | $40/hr | 0-25 hrs/week
We’re looking for engineers who can trace the intent behind a code change, understand what changes were made and why, and express that clearly in natural language.
This is judgment work, not mechanical testing. You'll make philosophical decisions about fairness and clarity in technical contexts.
across unfamiliar repositories and tech stacks
for your assessments (2-3 sentences per decision)
Flag coherence issues: Inconsistent artifacts, unclear requirements, missing context
Time commitment: 1-4 hours per problem, fully async — work whenever fits your schedule.
in CS/computational fields seeking long-term flexible income
who care deeply about code quality
in CS, ECE, Computational Biology, or related technical programs
seeking intellectually interesting part-time work
Have strong software engineering fundamentals
Can work independently with minimal supervision
Write clearly and precisely
Enjoy making judgment calls in ambiguous situations
Stay focused during 1-3 hour review sessions
experience (internships, research projects, or professional work)
Responsive, independent, and attentive to detail
0–25 hours per week, flexible
for all review work
— answer 4 questions about your technical background and work style
— demonstrate your assessment skills on a real coding problem
— join our active problem design engineering community and start reviewing
Yes — many of our top contributors are PhD students. The work is fully async and fits around research schedules.
No. Work whenever you want. Some problem design engineers work weekends, others weekday evenings.
That's expected. You're assessing problem fairness and clarity, not implementing solutions. Familiarity with general software engineering principles is what matters.
If you pass the video interview and sample review, you can start within 1-2 weeks.
This is a long-term contract role, not a pipeline to full-time positions. We value contributors who excel at this specific type of work.
The Widget Factory is the quality assurance division of Idler (YC S25), where we ensure coding problems meet research-grade standards for AI training. Our mission is to build the most reliable QA system for AI coding benchmarks.
vision
How to apply for Problem Design Engineer at Widgetfactory?
Click the "Apply on Company Website" button on this page to submit your application directly on the employer's official portal.
What is the salary for this role?
The salary for this role is USD 40 - 40/hr per annum.
What experience is required?
1+ years of experience is required.
Is this position still open?
Yes, currently active and accepting applications.
Explore related active roles in engineering_design
Problem Design Engineer
Widgetfactory · United States
