Technical Researcher, LLM Evaluation
The person who runs and documents our model testing so every result can be reproduced.
📍 US Eastern Time mornings - Remote, Open globally
Individual
Employment type — Contract - Independent Expert
As Eticas expands its work on AI in Education in Europe and the US, we are looking to engage independent experts with different levels of expertise to make sure we have everyone at the table and different perspectives to assess our work. Engagements are remote, hourly-based and paid, with an initial defined scope.
Our work is focused on understanding how we can measure AI impacts on learning, and whether we have the tools to measure the right things and support the students who need it. We are also working to build standards so we have homogenized, comparable datasets to understand how learning is changing over time, what works and what doesn't, and for whom.
Working with schools, education authorities and edtech developers, we are testing AI scoring of complex student skills (written reasoning, collaborative problem solving, spoken reasoning) against trained teacher raters, and producing metrics and evidence that show when AI in learning can be trusted, and when it can't.
Our focus is on how AI performs on real student work, judged by real teachers, under real classroom conditions. We get as close as possible to the lived experience of learning and measure it, with the same discipline Eticas has applied for over a decade to AI in hiring, health and public services, among others.
This open call is part of our commitment to openness, inclusion and transparency, so that more voices can join this crucial work.
About the role
Prepare evaluation datasets of student work and teacher ratings, including de-identification and format checks.
Run LLM scoring tests across models, configurations, prompts and matched variants, with repeated runs to measure drift and stability.
Log every model version, configuration and run.
Compute agreement between model and teacher scores and basic reliability statistics.
Write reproducibility and provenance documentation others can audit.
Works under our senior technical lead.
Who we are looking for / Competencies
Strong Python; experience calling LLM APIs and building evaluation pipelines or test harnesses.
Hands-on NLP or LLM evaluation experience (LLM-as-judge, automated scoring, benchmark construction or similar).
Disciplined version control and documentation (Git, experiment logging).
Basic statistics; interest in measurement or education is a plus.
Not a fit: generalist data science or software QA without NLP or LLM evaluation work.
Dedication
About 130 hours, mainly December 2026 to March 2027, with light set-up work in November.
Compensation
Hourly, paid monthly as an independent contractor. Please state your expected hourly rate in your note.
How to apply
Send us an email to: careers@eticas.ai, and include a CV and a short separate note, one page at most, highlighting how your experience fits with the role and competencies required, references to relevant work and confirmation of availability in the times described in each position. Positions are assessed as they come.