LLM Evaluator – Model Response Analyst (Remote)

About the role

We are looking for a detail‑oriented LLM Evaluator (Model Response Analyst) to assess and improve the performance of large language models. Working remotely with a global team, you will analyze AI‑generated text for factual accuracy, coherence, safety, bias and alignment with defined guidelines.

Key responsibilities

• Evaluate and rank model‑generated responses using complex rubrics covering factuality, coherence, safety, instruction‑following and creativity.
• Compare multiple model outputs for the same prompt, select the preferred response and justify the choice.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Design adversarial prompts to expose biased, harmful or insecure model behavior and help patch safety gaps.
• Collaborate with QA to refine evaluation guidelines for ambiguous or edge‑case scenarios.
• Participate in cross‑checking sessions to calibrate scoring standards and ensure inter‑rater reliability.
• Investigate underperforming model outputs, hypothesize root causes and flag novel behaviors to the research team.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP QA or related fields.
• Bachelor’s degree in Computer Science or a related discipline.
• Strong understanding of prompt engineering and the ability to explain why a model output is good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection.
• Proven ability to monitor and improve consistency among evaluation teams.
• Familiarity with IAA scoring, calibration sessions and A/B testing concepts applied to AI.

Required skills

• Prompt engineering
• RLHF data collection
• Dataset annotation and cleaning
• A/B testing for AI models
• Experiment design and evaluation
• Data analysis