Job Description

Job Responsibilities:

• Aggregate cross-functional requests and translate requests into actionable, high quality, datasets via synthetic data collections and on live user traffic.
• Deliver analytics, statistics, and model performance results on datasets.
• Design and conduct experiments for evaluating rater quality, inter-rater reliability; write summary reports based on findings and speak on those findings in meetings.
• Develop manual and automated processes for multiple concurrent projects ensuring high-quality labeled data.
• Create and perfect LLM Quality grading queues and LLM related component quality error attribution grading queues.
• Build datasets and write guidelines in rapid fashion for new and emergent LLM data rating, creation, and annotation needs.
• Analyze system metrics such as factuality, brevity, coherence, as well as subcomponent performance, summarize findings, and communicate results.

Skills:

• Basic Python, basic SQL with strong desire to grow required for this position. Intermediate level preferred.
• Basic knowledge of linguistics (some of Phonetics, Syntax, Semantics, Dialectology) required.
• Ability to analyze numerical rating results and draw conclusions based on data.
• Basic stats / math for data analysis.
• Written and oral communication skills necessary for communicating on project milestones and for reporting findings

Education/Experience:

• Bachelor’s degree in Linguistics, Computational Linguistics, Speech Science, or related field OR graduate degree in linguistics OR industry similar experience required.
• Master’s degree or beyond preferred OR equivalent industry experience.
• Additional learning via Coursera, Udemy, or other online resources is great too.

Job Tags

Similar Jobs