Chinese LLM Evaluator

Chinese LLM Evaluator - Project Spearmint

About the Project:

Join Project Spearmint, a multilingual AI response evaluation project focused on refining large language model (LLM) outputs. We are seeking native-level Chinese speakers to evaluate model replies for either Tone or Fluency. Your contributions will directly impact the development of more accurate and natural AI interactions.

Responsibilities:

• Evaluate short, pre-segmented datasets of model-generated replies in Chinese.
• Assess the quality, correctness, and naturalness of responses based on specific criteria.
• Rate model replies using a five-point scale, providing brief rationales for extreme ratings.
• Analyze user prompts and identify potential issues related to Tone or Fluency.

Project Breakdown:

• Batch 1 (Tone): Determine if replies are helpful, insightful, engaging, and fair. Flag instances of formality mismatches, condescension, bias, or other tonal issues.
• Batch 2 (Fluency): Assess grammatical accuracy, clarity, coherence, and natural flow of the responses.

Qualifications:

• Native-level fluency in Chinese.
• Strong English comprehension skills.
• Ability to provide clear and concise feedback.
• Attention to detail and commitment to quality.

This is a project-based opportunity with CrowdGen, offering flexible work arrangements as an Independent Contractor.

Qualifications:

• Native-level fluency in Chinese: Demonstrated proficiency in written and spoken Chinese is essential.
• Strong English comprehension: Ability to understand and follow instructions written in English is required.
• Attention to detail: Meticulousness and accuracy in evaluating model responses are crucial.
• Analytical skills: Ability to critically assess the quality, correctness, and naturalness of language.
• Computer proficiency: Comfortable using computers and navigating online platforms.
• Time management: Ability to manage time effectively and meet project deadlines.

Evaluation Tasks

• Assess the quality, correctness, and naturalness of large language model (LLM) responses in Chinese.
• Evaluate model replies based on either Tone or Fluency, as assigned.
• Read user prompts and two model responses for each evaluation task.
• Rate each model response using a five-point scale according to specific quality dimensions.
• Provide concise explanations for ratings of 1 or 5.

Project Understanding

• Adhere to established evaluation guidelines and criteria for Tone and Fluency.
• Understand and apply the nuances of Chinese language and culture in evaluating responses.
• Maintain consistent and accurate ratings throughout the project.

Applicants interested in the Chinese LLM Evaluator role at CrowdGen by Appen will first submit their applications through the job portal. CrowdGen will then review applications, focusing on fluency in Chinese and English comprehension as outlined in the job description. Shortlisted candidates will be invited to complete a brief online assessment testing their language skills and understanding of the evaluation criteria. Successful candidates will then be contacted to set up an account with CrowdGen and complete the onboarding process. Finally, they will be invited to participate in a trial project to demonstrate their ability to evaluate LLM outputs effectively.

CrowdGen, powered by Appen, is a global leader in providing high-quality data annotation and evaluation services for artificial intelligence (AI) development. We connect businesses with a vast network of skilled individuals worldwide, enabling them to build and refine AI models that are accurate, reliable, and culturally relevant. Through our innovative platform, CrowdGen empowers individuals to contribute to the advancement of AI technology while earning income from the comfort of their homes. We are committed to fostering a diverse and inclusive community of contributors, ensuring that AI development reflects the richness and complexity of the world we live in.