Kablio AI
AI Turbo Recruiter
Want to hear how I work? Hit play.Find roles with Kablio AI to help build and power the world.Kablio AI helps you secure roles in construction, clean energy, facilities management, engineering, architecture, sustainability, environment and other physical world sectors.
Get hired, get rewarded!
Land a job through Kablio and earn a 5% salary bonus.
Exclusive benefits
5%Bonus
QA Engineer III
realpage india
Design and execute QA evaluation for LLMs, building automated tests and tools.
9d ago
Intermediate (4-7 years), Experienced (8-12 years)
Full Time
Serilingampalli, Telangana, India
Hybrid
Company Size
1500+ globally Employees
Service Specialisms
serviceSpecialismFormat1
serviceSpecialismFormat2
Sector Specialisms
sectorSpecialismFormat1
sectorSpecialismFormat2
Role
What you would be doing
pipeline automation
llm evaluation
test harnesses
benchmarking
tool development
model analysis
- Act as a quality advocate for AI-driven features across the product lifecycle.
- Develop automated pipelines for regression testing, prompt evaluation, and model comparison.
- Collaborate with product managers, researchers, and engineers to define evaluation metrics aligned with business and ethical goals.
- Mentor junior team members in AI evaluation best practices and tooling.
- Design and execute evaluation strategies for LLM-based systems using LangSmith, OpenAI tools, and custom frameworks.
- Build reusable test harnesses and datasets for benchmarking AI capabilities across tasks and domains.
- Stay current with advancements in LLM evaluation, interpretability, and responsible AI.
- Contribute to the development of internal tools for prompt engineering, evaluation visualization, and feedback loops.
- Analyze model outputs for correctness, bias, safety, and hallucination using both qualitative and quantitative methods.
What you bring
python
openai apis
playwright
ci/cd
github
jira
- Strong Python skills, especially in building test automation and data analysis scripts.
- Hands-on experience with OpenAI APIs (e.g., GPT-4, function calling, Assistants API).
- Excellent communication and collaboration skills, with the ability to explain complex AI behaviors to diverse stakeholders.
- Experience with prompt engineering, LLM fine-tuning, or RAG pipelines is a plus.
- Proficiency in using tools like Jupyter, Pandas, and/or SQL for evaluation analytics.
- Proficient in the use of various automation frameworks like Playwright (preferred), Protractor, Rest Assured, karate etc.
- Minimum 6-8 years of experience into QA and Automation
- Bachelor's or master's degree in computer science or a related technical field.
- Experience with version control (GitHub), CI/CD pipelines, and test management tools (e.g., JIRA, Azure Test Plans).
- Familiarity with OpenAI Evals and LangSmith for prompt orchestration and evaluation.
- Basic level understanding of software development lifecycle and quality best practices and tools
- Understanding of ethical AI principles, model alignment, and safety concerns.
- Good knowledge on Automation Framework Development
Information not given or found
Training + DevelopmentInformation not given or found
Interview process
Information not given or found
Visa SponsorshipInformation not given or found
Security clearanceInformation not given or found
Company
Overview
Nope
Culture + Values- Bring your expertise in software development
- Bring your creativity and problem-solving skills
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam tortor lacus, ultrices eu purus sit amet, ullamcorper maximus lacus. Suspendisse in iaculis massa, in ullamcorper quam. Interdum et amet.
- Bring your expertise in software development
- Bring your creativity and problem-solving skills
- Bring your expertise in software development
- Bring your creativity and problem-solving skills
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam tortor lacus, ultrices eu purus sit amet, ullamcorper maximus lacus. Suspendisse in iaculis massa, in ullamcorper quam. Interdum et amet.
- Bring your expertise in software development
- Bring your creativity and problem-solving skills
Articles
Kablio AIIf you're someone who helps build and power the world (or dreams to), Kablio AI is your pocket-sized recruiter that gets you hired.
Copyright © 2025 Kablio