AI Product Analyst - AI Receptionist Team

At MEDvidi, we are scaling AI-powered systems across critical parts of our product and operations.

As these systems become more autonomous and deeply integrated into user workflows, understanding how they behave in production becomes a core business challenge.

We are hiring a Product Analyst to help us evaluate, monitor, and improve AI agents and LLM-powered workflows operating at scale.

This role is focused on analyzing real AI system behavior in production environments:

• understanding product and operational impact,
• identifying failure patterns on generational tasks,
• measuring quality and reliability of outputs,
• helping teams make accurate decisions under ambiguity.

Responsibilities

• Analyze production behavior of AI agents, automation systems, and LLM-powered workflows,
• Investigate hallucinations, inconsistencies, failure modes, and quality degradation across automated user journeys,
• Work directly with logs, conversations, raw system outputs, and operational datasets to identify hidden patterns and actionable insights,
• Design evaluation approaches and analytical frameworks for measuring AI system quality and reliability,
• Build monitoring systems and dashboards for AI performance visibility,
• Identify operational and product risks introduced by automation,
• Measure the impact of AI systems on business metrics, operational efficiency, and user experience,
• Detect data quality issues, shifts, and anomalies affecting AI system performance,
• Partner closely with Product, Engineering, QA, and Operations teams on AI-related initiatives,
• Support rapid decision-making in fast-changing and ambiguous environments.

Requirements

• 3+ years of experience in analytical roles related to AI systems, ML products, automation platforms, or complex production systems,
• Strong Python skills for deep analytical investigations and working with large-scale noisy text data,
• Advanced SQL skills,
• Understanding of LLM limitations,hallucinations, evaluation challenges, prompt sensitivity, model trade-offs, and AI failure modes,
• Experience building monitoring systems, analytical frameworks, or operational dashboards for production systems,
• Ability to work with incomplete, ambiguous, and rapidly changing data environments,
• Strong ownership mindset and ability to independently drive investigations and analytical conclusions,
• Ability to connect AI system quality with business and operational impact,
• Fluent Russian; English level B2 or higher,
• Ability to work CET-overlapping hours.

Nice to Have

• Experience with conversational AI or AI agents,
• Experience evaluating LLM outputs or annotation quality,
• Experience with AI quality metrics and evaluation methodologies,
• Experience working with Langchain infrastructure,
• Experience with ML data pipelines or synthetic data generation,
• Background in anomaly detection.

What we offer

• Work on a modern AI product where quality is a strategic function, not a formality
• Real influence on product direction and engineering practices
• High-impact work in a product-driven medical tech company
• Startup environment: fast iteration, high ownership, minimal bureaucracy
• Competitive compensation and growth opportunities

Benefits

• Health insurance after probation period
• Sports & wellness compensation
• Fully remote, long-term B2B collaboration
• 19 working days of vacation per year
• 3 additional wellness days per year
• Paid sick leave for the first 5 working days
• Gifts to employees for important events
• Career Development
• Personalized English lessons via Preply

Ready to make an impact? Send us your profile - let’s build something meaningful together.