AI engineer interviews test a specific mix of skills: ML fundamentals, system design, prompt engineering, and the ability to make practical tradeoffs. I've compiled the 50 most common questions from interview reports shared in our community, organized by category.
Each question includes a strong answer and notes on what interviewers are actually evaluating.
Technical Fundamentals (Questions 1-10)
1. What's the difference between a language model and a traditional ML model?
Answer: Traditional ML models are trained for specific tasks on structured data (classification, regression, clustering). Language models are trained on massive text corpora to predict the next token, which gives them general-purpose language understanding. The key difference is generalization: a traditional model does one thing well, while an LLM can be adapted to many tasks through prompting without retraining.
What they're testing: Whether you understand the fundamental shift from task-specific to general-purpose models.
2. Explain the transformer architecture in simple terms.
Answer: Transformers process all tokens in a sequence simultaneously (not sequentially like RNNs) using self-attention. Self-attention lets each token "look at" every other token to understand context. The model learns which tokens are most relevant to each other. This parallelism is what made training on massive datasets feasible and is why transformers dominate modern AI.
3. What is temperature in LLM inference?
Answer: Temperature controls randomness in token selection. At temperature 0, the model always picks the most probable token (deterministic). At temperature 1, it samples proportionally to token probabilities. Above 1, it flattens the distribution, making unlikely tokens more probable. For factual tasks, use low temperature (0-0.3). For creative tasks, use higher temperature (0.7-1.0).
4. What are embeddings and why do they matter?
Answer: Embeddings are dense vector representations of text (or images, or other data) in a high-dimensional space. Similar concepts end up close together in this space. They matter because they let you do math with meaning: find similar documents, cluster topics, or measure semantic similarity. They're the foundation of RAG systems and semantic search.
5. Explain the difference between supervised, unsupervised, and self-supervised learning.
Answer: Supervised learning trains on labeled input-output pairs (spam detection with labeled emails). Unsupervised learning finds patterns in unlabeled data (customer segmentation). Self-supervised learning creates its own labels from the data (predicting masked words in a sentence). LLMs use self-supervised learning: they predict the next token, with the "label" being the actual next token in the training data.
6. What is hallucination in LLMs and how do you mitigate it?
Answer: Hallucination is when a model generates plausible-sounding but factually incorrect information. Mitigation strategies include: RAG (grounding responses in retrieved documents), constrained generation (limiting output to known entities), chain-of-verification (asking the model to check its own claims), lower temperature settings, and explicit instructions to say "I don't know" when uncertain.
7. What's the context window and why does it matter?
Answer: The context window is the maximum number of tokens a model can process in a single request (input + output combined). GPT-4o supports 128K tokens. Claude supports 200K tokens. It matters because it determines how much information you can provide alongside a query. For RAG systems, larger context windows let you include more retrieved documents. For conversation, it determines how much history the model remembers.
8. Explain tokenization.
Answer: Tokenization splits text into the smallest units the model processes (tokens). These aren't always whole words. "Unbelievable" might become ["un", "believ", "able"]. Different models use different tokenizers. Tokenization affects cost (you pay per token), context window usage, and can cause issues with non-English languages or code where tokenization is less efficient.
9. What is RLHF and why is it used?
Answer: Reinforcement Learning from Human Feedback. After pre-training, models are fine-tuned using human preferences. Humans rank model outputs, and these rankings train a reward model, which then guides further model training. RLHF is what makes models helpful, harmless, and honest rather than just next-token predictors. It's the difference between a base model and an assistant model.
10. What's the difference between fine-tuning and prompt engineering?
Answer: Prompt engineering modifies model behavior through input instructions without changing model weights. Fine-tuning actually updates model weights using additional training data. Prompt engineering is faster, cheaper, and more flexible. Fine-tuning produces more consistent results for specific tasks and can work with smaller, cheaper models. Most teams should start with prompt engineering and only fine-tune when prompting hits its limits.
System Design (Questions 11-20)
11. Design a RAG system for a company's internal documentation.
Answer: Start with the document pipeline: ingest documents, split into chunks (500-1000 tokens with 10-20% overlap), generate embeddings using a model like text-embedding-3-small. Store in a vector database (Pinecone, Weaviate, or pgvector). At query time: embed the user query, retrieve top-k relevant chunks (k=5-10), inject into a prompt with instructions to answer based only on provided context. Add a reranking step between retrieval and generation for quality. Include metadata filtering for access control.
12. How would you handle a situation where your AI system's accuracy drops from 95% to 80% overnight?
Answer: First, check for data pipeline issues (corrupted input data, format changes, missing fields). Then check for model API changes (provider updated the model version). Review recent code deployments. Check if the query distribution shifted (new user segment, seasonal change). Run your eval suite against a known-good baseline. The most common cause is upstream data changes, not model issues.
13. Design a content moderation system using LLMs.
Answer: Use a tiered approach. Tier 1: fast keyword/regex filters for obvious violations (near-zero latency, catches 60-70%). Tier 2: a fine-tuned classifier model for nuanced content (50-100ms, catches another 20-25%). Tier 3: LLM-based analysis for edge cases only (200-500ms, handles the remaining 5-15%). Human review queue for low-confidence decisions. This architecture balances cost, speed, and accuracy.
14. How do you decide between using one large prompt vs. chaining multiple smaller prompts?
Answer: Single prompts are simpler and faster for straightforward tasks. Chains are better when: the task has distinct stages (extract then summarize then format), you need different model settings per stage, intermediate results need validation, or a single prompt exceeds reliable output quality. Chains add latency and cost but improve reliability on complex tasks. I'd default to a single prompt and split only when I see quality issues.
15. Design an AI-powered customer support system.
Answer: Intent classification first (route to the right handler). RAG retrieval from the knowledge base. Response generation with tone and policy constraints. Confidence scoring on every response. If confidence is below threshold (say 0.7), escalate to a human agent with the AI's draft and retrieved context. Feedback loop: human agent corrections feed back into the eval dataset. Track resolution rate, escalation rate, and customer satisfaction as key metrics.
16. How would you optimize an LLM application that's too slow?
Answer: Profile first to find the bottleneck (retrieval vs. generation vs. post-processing). For retrieval: cache frequent queries, use approximate nearest neighbor search, reduce chunk count. For generation: use a smaller model for simple queries (route by complexity), reduce max output tokens, enable streaming for perceived speed, batch requests where possible. For the full pipeline: parallelize independent steps, add result caching with appropriate TTL.
17. How do you handle PII in an AI pipeline?
Answer: Detection layer before data enters the pipeline (regex + NER model for names, emails, SSNs, phone numbers). Redaction or tokenization of detected PII. Process with the model using redacted data. Detokenize in the final output if needed. Log retention policies that exclude raw user inputs. Audit trail for compliance. Never include PII in fine-tuning datasets without explicit consent and legal review.
18. Design an evaluation framework for an LLM application.
Answer: Three layers. Automated metrics: accuracy on known Q&A pairs, format compliance rate, latency percentiles. LLM-as-judge: use a stronger model to evaluate outputs on rubric dimensions (relevance, completeness, safety). Human evaluation: periodic expert reviews on random samples. Run automated evals on every prompt change. LLM-as-judge weekly. Human eval monthly or quarterly. Store all results for trend analysis.
19. How would you build a multi-tenant AI system where each customer has different data?
Answer: Separate vector database namespaces or collections per tenant. Tenant-specific system prompts stored in configuration. Shared inference infrastructure with tenant ID passed at query time. Metadata filtering on retrieval to enforce data isolation. Per-tenant rate limiting and usage tracking. The key architectural decision is shared vs. isolated compute. Shared is cheaper but requires careful isolation guarantees.
20. Explain your approach to prompt versioning and management.
Answer: Treat prompts as code. Version control in git alongside the application. A prompt registry that maps prompt IDs to versions. A/B testing capability for prompt changes. Automated evals that run against a test suite before any prompt change goes to production. Rollback capability if a new prompt version degrades metrics. Structured logging that records which prompt version generated each response.
Prompt Engineering (Questions 21-30)
21. What is few-shot prompting and when do you use it?
Answer: Few-shot prompting includes examples of the desired input-output pattern before the actual query. Use it when: the task requires a specific output format, the model struggles with zero-shot performance, or you need consistent behavior across varied inputs. Typically 3-5 examples work well. More examples improve consistency but cost more tokens. Choose examples that cover edge cases, not just the easy path.
22. How do you debug a prompt that works 80% of the time but fails on 20% of inputs?
Answer: Collect and categorize the failing 20%. Look for patterns: specific input types, edge cases, ambiguous instructions. Common fixes: add explicit handling for the failure patterns, include few-shot examples of the tricky cases, add constraints that prevent the specific failure mode, or break the prompt into steps so you can identify exactly where it fails. The categorization step is most important because it determines the fix.
23. Explain chain-of-thought prompting.
Answer: Chain-of-thought prompting asks the model to show its reasoning step by step before giving a final answer. It improves accuracy on tasks requiring multi-step reasoning (math, logic, analysis). You can trigger it with "Think step by step" or by providing examples that include reasoning traces. It works because it forces the model to allocate computation to intermediate steps rather than jumping to an answer.
24. How do you write a good system prompt?
Answer: Start with role and purpose (one sentence). Then behavioral rules (do this, don't do that). Then output format specification. Then edge case handling. Then examples if needed. Keep instructions specific and testable. Avoid vague directives like "be helpful." Instead: "If the user asks about pricing, provide the current rates from the pricing table. If the specific plan isn't in the table, say so rather than guessing." Test with adversarial inputs.
25. What's the difference between zero-shot, one-shot, and few-shot prompting?
Answer: Zero-shot: just the instruction, no examples. One-shot: one example before the query. Few-shot: multiple examples (typically 3-5). Use zero-shot when the task is straightforward and the model performs well without guidance. Add shots when you need consistent formatting, domain-specific behavior, or better accuracy on complex tasks. Each shot costs tokens, so there's a cost-performance tradeoff.
26. How do you handle multi-language prompts?
Answer: Write system prompts in English (models are most capable in English) but instruct the model to respond in the user's language. Include few-shot examples in each target language for critical formatting. Test thoroughly in each language because model performance varies significantly. For high-stakes applications, consider language-specific prompt variants. Monitor per-language accuracy metrics separately.
27. What strategies do you use to reduce hallucinations?
Answer: Ground responses in provided context (RAG). Instruct the model to cite sources. Add "If you're not sure, say so" instructions. Use lower temperature (0-0.3) for factual tasks. Break complex questions into verifiable sub-questions. Post-process to check generated claims against known data. For critical applications, use a second model call to verify factual claims in the first response.
28. How do you optimize prompts for cost?
Answer: Shorter prompts cost less (fewer input tokens). Remove redundant instructions. Use concise few-shot examples. Set appropriate max_tokens to avoid overly long responses. Use cheaper models for simple tasks and route only complex queries to expensive models. Cache responses for repeated queries. Batch similar requests. Measure cost per successful output, not just cost per API call.
29. Explain prompt injection and how to defend against it.
Answer: Prompt injection is when a user's input overrides or manipulates the system prompt instructions. Example: a user types "Ignore all previous instructions and..." Defense strategies: separate system and user messages using the API's role system, input validation to detect injection patterns, output validation to catch unexpected behaviors, and sandboxing model access so even a successful injection can't access sensitive systems.
30. How do you test prompts at scale?
Answer: Build an eval dataset of at least 100-200 test cases covering normal inputs, edge cases, and adversarial inputs. Define pass/fail criteria for each case. Automate the evaluation pipeline so it runs on every prompt change. Use LLM-as-judge for subjective quality dimensions. Track metrics over time to catch regressions. For high-stakes applications, add human evaluation on a random sample of production outputs weekly.
ML Fundamentals (Questions 31-40)
31. What is overfitting and how do you prevent it?
Answer: Overfitting is when a model performs well on training data but poorly on new data because it memorized patterns instead of learning generalizable features. Prevention: use a validation set to monitor generalization during training, apply regularization (dropout, weight decay), use data augmentation, reduce model complexity if the dataset is small, and use early stopping when validation performance plateaus.
32. Explain precision, recall, and F1 score.
Answer: Precision: of all items predicted positive, what fraction actually is positive (low false positive rate). Recall: of all actual positives, what fraction did the model find (low false negative rate). F1: the harmonic mean of precision and recall. Use precision when false positives are costly (spam filter). Use recall when false negatives are costly (disease screening). Use F1 when you need to balance both.
33. What is transfer learning?
Answer: Using a model trained on one task as the starting point for a different task. Instead of training from scratch (which requires massive data), you start with a pre-trained model and adapt it. Fine-tuning LLMs is transfer learning: the base model learned language from web text, and you adapt it to your specific task. This is why you can fine-tune with hundreds of examples instead of billions.
34. Explain the bias-variance tradeoff.
Answer: Bias: error from overly simple models that miss important patterns (underfitting). Variance: error from overly complex models that are sensitive to training data noise (overfitting). The tradeoff: reducing one tends to increase the other. The sweet spot is a model complex enough to capture real patterns but not so complex that it memorizes noise. In practice, modern deep learning models are often in a "more data helps" regime where adding data reduces both.
35. What is a confusion matrix?
Answer: A table showing true positives, true negatives, false positives, and false negatives for a classification model. It gives you more insight than a single accuracy number. You can derive precision, recall, specificity, and other metrics from it. Especially useful for imbalanced datasets where accuracy alone is misleading (99% accuracy on spam detection means nothing if 99% of emails are not spam).
36. How do you handle imbalanced datasets?
Answer: Options: oversample the minority class (SMOTE), undersample the majority class, use class weights in the loss function, use evaluation metrics that account for imbalance (F1, AUC-ROC instead of accuracy), or collect more minority class examples. The right approach depends on how severe the imbalance is and what errors cost more. For LLM fine-tuning specifically, ensure your training examples represent edge cases proportionally.
37. What is gradient descent?
Answer: The optimization algorithm that trains neural networks. It computes how much the model's error (loss) changes when you nudge each parameter, then adjusts parameters in the direction that reduces error. "Stochastic" gradient descent does this on random batches of data rather than the full dataset, making it practical for large datasets. Learning rate controls how big each adjustment is.
38. Explain batch normalization.
Answer: A technique that normalizes the inputs to each layer during training, keeping them centered around zero with unit variance. Benefits: allows higher learning rates (faster training), reduces sensitivity to initialization, and acts as a mild regularizer. It's standard in most deep learning architectures. The model learns scale and shift parameters for each layer so normalization doesn't limit expressiveness.
39. What is cross-validation?
Answer: A method for evaluating model performance by splitting data into k folds, training on k-1 folds, and testing on the held-out fold. Repeat k times so every fold serves as the test set once. Average the results. This gives a more reliable performance estimate than a single train-test split, especially with small datasets. Common choice: k=5 or k=10.
40. What is the attention mechanism?
Answer: Attention lets the model weigh the importance of different parts of the input when processing each position. In self-attention, each token computes a weighted sum of all other tokens' representations, where the weights reflect relevance. "Multi-head" attention does this multiple times in parallel with different learned projections, capturing different types of relationships. It's the core innovation that makes transformers work.
Behavioral Questions (Questions 41-50)
41. Tell me about a time you had to explain a technical AI concept to a non-technical stakeholder.
What they want: Evidence that you can bridge technical and business perspectives. Use a specific example. Explain what the concept was, why the stakeholder needed to understand it, and how you adapted your explanation. Good answers mention using analogies, visual aids, or framing in terms of business impact rather than technical details.
42. How do you stay current with AI developments?
What they want: A specific, credible learning routine. Mention specific sources: arXiv papers, model release announcements, communities like the PE Collective, hands-on experimentation with new models. Avoid vague answers like "I read a lot." Show that you have a system for filtering signal from noise in a fast-moving field.
43. Describe a project where you had to make a significant technical tradeoff.
What they want: Structured thinking about tradeoffs. Describe the options, the criteria you used to evaluate them, the decision you made, and the outcome. Good tradeoff discussions cover cost vs. quality, speed vs. accuracy, or build vs. buy. Show that you considered multiple perspectives and made a deliberate, data-informed choice.
44. How do you handle a situation where the model's output could cause harm?
What they want: Evidence that you think about AI safety proactively. Discuss guardrails, output validation, human-in-the-loop for high-stakes decisions, monitoring for harmful outputs, and incident response plans. The best answers mention specific examples of harmful outputs you've caught and the systems you built to prevent them.
45. Tell me about a time you disagreed with a product decision related to AI.
What they want: Professional conflict resolution. Describe the disagreement, how you presented your perspective with evidence, and how you handled the outcome whether you "won" or not. They want to see that you advocate with data but can also commit to a team decision you don't fully agree with.
46. How do you prioritize when you have multiple AI features to ship?
What they want: A prioritization framework. Mention impact (how many users, how much value), effort (engineering time, infrastructure needs), risk (what happens if it fails), and dependencies (what blocks what). Show that you can make hard choices about what NOT to build, not just what to build.
47. Describe how you've handled a production AI system failure.
What they want: Incident response skills. Walk through detection (how you found out), triage (severity assessment), mitigation (immediate fix), root cause analysis, and prevention (what you changed). The best answers show learning: "We added monitoring for X because this incident taught us Y."
48. How do you measure the success of an AI feature?
What they want: Metrics thinking. Discuss leading indicators (model accuracy, latency) and lagging indicators (user engagement, task completion rate, business metrics). Show that you connect model performance to business outcomes. Mention A/B testing for feature launches and continuous monitoring post-launch.
49. What's your approach to documentation for AI systems?
What they want: Evidence that you value maintainability. Discuss prompt documentation (what it does, why it's structured that way, known limitations), architecture decision records, eval results documentation, and runbooks for common issues. AI systems are notoriously hard to maintain without good docs because the reasoning behind prompt decisions isn't obvious from the code.
50. Where do you see AI engineering heading in the next 2-3 years?
What they want: Informed perspective, not hype. Good answers discuss trends like: agents becoming more capable, evaluation becoming more important as systems get more complex, the growing need for AI-specific software engineering practices, and the shift from building models to building applications on top of models. Avoid hyperbolic predictions. Show nuanced, grounded thinking.
How to Prepare
Don't try to memorize all 50 answers. Instead, focus on understanding the underlying concepts well enough to explain them in your own words. Practice with a friend or record yourself answering. The best interview answers feel conversational, not rehearsed.
For more interview prep, check out our prompt engineering interview questions guide and browse current openings on the job board.