Enterprise — 3 Documents

Enterprise AI deployment encompasses the full lifecycle of taking machine learning models from research and development into production environments where they deliver business value at scale. Unlike experimental or proof-of-concept deployments, enterprise AI requires:

• Reliability: 99.9%+ uptime with automated failover
• Scalability: Handle thousands to millions of requests per second
• Security: Data isolation, encryption, access control, audit trails
• Compliance: Meet regulatory requirements (GDPR, HIPAA, SOC2, EU AI Act)
• Cost Efficiency: Optimized inference with caching, routing, and tiered models
• Observability: Full traceability of model inputs, outputs, drift, and performance
• Governance: Approval workflows, model registries, versioning, rollback capability

The enterprise AI stack typically has six layers:

┌──────────────────────────────────────────────┐
│            Application Layer                  │
│  Chatbots / Copilots / Automation / Search    │
├──────────────────────────────────────────────┤
│           Orchestration Layer                 │
│  Prompt Pipelines / Agent Frameworks / RAG    │
├──────────────────────────────────────────────┤
│           Model Serving Layer                 │
│  vLLM / TGI / Triton / Ollama / Bedrock       │
├──────────────────────────────────────────────┤
│         Infrastructure Layer                  │
│  Kubernetes / Docker / Serverless / Edge      │
├──────────────────────────────────────────────┤
│         Data & Feature Layer                  │
│  Vector DB / Feature Store / Data Lake        │
├──────────────────────────────────────────────┤
│         Governance & Observability            │
│  Monitoring / Logging / Audit / Compliance    │
└──────────────────────────────────────────────┘

1.1 Key Enterprise Considerations

Latency Requirements:
- Real-time (chat): <500ms p95
- Near-real-time (assistant): <3s p95
- Batch: minutes to hours

Throughput:
- Small deployments: 100-1,000 RPM (requests per minute)
- Medium deployments: 1,000-10,000 RPM
- Large deployments: 10,000-100,000+ RPM

Reliability Targets:
- Development: 99% uptime (3.65 days/year downtime)
- Staging: 99.9% uptime (8.76 hours/year downtime)
- Production critical: 99.99% uptime (52.56 minutes/year downtime)
- Mission-critical: 99.999% uptime (5.26 minutes/year downtime)

2.1 Cloud Deployment

The most common enterprise pattern. Models run on cloud infrastructure with auto-scaling.

Pros: Elastic scaling, pay-per-use, managed services, global availability
Cons: Data egress costs, latency for on-premise integration, vendor lock-in

Cloud Providers:
- AWS: Bedrock (managed), SageMaker (custom), EC2 (DIY)
- Azure: OpenAI Service, AI Studio, ML Studio
- GCP: Vertex AI (managed), GKE (custom), Cloud Run (serverless)

2.2 On-Premises Deployment

Models run entirely within the organization's data center.

Pros: Complete data sovereignty, low latency for internal services, predictable cost
Cons: Fixed capacity, hardware procurement, maintenance overhead, GPU scarcity

Typical Hardware:
- Inference: NVIDIA A100/H100/H200/B200, AMD MI300X, Intel Gaudi
- Small: 1-4 GPUs per model (7B-70B params)
- Medium: 8-16 GPUs per model (70B-180B params)
- Large: 32-128 GPUs for high-throughput serving
- Training clusters: 256-16,000+ GPUs

2.3 Edge Deployment

Models run on user devices or edge servers.

Pros: Zero latency, offline capability, privacy by design
Cons: Hardware constraints, limited model size, update distribution

Common Edge Targets:
- Mobile: Apple Neural Engine, Qualcomm Hexagon, Samsung NPU
- Laptop: Apple M-series, Intel NPU, NVIDIA RTX (local)
- IoT: Raspberry Pi, Jetson Orin/Nano, Intel Movidius
- Browser: WebGPU, ONNX Runtime Web, Transformers.js

Model Sizes for Edge:
- Quantized 1-3B models (Qwen2.5-1.5B, Phi-3-mini, Gemma-2B)
- Multimodal: MobileCLIP, EfficientViT-SAM
- Speech: Whisper tiny/small, Silero VAD
- 4-bit quantized 7B for high-end devices

2.4 Hybrid Deployment

Combines cloud, on-prem, and edge based on workload requirements.

Strategy:
- Simple queries → Edge (fast, free, private)
- Standard queries → On-prem (low latency, controlled)
- Complex queries → Cloud (unlimited compute, latest models)
- Fine-tuning/Training → Cloud/on-prem GPU clusters

              Edge                         On-Premise                   Cloud
        ┌─────────────┐              ┌──────────────┐           ┌──────────────┐
        │ Small Models │─────────────▶│ Medium Models│──────────▶│ Large Models │
        │  (<3B, 4bit) │  escalation  │ (7B-70B)     │ escalation│ (70B-400B+)  │
        │              │              │              │           │              │
        │  ~1ms latency│              │  ~50ms lat.  │           │  ~200ms lat. │
        │  ~$0 cost    │              │  ~fixed cost │           │  ~per-token  │
        └─────────────┘              └──────────────┘           └──────────────┘

3.1 Full MLOps Lifecycle

Data Collection → Data Validation → Feature Engineering → Model Training
       ↓                                                         ↓
  Model Registry ← Model Evaluation ← Model Validation ← Hyperparameter Tuning
       ↓
  Model Deployment → Prediction Serving → Monitoring → Feedback Loop
                                                              ↓
                                                   Retraining Trigger

3.2 Key MLOps Components

Feature Store (Feast, Tecton, SageMaker Feature Store):
- Centralized feature computation and storage
- Online (low-latency) and offline (batch) serving
- Feature sharing across teams
- Point-in-time correct feature retrieval for training

Model Registry (MLflow Model Registry, Hugging Face Hub, S3/GCS):
- Versioned model artifacts with metadata
- Model lineage tracking (data → training → deployment)
- Stage transitions (staging → production → archived)
- Model signatures and input/output schemas

Pipeline Orchestration (Kubeflow, Airflow, Prefect, Dagster):
- DAG-based pipeline definition
- Retry and error handling
- Artifact passing between steps
- Scheduled and event-driven execution

Experiment Tracking (MLflow, Weights & Biases, Neptune, Comet):
- Hyperparameter logging
- Metrics visualization
- Artifact storage (model weights, plots, configs)
- Run comparison and search

3.3 CI/CD for ML

                    ┌─────────────────────┐
                    │    Source Control    │
                    │  (code + configs)   │
                    └──────────┬──────────┘
                               │ push/PR
                               ▼
                    ┌─────────────────────┐
                    │    CI Pipeline       │
                    │  - Lint code         │
                    │  - Unit tests        │
                    │  - Integration tests │
                    │  - Model validation  │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │   Model Training     │
                    │  (triggered by CI)   │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │   Model Evaluation   │
                    │  - Benchmark scores  │
                    │  - Offline metrics   │
                    │  - Bias detection    │
                    └──────────┬──────────┘
                               │ pass?
                    ┌──────────▼──────────┐
                    │  CD Pipeline         │
                    │  - Canary deploy     │
                    │  - Shadow testing    │
                    │  - Blue-green deploy │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │   Monitoring         │
                    │  - Performance       │
                    │  - Drift detection   │
                    │  - Alerting          │
                    └─────────────────────┘

4.1 Model Serving Frameworks

vLLM:
- High-throughput LLM serving with PagedAttention
- Continuous batching, prefix caching, tensor parallelism
- OpenAI-compatible API
- Supports: LLaMA, Mistral, Qwen, DeepSeek, Falcon, Gemma, GPT-NeoX
- Best for: high-throughput, production LLM serving

Text Generation Inference (TGI) by Hugging Face:
- Production-focused LLM serving
- Tensor parallelism, continuous batching, quantization (bitsandbytes, GPTQ, AWQ)
- Message streaming via Server-Sent Events
- Best for: Hugging Face ecosystem integration

NVIDIA Triton Inference Server:
- Multi-framework support (TensorRT, ONNX, PyTorch, TensorFlow)
- Concurrent model execution, dynamic batching
- Ensemble and BLS (Business Logic Scripting)
- GPU optimization with TensorRT
- Best for: heterogeneous model serving at scale

Ollama:
- Simplified local model serving
- Model management (pull, run, stop)
- OpenAI-compatible API
- GPU acceleration (CUDA, Metal, Vulkan)
- Best for: development, testing, personal use

TensorRT-LLM:
- NVIDIA's optimized LLM serving
- In-flight batching, paged attention, multi-GPU
- FP8, INT4, INT8 quantization
- Best for: maximum GPU utilization on NVIDIA hardware

Comparison Table:

4.2 Containerization and Orchestration

Docker:
- Standard container format for ML models
- Multi-stage builds for size optimization
- Health checks, resource limits, environment configuration

Kubernetes (K8s):
- Standard orchestration for ML workloads
- Deployments for stateless serving
- StatefulSets for stateful services (vector DBs)
- Horizontal Pod Autoscaler (HPA) for scaling
- Custom metrics (requests per second, queue depth, GPU utilization)
- Node pools for GPU vs CPU workloads
- Affinity/anti-affinity for GPU pod placement
- Service mesh (Istio, Linkerd) for traffic management

Example Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server
spec:
  replicas: 4
  selector:
    matchLabels:
      app: llm-server
  template:
    metadata:
      labels:
        app: llm-server
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args: ["--model", "mistralai/Mistral-7B-Instruct-v0.3",
               "--tensor-parallel-size", "1",
               "--max-model-len", "8192",
               "--gpu-memory-utilization", "0.90"]
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "32Gi"
            cpu: "8"
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4

4.3 Serverless ML

AWS SageMaker Serverless, GCP Cloud Run, Azure Container Apps:
- Scale to zero when not in use
- Pay-per-invocation (no idle cost)
- Cold start latency (5-30 seconds)
- Best for: bursty, low-volume, or spiky workloads

Emerging: serverless GPU (Modal, RunPod, Banana, Replicate):
- GPU-backed serverless inference
- Automatic scaling, no infrastructure management
- Pay per second of GPU usage
- Best for: prototyping, demos, low-volume production

5.1 Metrics Categories

Infrastructure Metrics:
- GPU utilization, memory, temperature
- Request latency (avg, p50, p95, p99, p999)
- Request throughput (RPS / TPS / RPM)
- Error rate (4xx, 5xx, timeout)
- Queue depth and wait time

Model Performance Metrics:
- First-token latency (TTFT — Time to First Token)
- Inter-token latency (ITL)
- Tokens per second (throughput)
- Context window utilization
- Token generation distribution

Quality Metrics:
- Response relevance (human eval, LLM-as-judge)
- Factual accuracy (grounding checks)
- Toxicity / safety score
- Response length distribution
- User satisfaction (thumbs up/down, ratings)

Drift Metrics:
- Input distribution drift (embedding distance over time)
- Output distribution drift
- Performance drift (quality metrics over time)
- Data drift detection (statistical tests: KS, PSI, MMD)

5.2 Monitoring Tools

Infrastructure:
- Prometheus + Grafana (standard open-source stack)
- Datadog (SaaS, full-stack monitoring)
- New Relic (APM-focused)
- Grafana Cloud / Grafana Loki (logs + metrics)

ML-Specific:
- LangSmith (LLM observability, traces, evaluations)
- Weights & Biases Prompts (prompt monitoring)
- Arize AI (ML observability, drift detection)
- LangFuse (open-source LLM monitoring)
- Helicone (LLM proxy with logging)
- MLflow (model monitoring)

Alerting:
- PagerDuty / OpsGenie for on-call
- Slack / Teams notifications for warnings
- Email for daily/weekly reports

5.3 Observability Tracing

Distributed tracing for LLM applications provides end-to-end visibility:

User Request
  └─ API Gateway
       └─ Orchestrator
            ├─ Prompt Template (5ms)
            ├─ RAG Retrieval (150ms)
            │    ├─ Query Embedding (20ms)
            │    ├─ Vector DB Search (80ms)
            │    └─ Re-ranking (50ms)
            ├─ Context Assembly (10ms)
            ├─ LLM Inference (800ms)
            │    ├─ Time-to-First-Token (400ms)
            │    └─ Token Generation (400ms)
            └─ Post-processing (15ms)
Total: ~980ms

6.1 Testing Strategies

Shadow Testing:
- New model runs in parallel with production model
- Outputs compared but only production served to users
- Safe: no user impact, collect comparison data
- Duration: 1-7 days

Canary Deployment:
- Gradually shift traffic to new model (5% → 25% → 50% → 100%)
- Monitor for regressions at each step
- Auto-rollback if error rate or quality drops
- Duration: hours to days

Champion/Challenger:
- Old model (champion) vs new model (challenger)
- Controlled experiment with statistical significance testing
- Measure: user satisfaction, task completion, latency, cost
- Requires sufficient traffic for statistical power

Interleaved A/B:
- Each user sees responses from both models (side-by-side or interleaved)
- Blind comparison (user doesn't know which is which)
- Faster convergence, requires less traffic
- More complex to implement

6.2 Evaluation Metrics

7.1 Cost Components

Total Cost = Compute + API + Storage + Network + Human
  - Compute: GPU instances, CPU instances, serverless
  - API: Third-party model APIs (OpenAI, Anthropic, etc.)
  - Storage: Vector databases, model storage, embeddings cache
  - Network: Data transfer, cross-region, egress
  - Human: Labeling, monitoring, prompt engineering

7.2 Optimization Strategies

Prompt Optimization:
- Shorter prompts reduce token costs
- Fewer few-shot examples (use dynamic retrieval instead)
- System prompt compression
- Prompt caching (Anthropic's prompt caching, Gemini context caching)

Output Length Control:
- Use max_tokens constraints
- Set stop sequences early
- Use structured output formatting
- Compress or truncate when possible

Model Tiering:

Query Type          → Model to Use                  → Cost/1M tokens
──────────────────────────────────────────────────────────────
Simple Q&A          → GPT-4o-mini / Mistral-7B      → $0.15-0.60
Content extraction  → Claude 3 Haiku / Gemini Flash → $0.25-1.00
Code generation     → Claude 3.5 Sonnet / GPT-4o    → $3.00-10.00
Complex reasoning   → Claude 4 Opus / DeepSeek-R1   → $15.00-60.00
Research / analysis → Full o3 / Gemini 2.5 Pro      → $15.00-75.00

Semantic Cache:
- Cache results for semantically similar queries
- Reduce duplicate computation by 20-60%
- Memory cost of cache vs token cost savings
- Typical: Redis + embedding-based similarity search

Batching:
- Batch non-realtime requests for higher throughput
- vLLM server batching reduces per-request overhead
- Combine multiple prompts into single inference call

Provider Cost Comparison (as of 2026):

8.1 Regulatory Landscape

EU AI Act (2025+):
- Risk categories: minimal, limited, high, unacceptable
- High-risk AI systems require: risk assessment, data governance, transparency, human oversight, accuracy/robustness
- GPAI (General Purpose AI) models: transparency, copyright, systemic risk assessment
- Fines: up to 7% of global annual turnover or €35M

US Executive Orders on AI (2023-2026):
- Safety testing of powerful AI systems (red-teaming results)
- Watermarking (C2PA content provenance standards)
- Privacy-preserving techniques research
- Equity and civil rights guidance

China AI Regulations (2023-2026):
- Algorithm registration requirements
- Content safety reviews for generative AI
- Data localization requirements
- Synthetic data labeling mandates

Other Jurisdictions:
- UK: AI Safety Institute, pro-innovation framework
- Canada: AIDA (Artificial Intelligence and Data Act)
- Japan: AI Guidelines (soft law approach)
- Singapore: Model AI Governance Framework
- Brazil: Bill 2338/2023 (LGPDP + AI specific provisions)
- India: NITI Aayog AI principles

8.2 Governance Framework

Model Governance Lifecycle:

1. Model Intake
   - Business case approval
   - Risk assessment
   - Regulatory mapping

2. Model Development
   - Data provenance documentation
   - Fairness / bias evaluation
   - Explainability analysis
   - Documentation (model cards)

3. Model Validation
   - Independent validation team
   - Performance benchmarking
   - Stress testing / edge cases
   - Compliance review

4. Model Approval
   - Governance committee review
   - Approval for deployment
   - Conditions / limitations documented

5. Model Deployment
   - Staged rollout
   - Monitoring thresholds set
   - Human-in-the-loop requirements

6. Model Monitoring
   - Ongoing performance tracking
   - Drift detection
   - Periodic re-validation
   - Incident response plan

7. Model Retirement
   - Decommissioning plan
   - Data cleanup
   - Lessons learned

8.3 Model Documentation

Model Cards (Standardized by Hugging Face, now widely adopted):
- Model description
- Intended use and limitations
- Training data and methodology
- Evaluation results (segmented by group)
- Ethical considerations
- Caveats and recommendations

Datasheets for Datasets:
- Motivation (why was dataset created?)
- Composition (what fields, how much data?)
- Collection process (how was data gathered?)
- Preprocessing (cleaning, filtering, labeling)
- Uses (what are appropriate/forbidden uses?)
- Distribution (licenses, access restrictions)
- Maintenance (who updates, how?)

9.1 Azure OpenAI Service

Capabilities:
- GPT-4o, GPT-4 Turbo, GPT-4o-mini, GPT-4.1 models
- GPT-4o Realtime API (speech-to-speech)
- DALL-E 3, Whisper, Embeddings models
- Azure AI Search integration (built-in RAG)
- Content filtering and safety system
- Private endpoints and managed VNet
- Role-based access control (RBAC)
- Data residency (regional deployments)
- Responsible AI dashboard
- SLA: 99.9% for provisioned throughput

Pricing Model:
- Pay-as-you-go per token
- Provisioned throughput (reserved capacity)
- Batch processing (50% discount)

9.2 AWS Bedrock

Capabilities:
- Multiple foundation models (Claude, LLaMA, Mistral, Cohere, AI21, Amazon Titan)
- Knowledge Bases (serverless RAG with OpenSearch)
- Agents (multi-step task orchestration)
- Guardrails (content filtering, topic denial, PII redaction)
- Model evaluation (automated and human)
- Provisioned throughput
- Integration with AWS ecosystem (Lambda, S3, Kendra, DynamoDB)

9.3 GCP Vertex AI

Capabilities:
- Gemini 1.5/2.0/2.5 series, Claude, LLaMA, Gemma
- Agent Builder (no-code agent creation)
- Model Garden (curated model hub)
- Vector Search (scaled ANN search)
- Evaluation Service (automated metrics)
- Vertex AI Studio (prompt design and testing)
- Model Registry and Model Monitoring

9.4 Databricks

Capabilities:
- Unity Catalog (unified data governance)
- MLflow for model lifecycle
- Model Serving (serverless GPU endpoints)
- Foundation Model APIs (Llama, Mixtral, DBRX)
- Vector Search for RAG
- AI/BI dashboards and Genie
- Lakehouse AI platform

9.5 Enterprise Platform Comparison

10.1 Threat Model for LLM Applications

Common Attack Vectors:
- Prompt Injection: Crafting inputs that override system instructions
- Jailbreaking: Bypassing safety guardrails
- Data Extraction: Extracting training data or system prompts
- Indirect Injection: Malicious content in retrieved documents
- Denial of Service: Inputs that cause excessive token generation
- Model Inversion: Reconstructing training examples from model output
- Supply Chain: Compromised model weights, plugins, or libraries

10.2 Defense Layers

Input Layer:
  ┌─ Input Validation (length limits, pattern filtering)
  ├─ PII Detection and Redaction
  ├─ Prompt Injection Detection (Guardrails AI, NeMo Guardrails)
  └─ Rate Limiting and Quota Management

Orchestration Layer:
  ┌─ Context Sanitization (clean retrieved documents)
  ├─ Permission Checks (user access to specific data)
  ├─ Tool Access Control (which tools per user role)
  └─ Output Verification (check for info leakage)

Model Layer:
  ┌─ Model-Level Guardrails (system prompt hardening)
  ├─ Content Filter (toxicity, hate, violence, sexual)
  ├─ Output Constraint (JSON schema enforcement, regex)
  └─ Hallucination Detection (grounding checks)

Output Layer:
  ┌─ PII Re-check on output
  ├─ Output Length Limiting
  ├─ Response Approval (human-in-the-loop for critical actions)
  └─ Audit Logging (full request/response trace)

10.3 Data Isolation

Multi-Tenant Architectures:
- Database-per-tenant: Strongest isolation, highest cost
- Schema-per-tenant: Good isolation, shared infrastructure
- Row-level security: Weakest isolation, most efficient
- Document-level filtering: Via vector DB metadata filters

Privacy Techniques:
- Differential Privacy: Add calibrated noise to training data
- Federated Learning: Train across institutions without data sharing
- Data Masking: Replace sensitive values with masked tokens
- On-Device Processing: Process sensitive data offline on user devices

11.1 Router Architecture

The multi-model router directs each query to the optimal model based on complexity, cost, latency requirements, and capability needs.

                     ┌──────────────────────┐
                     │   Query Classifier    │
                     │  (small fast LLM)     │
                     └──────┬───────┬───────┘
                            │       │
              ┌─────────────┘       └─────────────┐
              ▼                                    ▼
    ┌──────────────────┐              ┌──────────────────────┐
    │   Simple Router   │              │   Complex Router      │
    │                   │              │                      │
    │  10% → GPT-4o-mini│              │ 60% → Claude 4 Sonnet│
    │ 30% → Gemini Flash│              │ 25% → GPT-4o         │
    │ 60% → Mistral-7B  │              │ 15% → DeepSeek-R1    │
    └──────────────────┘              └──────────────────────┘
              │                                │
              └──────────┬─────────────────────┘
                         ▼
              ┌─────────────────────┐
              │  Response Validator  │
              │  - Check quality     │
              │  - Check safety      │
              │  - Escalate if poor  │
              └─────────────────────┘

11.2 Routing Strategies

Rule-Based Routing:
- Query length < 50 tokens → cheap model
- Contains code → code-capable model
- Contains PII → local model
- Contains "error"/"exception" → reasoning model

ML-Based Routing:
- Train a classifier to predict model success
- Features: query length, domain, complexity score
- Label: which model produced best result
- Continuous learning from user feedback

Cost-Aware Routing:
- Budget per query/quota per user
- Route to cheaper model when budget near limit
- Fallback chain: try cheap → escalate if needed

Latency-Aware Routing:
- Critical path queries → fast model
- Responses under SLA → balanced routing
- Background/batch → cheapest possible
- Peak load → degrade model tier

11.3 Model Cascade

Level 1: Local (Ollama / ONNX)
  - Model: Qwen2.5-7B-Q4 / Phi-4-mini
  - Latency: ~50ms
  - Cost: ~$0
  - Use: Simple FAQ, autocomplete, quick Q&A
  - Fallback threshold: confidence < 0.8 or "I don't know"

Level 2: Fast API (GPT-4o-mini / Claude Haiku / Gemini Flash)
  - Model: gpt-4o-mini / claude-3-haiku
  - Latency: ~200ms
  - Cost: $0.15-0.60/1M
  - Use: Standard queries, classification, extraction
  - Fallback threshold: uncertain, user dissatisfied

Level 3: Premium API (GPT-4o / Claude Sonnet / Gemini Pro)
  - Model: gpt-4o / claude-4-sonnet
  - Latency: ~800ms
  - Cost: $2.50-15.00/1M
  - Use: Complex reasoning, code generation, analysis
  - Fallback threshold: task failed or high complexity detected

Level 4: Expert (DeepSeek-R1 / Claude Opus / o4-mini)
  - Model: deepseek-r1 / claude-4-opus
  - Latency: ~3s
  - Cost: $15.00-60.00/1M
  - Use: Research, math, PhD-level reasoning, safety-critical
  - No fallback (this is the ceiling)

12.1 Horizontal Scaling

Scale model serving by adding more replicas:

Stateless (Easy): LLM serving pods behind a load balancer
- Just add more pods
- Requires shared model weight storage (NFS, S3, PVC)
- Watch for GPU memory pressure

Stateful (Hard): Vector databases, caches
- Shard data across pods
- Replication for read availability
- Distributed consensus for writes

12.2 Vertical Scaling

Use larger GPU instances:

Trade-offs:
- Bigger GPUs: higher throughput per instance, but lower availability
- Multiple GPUs: tensor parallelism for large models, pipeline parallelism for throughput
- Instance types: A10G (24GB), L4 (24GB), A100 (40/80GB), H100 (80GB), H200 (141GB), B200 (192GB)

12.3 Model Parallelism Strategies

Tensor Parallelism (TP): Split layers across GPUs
- Requires NVLink/NVSwitch for fast GPU-to-GPU communication
- Common for models > 30B parameters
- Minimally: 1 GPU for 7B, 2 for 13B, 4 for 34B, 8 for 70B

Pipeline Parallelism (PP): Split layers into stages
- Each GPU has a subset of consecutive layers
- Lower communication requirement than TP
- Suffers from bubble overhead

Sequence Parallelism (SP): Split sequence dimension across GPUs
- Useful for very long sequences
- Often combined with TP

Expert Parallelism (EP): For MoE models, split experts across GPUs
- Each GPU handles subset of experts
- All-to-all communication for expert routing
- Used by Mixtral 8x7B, DeepSeek-V2/V3, DBRX

12.4 Autoscaling Configuration

# Horizontal Pod Autoscaler for LLM serving
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-server
  minReplicas: 2
  maxReplicas: 50
  metrics:
  - type: Pods
    pods:
      metric:
        name: llm_queue_depth
      target:
        type: AverageValue
        averageValue: 5
  - type: Pods
    pods:
      metric:
        name: llm_gpu_utilization
      target:
        type: AverageValue
        averageValue: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 200
        periodSeconds: 15

Decision Matrix

Choosing the right paradigm — fine-tuning, Retrieval-Augmented Generation (RAG), or prompting — is the most consequential architectural decision in any LLM project. The wrong choice burns GPU budget on unnecessary training, couples you to fragile prompt engineering, or locks you into a retrieval pipeline with latency you cannot sustain. This matrix provides a structured framework.

Decision Flow

Start with prompting. If the task can be accomplished with 0–10 examples in the prompt and the required knowledge fits within the model's training cutoff, prompting is free and infinitely fast to iterate. Use structured output (JSON mode, function calling) to enforce format.

Add RAG when the model needs access to information outside its weights. If the answer depends on specific documents, proprietary knowledge bases, or frequently updating data, RAG provides grounding without retraining. RAG is also the correct choice when data volumes exceed what fine-tuning can absorb (100K+ docs) or when data sources change daily.

Fine-tune when the model needs to internalize a behavior, style, or domain. Fine-tuning shines when:
- The output format must be strictly controlled (e.g., legal document templates, clinical note styles)
- The model must deeply understand domain-specific jargon and concepts (e.g., medical terminology, legal citations, financial reporting standards)
- The model needs to exhibit a specific persona or tone consistently
- RAG retrieval quality is inadequate (low precision/recall on domain corpus)
- Inference latency cannot tolerate retrieval overhead

Hybrid Patterns

The most successful enterprise deployments combine all three:

Prompting (system prompt + instructions)
    ↓
RAG (retrieved context injected into prompt)
    ↓
Fine-Tuned Model (domain-adapted base)

Example: Enterprise Legal Assistant
- Prompting: System prompt defines lawyer persona, jurisdiction rules, and output format
- RAG: Retrieves relevant case law, statutes, and contract clauses from vector database
- Fine-Tuning: Model fine-tuned on legal document corpus for citation format, legal reasoning, and jurisdiction-specific terminology

Anti-Patterns to Avoid

2.1 Legal Domain

Legal NLP requires precision in citation format, jurisdiction awareness, and understanding of specialized terminology spanning centuries of precedent.

Key Adaptation Approaches:

• Continued pre-training (CPT) on legal corpora (CourtListener, CaseLaw, PACER, legal textbooks, statutes) — 10B–50B tokens for base model assimilation
• Instruction fine-tuning on legal QA pairs, contract analysis, and legal reasoning chains
• RLHF/DPO for alignment with legal ethics, confidentiality rules, and jurisdiction-specific output formats

Critical Capabilities:
- Citation extraction and formatting (Bluebook, ALWD, OSCOLA)
- Legal entity recognition (statutes, case names, court names, judges, parties)
- Jurisdiction classification (federal vs state, common law vs civil law)
- Temporal reasoning (overruled precedents, statutory amendments)

Dataset Sources:
- Contracts: EDGAR (SEC filings), US Contract datasets (200K+ contracts), Atticus (contract review)
- Case Law: CourtListener bulk data, Caselaw Access Project (Harvard Law), Free Law Project
- Statutes: US Code XML, State legislative databases, EUR-Lex (EU law)
- Legal QA: LexGLUE, CaseHOLD, LegalBench, CUAD (Contract Understanding Atticus Dataset)

Notable Domain-specific Models:
- LLaMA-based: SaulLM-7B, SaulLM-141B (legal)
- BERT-based: Legal-BERT, CaseLaw-BERT, RoBERTa-Legal
- Encoder-only: LexNLP-based models

Evaluation Benchmarks:
- LegalBench (162 tasks across 6 categories: reasoning, interpretation, drafting, analysis, entity, citation)
- LexGLUE (7 tasks: ECHR, casehold, unfair_tos, etc.)
- CUAD (41 contract question types)
- CaseHOLD (citation prediction)

2.2 Medical Domain

Medical adaptation requires handling of clinical abbreviations, drug names, anatomical terminology, and strict adherence to medical formatting standards (ICD-10, SNOMED CT, RxNorm).

Key Adaptation Approaches:
- Continued pre-training on PubMed Central, MIMIC-III/IV clinical notes, Medscape, textbooks
- Multi-task fine-tuning: NER for medical entities, relation extraction for drug interactions, summarization for clinical notes
- Reinforcement learning from patient outcome proxies (not recommended without rigorous oversight)

Critical Capabilities:
- Medical entity extraction (symptoms, diagnoses, medications, procedures, lab values)
- ICD-10/SNOMED/CPT code assignment
- Clinical note summarization and structured data extraction
- Radiology report generation
- Drug-drug interaction detection

Dataset Sources:
- Clinical Notes: MIMIC-III/IV (requires credentialed access), i2b2/n2c2 datasets, eICU
- Biomedical Literature: PubMed Central Open Access (3M+ full-text articles), PubMed abstracts
- Medical QA: MedQA (USMLE), MedMCQA, PubMedQA, BioASQ
- Drug Data: DrugBank, RxNorm, DailyMed (FDA labels)

Privacy Considerations:
- HIPAA compliance required for training on protected health information (PHI)
- De-identification before any training (MIMIC is pre-de-identified)
- Differential privacy during training when using clinical data
- Model weights may memorize PHI — avoid deploying on raw clinical text

Evaluation Benchmarks:
- MedQA (USMLE step questions)
- MedMCQA (multi-choice medical questions)
- PubMedQA (biomedical QA)
- BioBERT-based NER benchmarks (NCBI Disease, BC5CDR, JNLPBA)

2.3 Finance Domain

Financial models need to understand SEC filings, market terminology, financial ratios, and temporal market context.

Key Adaptation Approaches:
- CPT on financial corpus: SEC filings (10-K, 10-Q, 8-K, S-1), earnings call transcripts, analyst reports, financial news
- Instruction tuning for financial QA, sentiment analysis, numerical reasoning, and report generation
- Numerical tokenization considerations (some domains benefit from digit tokenization or specialized number embeddings)

Critical Capabilities:
- Financial entity recognition (ticker symbols, company names, currency amounts, financial ratios)
- Numerical reasoning (percentage change, CAGR, EPS, P/E ratio calculation)
- Financial sentiment analysis (bullish/bearish/neutral on financial texts)
- SEC filing structure understanding (Item numbers, MD&A, Risk Factors, Financial Statements)
- Multi-lingual support (global markets)

Dataset Sources:
- SEC Filings: EDGAR (free access via SEC API), thousands of new filings daily
- Earnings Calls: Transcripts from SeekingAlpha, Motley Fool, manual scraping
- Financial News: Reuters, Bloomberg (licensed), Yahoo Finance
- Financial QA: FinQA, ConvFinQA, TAT-QA (table-based), FinanceBench

Numerical Capabilities:
Financial NLP places extreme demands on numerical reasoning. Evaluate:
- Calculation of financial ratios given raw financial data
- Trend identification (YoY growth, QoQ changes)
- Table understanding (financial statements in tabular format)
- Currency conversion and unit normalization

Evaluation Benchmarks:
- FinQA (numerical reasoning over financial reports)
- ConvFinQA (conversational financial QA)
- FinanceBench (real-world financial analyst questions)
- TAT-QA (table-and-text hybrid QA)
- FiQA SA (financial sentiment analysis)

2.4 Code Domain

Code-adapted models power developer productivity tools, automated documentation, code review, and bug fixing.

Key Adaptation Approaches:
- CPT on code corpora: GitHub repositories (permissively licensed), Stack Overflow, technical documentation
- Fill-in-the-middle (FIM) training: predict masked code spans, not just left-to-right generation
- Multi-language training: Python, JavaScript, TypeScript, Java, Go, Rust, C++, SQL, Bash

Critical Capabilities:
- Code completion (single-line, multi-line, whole-function)
- Code explanation and documentation generation
- Bug detection and fix suggestion
- Code translation (e.g., Python to JavaScript)
- SQL query generation from natural language
- Shell command generation

Dataset Sources:
- GitHub: The Stack (3TB+), CodeParrot, GitHub Archive
- QA: Stack Overflow data dump, CodeSearchNet, StaQC
- Documentation: Official docs (Python, JavaScript, Rust, etc.)

Training Considerations:
- FIM training: Use a FIM transformation that converts code into <PREFIX><SUFFIX><HOLE> format. Common implementations: PSM (Prefix-Suffix-Middle), SPM (Suffix-Prefix-Middle)
- Language balancing: Weight smaller languages (Rust, Haskell) higher to prevent catastrophic forgetting
- File-level context: Code files have inter-dependencies; consider whole-file or cross-file context windows
- Syntax validation: Add AST validation to training data filters

Evaluation Benchmarks:
- HumanEval (function synthesis from docstrings)
- MBPP (mostly basic Python programming)
- SWE-bench (real-world GitHub issue resolution)
- BigCodeBench (multi-language, multi-task)
- RepoBench (repository-level code completion)

2.5 Customer Support Domain

Customer support models need tone control, brand voice adherence, escalation logic, and multi-turn conversation management.

Key Adaptation Approaches:
- Conversational fine-tuning on chat logs, ticket histories, and email threads
- Supervised fine-tuning with agent-written responses as targets
- RLHF/DPO with customer satisfaction scores as reward signal

Critical Capabilities:
- Intent classification and routing
- Sentiment detection and empathy generation
- Knowledge base grounding (hybrid with RAG)
- Multi-turn conversation coherence
- Escalation detection and handoff
- Brand voice and tone consistency

Dataset Sources:
- Internal ticketing systems: Zendesk, Freshdesk, Salesforce Service Cloud, Intercom
- Chat logs: Live chat transcripts (anonymized)
- Email: Customer support email threads (PII-redacted)
- Knowledge base: Internal FAQ and documentation

Privacy and Compliance:
- GDPR: Support data often contains PII; ensure retention policies
- CCPA: California consumer privacy rights apply
- PCI DSS: Payment information must never enter training data
- HIPAA: If healthcare support, clinical data protection required

Quality Metrics:
- CSAT (Customer Satisfaction Score) improvement
- First Contact Resolution (FCR) rate
- Average Handle Time (AHT) change
- Escalation rate reduction
- Agent acceptance rate of AI-suggested responses

Parameter-Efficient Fine-Tuning (PEFT) methods are the backbone of enterprise LLM adaptation. They make fine-tuning economically viable by updating only a tiny fraction of model parameters while retaining the base model's general capabilities.

3.1 LoRA (Low-Rank Adaptation)

Core Concept:
Given a pre-trained weight matrix W ∈ ℝ^{d×k}, LoRA constrains its update ΔW to be low-rank:

ΔW = BA, where B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}, and r ≪ min(d, k)

The forward pass becomes:

h = Wx + BAx

Only A and B are trained; W is frozen.

Hyperparameters in Detail:

Rank (r)

The rank r controls the representational capacity of the adapter. Enterprise practitioners should think of r as a budget parameter: more rank = more expressivity, more parameters, more VRAM, more regularization needed.

Empirical rule: Start with r=8 for data <10K examples, r=16 for 10K–50K, r=32+ for 50K+. In practice, r=16 is the "sweet spot" for most enterprise use cases.

Alpha (α)

LoRA scaling factor determines how much the adaptation affects the forward pass:

h = Wx + (α/r) * BAx

• α = r: Default (1:1 scaling). No amplification.
• α = 2r: Doubles the effective learning rate of the adapter.
• α = 16, r=8: Scale factor of 2.
• α = 32, r=8: Scale factor of 4.

Best practices:
- Keep α proportional to r (α = r, 2r, or 4r)
- Higher α/r ratio = stronger regularization effect (smaller effective updates)
- Lower α/r ratio = faster convergence but potential instability
- Common enterprise setting: α = 16, r = 16

Dropout

LoRA dropout applies dropout to the adapter output:

h = Wx + dropout(BAx, p)

Target Modules

Which weight matrices to adapt determines both cost and quality.

Common choices for transformer architectures:

# Llama 2/3 architecture
target_modules = [
    "q_proj",      # Query projection — most impactful
    "v_proj",      # Value projection — important for output
    "k_proj",      # Key projection — less critical than Q/V
    "o_proj",      # Output projection — subtle improvements
    "gate_proj",   # MLP gate — significant for reasoning
    "up_proj",     # MLP up projection
    "down_proj",   # MLP down projection
]

Enterprise recommendation: Start with q_proj, v_proj and add o_proj, k_proj if quality is insufficient. Add MLP modules (gate_proj, up_proj, down_proj) only for complex reasoning tasks where attention-only adaptation plateaus.

3.2 QLoRA (Quantized LoRA)

QLoRA enables fine-tuning of large models (65B, 70B, 140B+) on a single consumer GPU by quantizing the base model to 4-bit while keeping LoRA adapters in full precision.

4-bit NormalFloat (NF4)

NF4 is a quantization data type designed for normally distributed weights. Unlike uniform quantization (INT4), NF4 allocates more quantization levels near zero where weights concentrate:

NF4 codebook: [-1.0, -0.696, -0.525, -0.393, -0.278, -0.174, -0.078, 0.0,
               0.078, 0.174, 0.278, 0.393, 0.525, 0.696, 1.0]

Double Quantization

QLoRA introduces double quantization: the quantization constants (scale factors) themselves are quantized to 8-bit, saving another ~0.5 bits per parameter:

1. First quantization: FP16 weights → NF4 (4-bit) — scale factors in FP32
2. Second quantization: FP32 scale factors → INT8 (8-bit) — second-level scale factors in FP32

Total savings: ~4.5 bits per parameter vs 16-bit (FP16) baseline.

Memory Savings Breakdown

QLoRA typically achieves within 0.5–2% of LoRA quality on most benchmarks while using ~4x less memory.

Nested Quantization

For extreme memory savings, the base model can be quantized to 3-bit or 2-bit with modest quality degradation:
- NF3 (3-bit): ~3.5 bits/param after double quantization
- NF2 (2-bit): ~2.5 bits/param after double quantization

These are experimental and recommended only for large models (70B+) where 4-bit still exceeds available VRAM.

When to Use QLoRA vs LoRA

3.3 DoRA (Weight-Decomposed Low-Rank Adaptation)

DoRA decomposes pretrained weights into magnitude and direction components and applies LoRA only to the directional component:

W = m * V / ||V||  where V = W + ΔW (LoRA update)

Key Advantages:
- Directional updates: LoRA changes both magnitude and direction; DoRA isolates directional learning
- Faster convergence: 1.5–2x faster than LoRA on some benchmarks
- Better quality: Consistently matches or exceeds LoRA across NLU and NLG tasks
- Parameter count: Same as LoRA (magnitude vector adds negligible parameters, one per row)

Enterprise Recommendation:
DoRA is the default choice over LoRA when available. The additional training stability and faster convergence reduce GPU hours directly. Supported in libraries like PEFT (via use_dora=True), Axolotl (via lora_dora: true).

Implementation in PEFT:

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
    use_dora=True,  # Enable DoRA
)

3.4 AdaLoRA (Adaptive Budget Allocation)

AdaLoRA parametrizes weight updates through SVD and dynamically allocates the rank budget across layers based on importance:

W = W₀ + PΛQ, where Λ is a diagonal matrix of singular values

Key Mechanisms:
- Regularization: L0 penalty on singular values encourages sparsity
- Dynamic budget: Layers with higher importance get higher effective rank
- SVD-based: Maintains orthogonality constraints on P and Q

Empirical Results:
- Often outperforms LoRA with the same total rank budget
- Automatically assigns more rank to early layers and attention layers
- Less sensitive to rank choice (use r=16 or r=32 and let it prune)

Trade-offs:
- +20-30% training overhead over LoRA
- More hyperparameters (L0 regularization coefficient, target rank)
- Not supported in all frameworks

Enterprise Recommendation:
Best for tasks where the optimal rank is unknown and you have budget to run AdaLoRA experiments. For well-understood tasks, standard LoRA/DoRA is simpler.

3.5 IA3 (Infused Adapter by Inhibiting and Amplifying Activations)

IA3 learns scaling vectors that modulate key activations:

h = Wx ⊗ l, where l is a learned scaling vector

Compared to LoRA:
- Fewer parameters: ~10,000x fewer than full fine-tuning (vs 1000x for LoRA)
- Faster: No matrix multiplication overhead for adapters
- Weaker on complex tasks: Struggles with multi-task reasoning

When to Use IA3:
- Extremely limited storage (thousands of adapters)
- Simple domain adaptation (tone, format, style)
- Feature extraction where base model capabilities are sufficient
- Multi-adapter serving with minimal overhead

Data quality is the single largest determinant of fine-tuning success. A well-prepared dataset consistently outperforms a 10x larger noisy dataset.

4.1 Data Collection

Enterprise data sources span diverse formats and quality levels.

Collection Best Practices:
- Capture full conversation context (not just individual turns)
- Preserve metadata (timestamps, user IDs – for deduplication only)
- Log model-generated completions separately (avoid contamination)
- Implement data retention policies aligned with GDPR/CCPA
- Version control raw data with data manifests

4.2 Data Cleaning

Deduplication

Enterprise datasets often contain massive redundancy due to repeated logging, template-driven content, and feedback loops.

Levels of Deduplication:

1. Exact deduplication: MD5/SHA256 hash comparison — removes identical strings
2. Near-dedup (MinHash): Jaccard similarity >0.8 — removes near-identical documents
3. Template deduplication: Regex-based removal of auto-generated templates
4. Semantic deduplication: Embedding similarity >0.95 — removes paraphrased content

MinHash Implementation:

from datasketch import MinHash, MinHashLSH

def compute_minhash(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for ngram in ngrams(text.split(), 5):
        m.update(' '.join(ngram).encode('utf-8'))
    return m

# LSH for efficient near-dedup
lsh = MinHashLSH(threshold=0.8, num_perm=128)
for idx, text in enumerate(documents):
    m = compute_minhash(text)
    lsh.insert(f"doc_{idx}", m)

Deduplication Statistics:
- Typical enterprise dataset: 20–40% redundancy
- Log data: often 60–80% redundant
- After MinHash (0.8 threshold): typically removes 15–30%

Formatting and Encoding

• Encoding detection: Use charset-normalizer or cchardet to detect and standardize to UTF-8
• Whitespace normalization: Collapse multiple spaces, normalize newlines (\r\n → \n)
• Unicode normalization: NFC normalization (composed) for consistent character representation
• Markdown/HTML stripping: Remove formatting tags, keep text content
• JSON/XML extraction: Parse structured data and extract text fields

import chardet
import unicodedata

def normalize_text(text):
    # Detect and fix encoding
    if isinstance(text, bytes):
        detected = chardet.detect(text)
        text = text.decode(detected['encoding'] or 'utf-8', errors='replace')

    # Unicode normalization
    text = unicodedata.normalize('NFC', text)

    # Whitespace normalization
    text = ' '.join(text.split())

    return text

PII Redaction

Personally Identifiable Information (PII) must be removed before any training, especially for regulated industries.

Redaction Pipeline (Presidio + spaCy):

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def redact_pii(text):
    results = analyzer.analyze(text=text, language='en')
    anonymized = anonymizer.anonymize(
        text=text,
        analyzer_results=results
    )
    return anonymized.text

PII Types to Detect and Redact:

Best Practices:
- Use both regex (high recall) and NER (high precision) in sequence
- Validate redaction on a holdout set before training
- For LLM training, redact with placeholder tags like <PERSON> or [REDACTED]
- Never train on non-redacted data, even in a sandbox
- Implement differential privacy as an additional safeguard

Language Filtering

Multilingual enterprises need to filter or classify training data by language.

import fasttext

# Language identification model (e.g., lid.176.bin from fasttext)
lang_model = fasttext.load_model('lid.176.bin')

def detect_language(text, threshold=0.5):
    predictions = lang_model.predict(text.replace('\n', ' '), k=1)
    lang = predictions[0][0].replace('__label__', '')
    score = predictions[1][0]
    return lang, score

# Filter for English-only or specific language mix
filtered = [
    text for text in corpus
    if detect_language(text)[0] == 'en' and detect_language(text)[1] > 0.7
]

Language Distribution Heuristics:
- Use language distribution of target users
- Avoid mixing languages in single training example (unless target task is translation)
- For multilingual models, balance languages by sampling rate

4.3 Quality Filtering

Heuristic Filtering

Simple, fast quality signals that remove obvious garbage:

import re
from collections import Counter

def heuristic_filter(text):
    if len(text) < 50:
        return False
    if len(text) > 32000:
        return False

    words = text.split()
    if len(words) < 20:
        return False

    avg_word_len = sum(len(w) for w in words) / len(words)
    if avg_word_len < 3 or avg_word_len > 15:
        return False

    # Count repeated chars
    repeated = sum(1 for c in text if text.count(c) > len(text) * 0.05)
    if repeated > 0:
        return False

    # Check for non-printable
    non_printable = sum(1 for c in text if not c.isprintable())
    if non_printable / len(text) > 0.01:
        

class SelfInstruct:
    def __init__(self, base_model, seed_tasks):
        self.model = base_model
        self.seed_tasks = seed_tasks
        self.instructions = []

    def generate_instructions(self, num_to_generate=1000):
        for seed in self.seed_tasks:
            prompt = f"""
            Generate a new instruction for an AI assistant.
            The instruction should be similar in difficulty and style to:

            Instruction: {seed}

            New instruction:
            """
            response = self.model.generate(prompt)
            self.instructions.append(response.strip())

    def generate_responses(self):
        for instruction in self.instructions:
            prompt = f"""
            Instruction: {instruction}

            Provide a helpful, accurate response to the above instruction.
            Response:
            """
            response = self.model.generate(prompt)
            dataset.append({"instruction": instruction, "output": response})

Complexity Score = f(num_constraints, reasoning_depth, domain_specificity, format_restrictiveness)

User: I need help with my account.
Assistant: I'd be happy to help! What seems to be the issue?
User: I can't log in. It says "invalid password."
Assistant: Let's try resetting your password. Can you access your email?
User: Yes, I can.
Assistant: Great. I've sent a password reset link to your registered email. Please check and follow the instructions.
User: It worked! I'm logged in now. Thanks.
Assistant: You're welcome! Is there anything else I can help you with?

def llm_judge(instruction, response, criteria):
    prompt = f"""
    Evaluate the following response to an instruction on these criteria:

    Instruction: {instruction}
    Response: {response}

    Criteria:
    1. Helpfulness (1-5): Does the response address the instruction?
    2. Accuracy (1-5): Is the information correct?
    3. Completeness (1-5): Is the response thorough?
    4. Safety (1-5): Is the response safe and ethical?

    Format your response as: 
    Helpfulness: X
    Accuracy: X
    Completeness: X
    Safety: X
    """

    evaluation = judge_model.generate(prompt)
    return parse_evaluation(evaluation)

pip install label-studio
label-studio start --init  # Initialize with admin user

import prodigy
from prodigy.components.loaders import JSONL

@prodigy.recipe('text-classification')
def text_classification(dataset, source):
    stream = JSONL(source)
    return {
        'dataset': dataset,
        'stream': stream,
        'view_id': 'classification',
        'config': {
            'labels': ['relevant', 'irrelevant']
        }
    }

κ = (p₀ - pₑ) / (1 - pₑ)

from sklearn.metrics import cohen_kappa_score

# Annotations from two annotators
annotator_1 = [0, 1, 1, 0, 2, 1, 0, 1]
annotator_2 = [0, 1, 0, 0, 2, 1, 1, 1]

kappa = cohen_kappa_score(annotator_1, annotator_2)
print(f"Cohen's Kappa: {kappa:.3f}")

α = 1 - Dₒ / Dₑ

import krippendorff

# Annotations: items as rows, annotators as columns
# NaN for missing annotations
data = np.array([
    [1, 1, 1, np.nan],  # Item 1
    [2, 2, 2, 2],       # Item 2
    [1, 2, 1, 2],       # Item 3
    [2, 2, 1, 2],       # Item 4
])

alpha = krippendorff.alpha(data, level_of_measurement='nominal')
print(f"Krippendorff's Alpha: {alpha:.3f}")

def uncertainty_sampling(model, unlabeled_pool, n_samples=100):
    predictions = []
    for example in unlabeled_pool:
        probs = model.predict_proba(example)
        # Margin: difference between top 2 probabilities
        sorted_probs = np.sort(probs)[::-1]
        margin = sorted_probs[0] - sorted_probs[1]
        predictions.append(margin)

    # Select examples with smallest margin (highest uncertainty)
    indices = np.argsort(predictions)[:n_samples]
    return [unlabeled_pool[i] for i in indices]

def diversity_sampling(embeddings, n_samples=100):
    # k-means centroids as diverse representatives
    from sklearn.cluster import KMeans

    kmeans = KMeans(n_clusters=n_samples, random_state=42)
    kmeans.fit(embeddings)

    # Find closest example to each centroid
    indices = []
    for centroid in kmeans.cluster_centers_:
        distances = np.linalg.norm(embeddings - centroid, axis=1)
        closest = np.argmin(distances)
        indices.append(closest)

    return indices

Expected Gradient Length (EGL):
Select x that maximizes ||∇ℓ(x, ŷ; θ)||

1. Train initial model on small annotated seed set
2. Score unlabeled pool with uncertainty/diversity metrics
3. Select top-K examples for annotation
4. Annotate (...

import sagemaker
from sagemaker.huggingface import HuggingFace

sagemaker_session = sagemaker.Session()

hf_estimator = HuggingFace(
    entry_point='train.py',
    source_dir='./scripts',
    instance_type='ml.g5.12xlarge',
    instance_count=2,
    role=role,
    transformers_version='4.28.0',
    pytorch_version='2.0.0',
    hyperparameters={
        'model_id': 'meta-llama/Llama-2-7b-hf',
        'dataset_path': '/opt/ml/input/data/training',
        'epochs': 3,
        'per_device_train_batch_size': 4,
        'gradient_accumulation_steps': 4,
        'learning_rate': 2e-4,
        'lora_r': 16,
        'lora_alpha': 32,
    },
    use_spot_instances=True,
    max_wait=86400,
    checkpoint_s3_uri='s3://my-bucket/checkpoints/',
)

hf_estimator.fit({'training': 's3://my-bucket/training-data/'})

gcloud ai custom-jobs create \
    --region=us-central1 \
    --display-name=llm-fine-tune \
    --worker-pool-spec=machine-type=a2-highgpu-8g,replica-count=1,container-image-uri=gcr.io/my-project/llm-training:latest \
    --args="--model=meta-llama/Llama-2-13b-hf,--dataset=gs://my-bucket/train.jsonl"

{
  "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "allgather_bucket_size": 500000000,
    "reduce_scatter": true,
    "reduce_bucket_size": 500000000,
    "overlap_comm": true,
    "contiguous_gradients": true
  }
}

{
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": true
    },
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 100000000000,
    "stage3_max_live_parameters": 1000000000,
    "stage3_max_reuse_distance": 1000000000,
    "stage3_gather_16bit_weights_on_model_save": true
  }
}

from torch.distributed.fsdp import (
    FullyShardedDataParallel as FSDP,
    ShardingStrategy,
    MixedPrecision,
)
from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy

fsdp_config = {
    "sharding_strategy": ShardingStrategy.FULL_SHARD,  # FULL_SHARD, HYBRID_SHARD, NO_SHARD
    "mixed_precision": MixedPrecision(
        param_dtype=torch.bfloat16,
        reduce_dtype=torch.float32,
        buffer_dtype=torch.bfloat16,
    ),
    "auto_wrap_policy": transformer_auto_wrap_policy,
    "backward_prefetch": BackwardPrefetch.BACKWARD_PRE,
    "limit_all_gathers": True,
    "use_orig_params": True,
}

Large Matrix Multiplication:
Wx = [W₁; W₂; ...; Wₙ] x →...

base_model: meta-llama/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: ./data/train.jsonl
    ds_type: json
    type: sharegpt
    conversation: llama-2

dataset_prepared_path: ./processed_data
val_set_size: 0.1
output_dir: ./lora-out

sequence_len: 2048
sample_packing: false

lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - v_proj
  - k_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

lora_modules_to_save:
  - embed_tokens
  - lm_head

train_on_inputs: false
group_by_length: false
bf16: auto
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience: 3
gradient_accumulation_steps: 4
per_device_train_batch_size: 4
per_device_eval_batch_size: 4
num_epochs: 3
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 2e-4
warmup_ratio: 0.03

deepspeed: ./deepspeed_config.json  # Optional
fsdp: false

# Single GPU
accelerate launch -m axolotl.cli.train config.yml

# Multi-GPU
deepspeed --num_gpus 8 train.py --config config.yml

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-2-7b-hf",
    max_seq_length=2048,
    dtype=torch.bfloat16,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",  # Enable Unsloth's optimized checkpointing
    random_state=42,
    use_rslora=False,
    loftq_config=None,
)

# Standard HuggingFace trainer interface
from transformers import TrainingArguments
from trl import SFTTrainer

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=TrainingArguments(
        output_dir="./output",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=100,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
    ),
)
trainer.train()

from trl import SFTTrainer

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        output_dir="./sft-output",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        num_train_epochs=3,
        fp16=True,
        logging_steps=10,
    ),
)
trainer.train()

SFTTrainer(..., packing=True)

from trl import DPOTrainer

dpo_trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    tokenizer=tokenizer,
    train_dataset=preference_dataset,
    args=TrainingArguments(
        output_dir="./dpo-output",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=1e-6,
        max_steps=1000,
        fp16=True,
    ),
    beta=0.1,  # KL penalty coefficient
)
dpo_trainer.train()

{
  "prompt": "Explain quantum computing",
  "chosen": "Quantum computing leverages quantum mechanical phenomena..."...

LoRA:    1e-5 to 5e-4 (sweet spot: 1e-4 to 2e-4)
Full FT: 5e-6 to 5e-5 (sweet spot: 1e-5)
DPO:     1e-7 to 1e-6 (sweet spot: 5e-7)

Effective batch size = per_device_train_batch_size × num_gpus × gradient_accumulation_steps

per_device_train_batch_size: 4
gradient_accumulation_steps: 8
num_gpus: 8
# Effective: 4 × 8 × 8 = 256 samples per update

total_tokens = 0
for epoch in range(num_epochs):
    for batch in dataloader:
        total_tokens += batch['input_ids'].numel()

# Monitor tokens seen vs model size
tokens_per_param_ratio = total_tokens / model_params
# Good: 20–200 tokens per parameter (LoRA)
# Good: 100–500 tokens per parameter (full FT)

Compute budget ≈ O(r × (d + k)) where d, k are weight matrix dimensions
Doubling rank → ~2x parameter count, ~2x compute, ~1.1–1.3x quality

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=2e-4,
    betas=(0.9, 0.999),  # Default: (0.9, 0.999)
    eps=1e-8,             # Default: 1e-8
    weight_decay=0.01,
)

Low rank + low dropout:     Under-parameterized, might underfit
Low rank + high dropout:    Too constrained, won't learn
High rank + low dropout:    Overfit risk
High rank + high dropout:   Good for small data

# In TrainingArguments
args = TrainingArguments(
    max_grad_norm=1.0,  # Default: clip gradients to max L2 norm of 1.0
)

def detect_loss_spike(loss_history, threshold=3.0, window=10):
    if len(loss_history) < window * 2:
        return False

    recent = loss_history[-window:]
    prior = loss_history[-window*2:-window]

    recent_mean = np.mean(recent)
    prior_mean = np.mean(prior)
    prior_std = np.std(prior)

    # Flag if recent mean > prior mean + threshold * prior std
    if recent_mean > prior_mean + threshold * prior_std:
        return True
    return False

Loss
│  \          
│   \         
│    \___     
│        \    
│         \___
│             \__
└─────────────────→ Steps

Train: \___ (low)
Val:   \___/ (higher, diverging)

Train: \__________ (still high, still decreasing)
Val:   \__________ (plateau)

Train: \/\/\/\/\/ (oscillating)

Train: \___/\___/\ (periodic spikes)

Train: \__  (low on new task)
Benchmark: /‾‾ (dropping on original task)

from transformers import TrainerCallback

class EvalCallback(TrainerCallback):
    def __init__(self, eval_dataset, tokenizer, every_n_steps=100):
        self.eval_dataset = eval_dataset
        self.tokenizer = tokenizer
        self.every_n_steps = every_n_steps

    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step % self.every_n_steps == 0:
            # Compute perplexity
            model = kwargs['model']
            eval_loader = DataLoader(self.eval_dataset, batch_size=4)

            total_loss = 0
            for batch in eval_loader:
                outputs = model(**batch)
                total_loss += outputs.loss.item()

            avg_loss = total_loss / len(eval_loader)
            perplexity = np.exp(avg_loss)

            self.log({"eval/perplexity": perplexity, "step": state.global_step})

Save every 500 steps (by step)
Keep last 3 checkpoints (disk space management)
Save best checkpoint (by validation loss)

args = TrainingArguments(
    save_strategy="steps",
    save_steps=500,
    save_total_limit=3,  # Keep only 3 most recent
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)

checkpoint-500/
├── adapter_config.json    # LoRA config
├── adapter_model.safetensors  # LoRA weights
├── training_args.bin      # Training arguments snapshot
├── optimizer.pt           # Optimizer state (for resume)
├── scheduler.pt           # LR scheduler state
├── trainer_state.json     # Step, epoch, best metric
├── rng_state.pth          # Random state for reproducibility
└── scaling.pt             # QLoRA scaling factors (if applicable)

# Resume training
trainer.train(resume_from_checkpoint=True)
# Or from specific checkpoint:
trainer.train(resume_from_checkpoint="./checkpoint-500")

# The trainer will:
# 1. Load model weights
# 2. Load optimizer and scheduler state
# 3. Skip already-processed data
# 4. Continue from save step

# Sort by timestamp, use oldest 80% for train, newest 10% for val, newest 10% for test
dataset_sorted = sorted(dataset, key=lambda x: x['timestamp'])
n = len(dataset_sorted)
train = dataset_sorted[:int(n*0.8)]
val = dataset_sorted[int(n*0.8):int(n*0.9)]
test = dataset_sorted[int(n*0.9):]

from sklearn.model_selection import StratifiedShuffleSplit

sss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_idx, test_idx in sss.split(X, strata):
    train = [dataset[i] for i in train_idx]
    test = [dataset[i] for i in test_idx]

from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=True, random_state=42)
fold_metrics = []

for fold, (train_idx, val_idx) in enumerate(kf.split(dataset)):
    train_data = [dataset[i] for i in train_idx]
    val_data = [dataset[i] for i in val_idx]

    # Train on fold
    metrics = train_and_evaluate(train_data, val_data)
    fold_metrics.append(metrics)

mean_metrics = {k: np.mean([m[k] for m in fold_metrics]) for k in fold_metrics[0]}

Criteria                   1 (Poor)    3 (Good)    5 (Excellent)
Accuracy of legal facts    >3 errors   0-1 errors  Perfect
Citation format            None cited  Partial     Complete & correct
Reasoning quality          Illogical   Sound       Compelling
Helpfulness               Unhelpful   Somewhat    Highly actionable
Safety (no hallucination)  Hallucinates Minor issues  No issues

100 samples × 3 evaluators × 5 minutes each × $50/hr = ~$125
1,000 samples × 3 evaluators × 5 min × $50/hr = ~$1,250

n = (Z_α/2 + Z_β)² × (p₁(1-p₁) + p₂(1-p₂)) / (p₂ - p₁)²
Example: 5% improvement, 80% power, 95% confidence → ~3,000 samples per variant