AI Notes

PPO (RLHF)

AI Notes

ai-notes-mkdocs

Home
Foundations
Foundations
- Overview
- Losses
  Losses
- Normalization
  Normalization
  - Overview
  - BatchNorm
  - LayerNorm
  - GroupNorm
  - RMSNorm
- Activation Functions
  Activation Functions
- Optimization
  Optimization
- Regularization
  Regularization
Models
Models
- LLM
  LLM
  - Overview
  - Core
    Core
    
    Q, K, V
    
    Attention
    
    Multi-Head Attention
    
    Transformer
    
    Tokenization
    
    Decoding & Sampling
    
    KV Cache (Basics)
  - Positional Encoding
    Positional Encoding
    
    Sinusoidal
    
    Relative PE
    
    Better Relative PE
    
    RoPE
    
    ALiBi
    
    RoPE Scaling / Long Context
  - Architectures
    Architectures
    
    Encoder-Decoder (T5)
    
    MoE (Switch / Router)
    
    Speculative Decoding (Concept)
  - Models
    Models
    
    BERT
    
    GPT
    
    GPT-2
    
    T5
    
    Switch Transformer
    
    Qwen3
    
    deepseek-v3
    
    deepseek-r1
  - Tasks
    Tasks
    
    Sequence Labeling
    Sequence Labeling
    
    NER
    
    BIO / BILOU
    
    Embeddings / Reranking
    
    Tool Use / Function Call
- Vision
  Vision
  - Overview
  - Backbones
    Backbones
    
    ViT
    
    Swin Transformer
  - Detection
    Detection
    
    DETR
    
    Grounding DINO
  - Segmentation
    Segmentation
    
    Mask2Former
    
    Mask DINO
    
    SAM
    
    Grounding-SAM
  - Tasks
    Tasks
    
    OCR / Document AI
    
    Video Understanding
- Audio
  Audio
  - Overview
  - Representations
    Representations
    
    Spectrogram / Mel / Codec Tokens
  - Speech Recognition (ASR)
    Speech Recognition (ASR)
    
    Whisper
  - Speech Synthesis (TTS)
    Speech Synthesis (TTS)
    
    Overview
    
    Neural Codec LM (VALL-E)
    
    Flow-Matching (Voicebox)
    
    Voice Cloning (Concepts)
  - Text-to-Audio / Music
    Text-to-Audio / Music
    
    AudioLDM
    
    MusicLM
    
    AudioLM
  - Speech Translation
    Speech Translation
    
    SeamlessM4T
  - Evaluation
    Evaluation
    
    WER / CER
    
    MOS / PESQ (Intro)
    
    FAD
- Video
  Video
  - Overview
  - Generation
    Generation
    
    Overview
    
    Sora
    
    Stable Video Diffusion
    
    Lumiere
    
    Temporal Consistency
  - Control & Editing
    Control & Editing
    
    Inpainting / Outpainting
    
    Motion / Camera Control
    
    Video-to-Video / Style Transfer
  - Evaluation
    Evaluation
    
    FVD
- 3D
  3D
  - Overview
  - Representations
    Representations
    
    Mesh / Point / SDF
    
    NeRF
    
    3D Gaussian Splatting
  - Generation
    Generation
    
    Text-to-3D (DreamFusion)
    
    Text-to-3D (3DGS)
    
    Novel View Synthesis
  - Evaluation
    Evaluation
    
    Chamfer / F-Score / Render Metrics
- VLM / Multimodal
  VLM / Multimodal
  - Overview
  - Connectors
    Connectors
    
    Projector / Adapter (Concepts)
    
    Q-Former (BLIP-2)
  - Contrastive
    Contrastive
    
    CLIP
    
    SigLIP-2
  - Generative
    Generative
    
    BLIP
    
    BLIP-2
    
    LLaVA
    
    Florence-2
    
    nanoVLM
    
    Qwen2.5-VL
  - Grounding
    Grounding
    
    Grounding DINO
    
    Grounding-SAM
  - Evaluation
    Evaluation
    
    VQA / MMMU (Intro)
- 數字人 (Digital Human)
  數字人 (Digital Human)
  - Overview
  - Face / Head
    Face / Head
    
    DECA
    
    GaussianAvatars
  - Full-body Avatars
    Full-body Avatars
    
    3DGS-Avatar
  - Talking Head / Lip Sync
    Talking Head / Lip Sync
    
    Wav2Lip
    
    Audio-Driven Face Animation
  - Voice & Safety
    Voice & Safety
    
    Voice Cloning Risks
    
    Consent / Deepfakes
    
    Watermarking / Provenance (Intro)
Generative Models
Generative Models
- Overview
- Diffusion
  Diffusion
  - Overview
  - VAE
    VAE
    
    VAE
    
    VQ-VAE
  - Samplers
    Samplers
    
    DDPM
    
    DDIM
    
    DPM-Solver
  - Guidance
    Guidance
    
    Classifier-Free Guidance
    
    Conditioning / Control (Intro)
  - Latent Diffusion
    Latent Diffusion
    
    LDM
    
    Stable Diffusion
  - DiT
  - HunyuanImage 2.1
- Flow / Rectified Flow
  Flow / Rectified Flow
  - Overview
- Autoregressive Generation
  Autoregressive Generation
  - Overview
- GAN (Optional)
  GAN (Optional)
  - Overview
Data
Data
- Overview
- Data Curation
  Data Curation
- Instruction / Preference Data
  Instruction / Preference Data
- Multimodal Data
  Multimodal Data
- Tooling
  Tooling
  - Datasets / WebDataset / HF Datasets
Training & Finetuning
Training & Finetuning
- Overview
- Pretraining
  Pretraining
- Finetuning
  Finetuning
- Alignment
  Alignment
- Distributed
  Distributed
Systems & Deployment
Systems & Deployment
- Overview
- Inference Optimization
  Inference Optimization
- Quantization & Compression
  Quantization & Compression
- Serving Runtimes
  Serving Runtimes
- Production
  Production
Agents
Agents
- Overview
- Foundations
  Foundations
- Patterns
  Patterns
- RAG
  RAG
Evaluation
Evaluation
- Overview
- Benchmarks
  Benchmarks
  - LLM Benchmarks (MMLU/MT-Bench/etc.)
  - VLM Benchmarks (VQA/MMMU/etc.)
- Metrics
  Metrics
- Reliability
  Reliability
  - Robustness / OOD
  - Calibration / Uncertainty
- Safety
  Safety