11-877: Advanced Topics in MultiModal Machine Learning (Fall 2022) - Carnegie Mellon University

Teaches architectural patterns for combining visual, audio, and textual modalities through cross-modal attention mechanisms, transformer-based fusion layers, and late/early/hybrid fusion strategies. Covers implementation of joint embedding spaces where heterogeneous data types are projected into shared representational spaces, enabling downstream tasks like visual question answering and video understanding through coordinated feature alignment.

Teaches design patterns for vision-language models (VLMs) including CLIP-style contrastive learning, image-text matching objectives, and transformer-based architectures that align visual and textual representations. Covers implementation of dual-encoder systems with shared embedding spaces, training strategies using contrastive losses (InfoNCE), and inference patterns for zero-shot classification and image-text retrieval.

Teaches design patterns for transformer-based multimodal models including vision transformers (ViT) for image encoding, text transformers for language understanding, and cross-attention mechanisms that enable interaction between modalities. Covers architectural choices like shared vs separate token spaces, positional encoding strategies for different modalities, and training techniques (masked language modeling, masked image modeling, contrastive learning) adapted for multimodal transformers.

Teaches temporal modeling approaches for video understanding including 3D CNNs (C3D), two-stream networks (spatial + temporal pathways), and transformer-based video encoders. Covers how to capture motion patterns through optical flow, frame sampling strategies, and temporal attention mechanisms that learn which frames are semantically important for action recognition and video classification tasks.

Teaches methods for learning and leveraging audio-visual synchronization, including cross-modal self-supervised learning where audio and video streams are used to supervise each other without labeled data. Covers synchronization detection (determining if audio and video are temporally aligned), audio-visual source separation (isolating individual speakers from mixed audio using visual cues), and learning joint representations through contrastive objectives that maximize agreement between aligned modalities.

Teaches methods for building retrieval systems that match queries in one modality (e.g., text) to candidates in another modality (e.g., images) using learned similarity metrics. Covers embedding-based retrieval where both modalities are projected into a shared space, ranking objectives like triplet loss and contrastive losses, and efficient indexing strategies (approximate nearest neighbor search) for scaling to millions of candidates while maintaining sub-second query latency.

Teaches principles of learning joint representations where different modalities are mapped into a shared embedding space that captures semantic relationships. Covers self-supervised learning objectives (contrastive, masked modeling), alignment losses that encourage modality-specific encoders to produce compatible embeddings, and evaluation metrics for measuring the quality of learned representations (downstream task performance, retrieval metrics, linear probe accuracy).

Teaches architectures and training strategies for visual question answering (VQA) systems that combine visual understanding with natural language reasoning. Covers attention mechanisms that identify relevant image regions for answering questions, fusion of visual features with question embeddings, and training objectives that handle multiple correct answers and answer frequency bias. Includes coverage of VQA datasets (VQA v2, GQA) and evaluation metrics (accuracy, BLEU, CIDEr).

Teaches methods for dense scene understanding including semantic segmentation (assigning class labels to every pixel), instance segmentation (distinguishing individual objects), and panoptic segmentation (unified segmentation of stuff and things). Covers encoder-decoder architectures with skip connections, multi-scale feature fusion, and how to leverage multimodal information (RGB-D, RGB-thermal) to improve segmentation accuracy in challenging conditions like low light or occlusion.

Teaches best practices for constructing and annotating multimodal datasets including data collection strategies, quality control mechanisms, inter-annotator agreement measurement, and handling of annotation disagreement. Covers practical considerations like managing multiple modalities with different temporal alignments, privacy-preserving data collection, and creating balanced datasets that avoid spurious correlations between modalities that models can exploit without learning robust representations.

Teaches evaluation methodologies for multimodal models including task-specific metrics (accuracy, F1, BLEU, CIDEr for different modalities), robustness evaluation under distribution shift, and analysis of what each modality contributes to predictions. Covers ablation studies that measure modality importance, adversarial robustness testing, and creation of diagnostic datasets that isolate specific capabilities (e.g., compositional reasoning, counting, spatial relationships).