Bilingual

Reduce GVisor Cold Starts with GPU Snapshotting
4 days ago
- GPU cold starts for AI models in production can cause long startup times, affecting scaling and resource utilization.
- Cerebrium reduces cold starts by over 80% using CPU and GPU memory snapshots to restore fully initialized containers in seconds.
- Checkpointing involves pausing execution, dumping CPU/GPU memory to files, uploading to storage, and restoring on demand.
- High-level architecture includes a checkpoint service and modified gVisor containerd shim to decide between normal boot or restore.
- Restoration times are fast, e.g., 2.25 seconds from S3 for a 9GB container, compared to 50 seconds for a full cold start.
- Real-world challenges include handling network state, multiprocessing, local runtime files, and ensuring compatibility across environments.
- Benchmarks show Cerebrium reduces cold starts by an average of 71% compared to without snapshots, and is faster than competitors like Baseten and Modal.
- Checkpointing is ideal for workloads with deterministic initialization like framework imports, model loading, and CUDA graph capture.
Popping the GPU Bubble
6 days ago
- Pipelined decoding overlaps GPU and CPU work to avoid GPU bubbles, where the GPU waits for CPU housekeeping.
- Three mechanisms ensure safety: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie refcounting manages finished requests.
- Performance gains depend on GPU speed and batch size, with up to 35% improvement on faster hardware as bubbles become more significant.
FlashAttention-4: Algorithm and Kernel Pipelining
2 days ago
- Asymmetric hardware scaling on modern GPUs like Blackwell leads to tensor core throughput growing faster than resources like shared memory bandwidth and special function units (SFUs).
- FlashAttention-4 is a co-designed algorithm and kernel for the Blackwell architecture, achieving up to 1605 TFLOPs/s on B200 with BF16, outperforming cuDNN and Triton.
- Key innovations include new pipelining for overlap, software emulation of exponential via polynomial approximation to mitigate SFU bottlenecks, and TMEM usage to reduce shared memory traffic.
- Blackwell hardware features like Tensor Memory (TMEM), fully asynchronous 5th gen tensor cores, and 2-CTA MMA mode enable larger tiles, reduced traffic, and improved performance.
- The backward pass is shared-memory bandwidth limited; optimizations include 2-CTA MMA to halve traffic and reduce atomic adds, plus deterministic mode for reproducible training.
- Scheduling improvements address load imbalance from causal masking and variable sequence lengths via grid linearization and longest-processing-time-first (LPT) scheduling.
- FlashAttention-4 is implemented in CuTe-DSL, cutting compile times significantly, and has influenced optimizations in cuDNN versions 9.13 and later.

Hasty Briefsbeta

#gpu optimization

Reduce GVisor Cold Starts with GPU Snapshotting

Popping the GPU Bubble

FlashAttention-4: Algorithm and Kernel Pipelining