Cooperative Vectors Introduction
a day ago
- #Rendering Engine
- #Neural Networks
- #Cooperative Vector
- Neural Networks (NN) usage in rendering engine began in 2021, initially for Neural Materials (NM) with offline training and hardware-accelerated inference via early Vulkan extensions.
- Framework expanded to support runtime training for Neural Radiance Caching (NRC), handling small NNs with features like 16, 32, or 64 widths, despite platform-specific hardware acceleration challenges.
- Cooperative Matrix extension (SPV_NV_cooperative_matrix, now KHR) abstracted matrix operations for hardware acceleration, while DirectX's WaveMatrix (Shader Model 6.8) was introduced but not fully released.
- Cooperative Vector (VK_NV_cooperative_vector) addressed divergent data challenges in Neural Materials and Neural Texture Compression (NTC), enabling vector-matrix operations for varied weights per pixel.
- Long vectors in HLSL are transparent data structures stored in VGPRs, accessible across shader stages, with each thread owning its vector for efficient handling of divergent inputs like in NRC.
- Matrices come in plain layouts (row/column major) and optimal layouts (MulOptimal for inference, OuterProductOptimal for training), with optimal layouts being opaque and requiring conversion for accessibility.
- Inference uses MulOptimal layouts via matrix conversions and operations like MatMul or MatMulAdd, while training involves gradients computed with cooperative vectors for input, weights, and bias updates.
- Future developments include DirectX's shift to linear algebra features combining Cooperative Vector and Matrix, and Vulkan's promotion of Cooperative Matrix to KHR with Cooperative Vector remaining NVIDIA-specific.