2 months ago
- Tensors are flat arrays of numbers with metadata for interpreting them as multi-dimensional objects, including shape and strides for efficiency.
- A basic tensor library in C includes structs for shape and tensor, with helpers for creation, indexing, and memory management using reference counting for views.
- Elementwise operations (unary and binary) are implemented for both CPU and GPU, with Metal used for GPU acceleration on Apple devices, requiring data synchronization between devices.
- Matrix multiplication is optimized using CPU libraries like Accelerate and GPU kernels, supporting 2D and batched cases, with transposition handled efficiently.
- Manual backpropagation is implemented for neural networks, starting with a linear layer for forward and backward passes, and cross-entropy loss for classification tasks.
- An optimizer like SGD with momentum and gradient clipping is used to update parameters, and the library is tested on MNIST, achieving high accuracy quickly.
- The library is extensible with more layers and operations, written in C for control and performance, and available on GitHub as a single-header, practical tool for training models.