Modular Manifolds

When we train large neural networks, we need to keep them healthy. We do not want the tensors in the network—either the weights, activations, or gradients—to grow too large or too small. Very small and very large tensors cause a variety of problems not just limited to numerical underflow and overflo