Satya Prakash Dash

Whether it is collartz conjecture or the random matrix univesality; the measurement problem of quantum mechanics or the left-handedness of neutrinos, universe is hinting toward some deeper structure which mathematics haven’t found out yet!

Hi 👋🏻, I am a final year PhD student in the Department of Computer Science at the University of Manchester. My PhD work focuses on understanding the training dynamics of LLMs under natural gradients and propose scalable, generalizable and robust training methods. “Natural” means it estimates the curvature of the loss function induced in the parameter space to estimate the sensitivity of each parameter of LLMs (and “accelerates” gradient descent).

From optimization perspective, I am interested in non-convex optimization (overparametrized NN setting and reinforcement learning). From numerical linear algebra perspective I am interested in scalable solutions to inversion of matrices and inverse square root of matrices (because that all we need to “accelerate” gradient descent). From empirical perspective, I am interested in developing methods on how Fisher Information Matrix can help improve generalizability of trained LLMs and reduce hallucinations and alignment issues.