Jurij Jukić

My name is Jurij (pronounced “Yuri”).

My aim is to mathematically understand recursive self-improvement. To do that, I first need to understand the way neural networks work. Hence my work on key search as a possible explanatory principle.

Further, how can we understand LLMs if we don’t have a mathematical model of language? So I am trying to do a full multi resolution analysis of language, analogously to how we do it for images. Tools probably necessary for that are information geometry, data driven wavelet decomposition, and hypergraphs.

I am inspired by logical depth as a key principle behind everything I do.

[Paper] Key Search Might Explain Neural Network Training

[Video] Brief History of Complexity; Logical Depth and Neural Networks

I go through the history of complexity: entropy, Kolmogorov complexity, Levin complexity, minimum description length, epiplexity, logical depth, and multiscale logical depth. I compare program length, runtime, and precision aspects of these theories. I relate them to neural network training dynamics and conjecture that logical depth is the most useful one.

[Post] Logical Depth as a Framework for Understanding Neural Networks

In this post I briefly introduce how we can use Charles Bennett’s logical depth as a general framework to understand neural networks. I find it incredibly rich, and one can apply it in many interesting ways when thinking about the training pipeline and interpretability. Intro - entropy and Kolmogorov complexity If one were to try to mathematically describe a neural network, neither entropy nor Kolmogorov complexity seem sufficient. Entropy could be described as a measure of average surprise. Order is, on average, very unsurprising, because we can easily predict where the particles are located (assuming a physics metaphor). Disorder is on average surprising, because we can never really predict where the next particle will show up. Neither end of the spectrum of entropy, order or disorder, seems to capture the intelligence that is contained within a neural network. ...