Ask HN: Is Everything Data Compression?
After thinking about it for a while I am wondering, is everything data compression? Perhaps trivially so?
Like chaotic and nonchaotic systems alike can be described by their ability to be compressed. The less compressible the less chaotic [1]. And this is where we define explicitly chaos is in the regime of positive Lyapunov exponents. And so that creates an intrinsic link between dynamics and information. And information is what the universe is fundamentally made out of to say metaphorically.
But then that is why neural networks are so good at their jobs compared to anything else when predicting chaotic things [2] because they literally are attempting to compress the space of data. Which is a descriptor of chaos.
And the reason we say that neural networks are unexplainable or at least hard to explain is that they don’t show directly the solution. But if we switch our mental model to that the universe is simply only able to be described by literally compressing it into something, then neural networks are the best estimator for any function [3].
Which implies that neural networks are the best we can get. So all you can do is work towards more sophisticated architectures to understand anything. Because there isn’t anything that’s a true data model, function approximations are all we will ever get [4]. And also the reason they can predict further into the future than though possible is because previously we simply had worse compression algorithms than before.
And to me this all seems trivially true, or likely less than that because there seems to be no implications from it that are more than cursory observations and perhaps I am simply naive. So I am hoping someone can point me in the right direction to read or do more to understand these things. Because I have organized a conference to understand these ideas [5] and I think I should organize another one but also I’m not sure how to interact with these ideas because most of my time is spent doing geophysics and data science and not this.
[1] https://www.sciencedirect.com/science/article/abs/pii/S0370157301000254
[2] https://link.aps.org/doi/10.1103/PhysRevResearch.5.043252
[3] https://www.sciencedirect.com/science/article/abs/pii/S089360809700097X
[4] https://projecteuclid.org/journals/statistical-science/volume-16/issue-3/Statistical-Modeling--The-Two-Cultures-with-comments-and-a/10.1214/ss/1009213726.full
[5] conference talks on YouTube (not all the talks were recorded and also the conference was broader in topic than this post). https://youtube.com/playlist?list=PL6zSfYNSRHalAsgIjHHsttpYfxJ_XIPbt&si=VVWAE-fsv_WfFwfK
The universe is fundamentally made of potential, not information.
Information is the removal of uncertainty. If it does not remove uncertainty, it is not information. Information is state resolve, constructive and destructive interference collapse into state (information, contextually) in the “moment of now” (whichever moment evaluated.)
You have an interesting and worthwhile curiosity towards communication and compression, however I will say “everything is about” is nothing of the sort.
Everything is potential distributing over the infinite surface area of negative potential (/ information). We call this “entropy.”
Everything is “about” conservation and capacity (because we intend to burn it) of potential. Whatever potential is necessary at that moment.
So everything is about burning what potential we have wisely, and compression is one way of framing that requirement.
We are naturally lazy and ambivalent and will take up as much time and bandwidth as is pleasurable.
You should read Stephen Wolfram, The Second Law. He has a useful perspective on what can be predicted (computed) by an observer and what can't, it is more helpful than thinking about things in terms of compression. His views on physics-in-the-world phenomena are not correct, to be clear, but the mental models are useful.
there's something of a semantic tarpit where a concept is complete enough to be able to applied universally. compression is tied to incomputability (Kolgomorov Complexity) & true AI (both in terms of prediction & that finding structure in data is tied to understanding the data), so it's not surprising it fits this semantic tarpit
you could make a religion out of this! - By the guy behind the sun is a deadly lazer.