Shared Objects and Content Addressing: a Survey of Techniques
When shipping software and systems at scale, it's desirable to bundle libraries into files that can be shared between applications, in order to reduce the system size. However, this comes with tradeoffs: in order to share things, we have to organize. More concretely: this means for any applications that want to share libraries and data objects, they have to agree on file naming conventions.
This simple need leads to a much bigger problem: the organizational agreement problem doesn't scale. Individual package managers solve this problem in distinct ways. And which libraries and data objects themselves can be shared can be a point of contention: some systems are based on ideas like "major version number is enough" (which, spoiler, invariably creates problems). The end result? Packages from different ecosystems can't share dependencies; whole linux distributions become rapidly balkanized from each other because of library versioning and filename collision issues; and in the worst scenarios, it becomes impossible to install different versions of some software and libraries on a single system, due to name collisions.
There has to be a solution. Where else in computing have we seen (and solved) the problem of "there are many small variations of a piece of data, and we need to keep all of them, despite naming each one being an inhuman problem?" Right: version control. And what was ultimately the solution in version control? Content-addressing: hash the thing, and index the storage by that.
So can we use the same solution to make a new golden age where shared objects and shared libraries are both easy and reliable and conflict-free?
In this talk, we'll explore the problem space -- what hurdles are there to sharing? What really needs to count as unique? How can we wire up existing library loading systems to meet our goals (without rewriting the universe?) -- we'll survey some prior art, and we'll wrap up with some questions that we hope can lead to a better future for all!