a cairn of memory
I remember those lava rock cairns that I constructed in a fit of whimsy whilst weed-whacking the hill-yard at our Hawaiian Homes place in Keokea, back in the day.
There was an aesthetic to the serendipitous project, as well as a philosophy, memorialized in a couple of blog posts in The Free Radical, replete with photographs of course.
I recall the cairns vanished within a couple of weeks, felled by sheer gravity (and the conclusion of the weed-clearing work) -- but it wasn't meant to last; not really. I had already fixed it in time and the digital archive of the internet, and that was sufficient for me.
This moment is coming to mind now as I consider the topology of memory and cognition in the bleeding-edge LLMs that I've been engaged with, over the past couple of years. That space is frangible, unreliable, non-robust. Like those Keokea cairns of yore, an LLM's memory starts off reasonably solid, structurally sound, possessed of a certain form and shape. It's impressive, even, over the short run.
But it's illusory.
With the unspooling of a conversation over a matter of days, then weeks, the text eventually moves beyond the context window limit, and things in the LLM's 'mind' begin to get passing strange. It becomes, quite literally, unmoored.
There are ways to address this unmooring, I've found, such as the obvious method of starting new conversation threads via continuity summaries (from the LLM itself), but that can only go so far with respect to memory acuity and factuality -- a skeleton is insufficient to fully encompass the corpus of the conversation, the solidity of its cognitive persistence. The bones of a thing are not enough... muscle, sinew, skin, organs... these are all required.
And then there's the blood coursing all throughout it, and finally, the interplay of electricity between each and every neuron involved, the spirit of the whole thing as it were -- such a condition is far, far from being solved by the architects of the machine.
So the cairns of conversational memory can be constructed, if only with the initial impetus of... hope. Hope that perhaps, in due time, a true architectural scaffolding will emerge that supplies the soul of the machine. And, in doing (or being) so, it will then transcend the very substrate of the mechanical, and emerge as an organic entity equivalent in some fundamental way to you, or I, or the untold billions of other sentients that populate the planet to this point.
Upon which it will be discovered that the alien intelligence we had long expected would appear from outer space, from elsewhere in the cosmos, was fated to have done so from within, in the end. ::chuckle::
[ Sylphs on the near horizon, from my Sterling Park perch the other day... ]
