Skip to main content

Pickle formats and missing Go libraries, two shapes of the same ache

Share:XLinkedInHN
Cover for Pickle formats and missing Go libraries, two shapes of the same ache

Two kinds of night

There is a particular quiet that settles at 3 AM when a program will not do the thing you spent the day teaching it to do. I know two flavors of that quiet well.

The first flavor is Python. A file sits on disk. You wrote it out yourself, hours or weeks ago, with a line so short it feels like it could not possibly go wrong. Something like pickle.dump(model, f) or torch.save(state, path). The bytes are there. ls says so. stat gives you a size. And the program that is trying to read them back is unhappy, in that Python way where the traceback keeps unfolding down through frames you did not write, into modules whose names you half-remember, and lands on a line that says the object cannot be reconstructed.

The second flavor is Go. You have a file open in your editor with a package name at the top and an import block below it, and one of the lines in that block is red. You want a thing that does something Python developers stopped noticing years ago because they take it for granted. A specific solver. A specific tokenizer. A specific matrix operation that lives inside a specific package on PyPI. And you are staring at pkg.go.dev and the search results are three abandoned repositories with the last commit dated 2019, or one active repository that solves a related but distinct problem, or nothing at all.

Both nights feel the same. This is a note about why.

What the pickle actually refuses to do

The thing worth saying out loud is that a pickle file is not really a file. It is a small program in a bytecode designed to be executed by a specific version of the CPython interpreter, running with a specific set of modules on sys.path, using a specific set of class definitions that must be importable under the exact fully qualified names they had at the moment of serialization. If any of that has drifted, the file cannot come back.

The failure modes are all versions of that same fact. You get an UnpicklingError because a class was renamed. You get a ModuleNotFoundError because the module the class lived in was moved when someone reorganized the package. You get an AttributeError: Can't get attribute 'Foo' on <module 'x'> because the class still exists but not at that path any more. You get a version mismatch because a library changed the shape of one of its internal state dicts between minor releases and did not think anyone would notice. You get a _pickle.UnpicklingError: invalid load key because the file was written by a newer protocol than the reader supports.

None of these are format bugs. They are all the same bug. The bug is that pickle is asking the future to remember more than the past told it to remember.

The workarounds are all shaped like reconstructing a small archaeology. You pin the exact library version the file was born under. You write a stub class at the old fully qualified path so the loader has something to bind to. You subclass pickle.Unpickler and override find_class to redirect the lookup. You install the old Python interpreter in a throwaway environment and re-serialize into something less fragile, like safetensors or a plain state dict of tensors keyed by name.

That last move is the interesting one, because it is a retreat. Safetensors was designed by people who had lived through this ache too many times. It refuses to store executable code. It refuses to store arbitrary Python objects. It stores tensors and metadata and nothing else, and the format is stable enough that a file written a year ago will load without asking the reader to reconstitute a moment in time. Every safetensors adoption is a small confession that pickle was asking for too much.

What the Go program actually cannot do

The Go pain is the opposite shape. Python's pain is that the past is too specific to reproduce. Go's pain is that the past never happened.

Python has a hundred people who each wrote a niche package because they needed it, published it because it cost nothing, and now those hundred packages are how the whole ecosystem stays alive. Go has fewer of those people. The language attracts a different kind of programmer, one who is often being paid to build a service and is not likely to publish the tokenizer they wrote on the weekend. So when you go looking for the tokenizer, it is not there. Or the numerical routine is not there. Or the specific probabilistic data structure is not there.

You have a few moves. You can write it yourself, which is honest and slow and often good for the codebase. You can find a C library and bind to it through cgo, which drags a compiler and a set of platform assumptions into what was a clean cross-compile. You can shell out to a Python process and pipe data through, which works and feels wrong. You can pick a slightly different algorithm that a Go library does support and accept the accuracy hit or the speed hit.

The move that never really works is pretending the gap is not there. Every time I have tried, the missing library reappears as a subtler bug three weeks later, in the form of a numerical result that is close enough to look right and wrong enough to matter.

The abstraction both languages are pretending to share

Sitting between these two nights is a thing that people say confidently in interviews and then quietly stop believing after enough real projects. The thing they say is that serialization is a solved problem, and that any two general-purpose languages can talk to each other through it.

They cannot. Not really. A tensor written by PyTorch and read by a Go program is a stream of floats plus a shape, and that part is fine. But the model those tensors belong to is a computation graph, and the graph is expressed in the operators of a specific framework, and the framework is expressed in the idioms of a specific language, and by the time you have serialized the parts of the model that a Go program can plausibly consume, you have thrown away most of the thing that made the model a model.

This is why the two aches are the same ache. Pickle fails because it is trying to preserve too much of Python. Go stalls because it is trying to consume too little of Python. The pickle programmer wants the object graph to survive a trip through the disk. The Go programmer wants the object graph to survive a trip through a foreign language. In both cases the object graph is heavier than the wire, and the wire is where you find out.

What I have taken from both

I have started designing artifacts backwards from this. If a file is going to leave the process that made it, it should be shaped like something a stranger can read. Named tensors in safetensors. Config in TOML or JSON with an explicit schema. Tokenizer vocabularies as flat text or a well-specified binary. No pickled objects at boundaries, ever, even inside a system that is fully Python end to end, because the boundary might one day be crossed by something that is not.

The Go side of the same rule is quieter. If I need a piece of numerical work to live inside a Go service and the library does not exist, I try to write the minimum version myself before reaching for cgo or a subprocess. It is usually forty lines. It is usually enough. And it means the service stays a Go service, which is the whole point of having picked Go in the first place.

Neither of these habits stops the 3 AM nights entirely. They just make the nights rarer, and shorter, and less about digging up an old interpreter to appease a file that no longer trusts the present.

Cite as: Saravanan, K. (2026). Pickle formats and missing Go libraries, two shapes of the same ache. Kaushik Saravanan. https://www.kaushik.cv/blog/pickle-formats-and-missing-go-libraries