You train the model with the loss applied at several prefix lengths at once: the first 64 dimensions, the first 128, and so on. This teaches it to pack the most important information at the start of the vector, like a set of nested matryoshka dolls.
Compression/loss, and the opposite - scaling up and oversmoothing - are fascinating in that any time even the tiniest innovation happens in those areas, all this other technology improves overnight, and a bunch of new technology becomes possible.
> You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.
Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation without losing accuracy. Ironically - almost the opposite of a PCA.
I do wonder if relevant here though. It relies on the embeddings having "structure", i.e. that principal components point along basis vectors, which may not be the case with text embeddings.
in my experience it's very similar to what you typically need to do in order to get the best results in other fields (like audio), where being really particular/deliberate in how you manicure and format the input can pay orders-of-magnitude dividends
sometimes adding noise over a frequency range is better than removing it entirely, the opposite tends to be true for text and especially code, where you'll want lists of language and framework keywords, and then a pass on top of that to scan the codebase itself for its own slang. You can then 'double dip' and use these lists to weight the results after the fact
Nice! I’ve been working on something similar and found similar results.
In my experiments, I used lots of embedding models and the results were not nearly as uniform as this curve, just FYI. I didn’t use any of the API-based models though
PCA is applied after the model, so there should be no difference in embedding throughput. Lookups in the index should be faster, but that speedup also applies equally to MRL.
PCA is good, but you could also try playing around with Sparse (robust) PCA. The sparsification loses orthogonality, but does not necessarily lose information, it can yield a different rotation and cleaner vectors. Now whether that matters in the context of LLMs/Embeddings - I cannot tell.
Nice! Thank you for doing this. I am not surprised PCA holds up. It comes with some pretty strong theoretical guarantees. The problem with it has always been scalability. The O(n^3) complexity makes it infeasible to use for massive datasets.
That is where something like Matryoshka embeddings has appeal. You trade a little bit of performance for a guarantee of training + validation set coverage.
Thanks for doing this benchmarking Dylan. I wanted to teach people PCA in my original article, but had no idea it would stack up this well against Matroyshka!
Feels like a “just use logistic regression” moment :)
This is fascinating if it works as well as the experiments make it seem. For example, how does it compare to classic image resize algorithms like Seam Carving or Inpainting: https://en.wikipedia.org/wiki/Seam_carving, https://en.wikipedia.org/wiki/Inpainting
(can they be compared?)
Compression/loss, and the opposite - scaling up and oversmoothing - are fascinating in that any time even the tiniest innovation happens in those areas, all this other technology improves overnight, and a bunch of new technology becomes possible.
> You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.
Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation without losing accuracy. Ironically - almost the opposite of a PCA.
I do wonder if relevant here though. It relies on the embeddings having "structure", i.e. that principal components point along basis vectors, which may not be the case with text embeddings.
Source: https://research.google/blog/turboquant-redefining-ai-effici...
sometimes adding noise over a frequency range is better than removing it entirely, the opposite tends to be true for text and especially code, where you'll want lists of language and framework keywords, and then a pass on top of that to scan the codebase itself for its own slang. You can then 'double dip' and use these lists to weight the results after the fact
In my experiments, I used lots of embedding models and the results were not nearly as uniform as this curve, just FYI. I didn’t use any of the API-based models though
I also wrote about this exact comparison when using PCA and MRL to quantize static models, see: https://stephantul.github.io/blog/mrl-pca/
I couldn't find much when I first looked into this, which is why I ended up writing the article.
So I guess the answer is: no
That is where something like Matryoshka embeddings has appeal. You trade a little bit of performance for a guarantee of training + validation set coverage.
Feels like a “just use logistic regression” moment :)