Load-sensitive Selective Pruning in Dense Retrieval

We introduce a load-aware selective embedding-pruning method to be used in dense retrieval systems. Our method adapts the embedding dimensionality at inference time based on per-query deadlines and the current queue occupancy. Under light load, full-dimensional embeddings maximise effectiveness; as load increases, dimensionality is adaptively reduced to maintain throughput and avoid query drops. Our approach is orthogonal to the specific dimensionality reduction strategy employed and operates independently of the embedding model. We evaluate our approach using both PCA-reduced embeddings and nested Matryoshka embeddings. Empirical results show that our load-aware strategy consistently achieves a better effectiveness-efficiency trade-off than static dimensionality reduction baselines across varying load conditions. As a by-product of this study, our experiments show that under selective pruning, the PCA-based approach consistently outperforms Matryoshka, indicating that specialised multi-representation training is not strictly required for robust load balancing in dense retrieval.