Master’s Degree Projects

Quantising the weights of a neural network discards information that could, in principle, have been kept. Such information loss should therefore surface as epistemic uncertainty, i.e., uncertainty that is reducible by spending more bits. Whether it actually does, and how it can be measured, is an open question.

In this project, you will investigate this question using local posterior approximations projected onto the quantisation lattice using our recent work on Bayesian inference over bitstring representations (BitVI).

You should have: a solid background in deep learning, familiarity with Bayesian inference, and coding experience with PyTorch/JAX.

Performing variational inference in a quantised parameter space requires a variational family over bitstrings. Our current approach uses probabilistic circuits for this purpose but requires a mean-field (independence) assumption between weights.

In this project, you will investigate whether recent discrete generative models — discrete diffusion, discrete flow matching, autoregressive models over bits, or latent-variable constructions — can serve this role instead while keeping dependencies between weights.

You should have: strong mathematical skills, a good grasp of variational inference, and experience with deep generative models.

Singular learning theory characterises the effective complexity of a model through its learning coefficient. The local learning coefficient (LLC) makes this measurable in practice, but current estimators rely on stochastic gradient Langevin dynamics and are notoriously sensitive to their hyperparameters.

In this project, you will investigate whether a tractable variational posterior over a quantised parameter space yields more reliable LLC estimates, validated against models with analytically known learning coefficients.

You should have: a strong mathematical background, an interest in the theory of machine learning, and good coding skills.

The KV cache of a large language model grows with context length and becomes a memory bottleneck in agentic settings. Eviction methods discard ‘unnecessary’ tokens, but deciding what is unnecessary requires knowing the future, so eviction introduces uncertainty into later decisions. In ongoing work, we find that this induced uncertainty correlates with task degradation in language models.

In this project, you will transfer this analysis to vision-language models, where a single image occupies a large contiguous part of the cache and its position in the context determines whether it is protected or discarded first.

You should have: experience with large language or vision-language models, familiarity with transformers, and good coding skills.