Masked Diffusion Language Models (MDLMs) have recently emerged as a promising alternative to autoregressive models. Unlike standard diffusion-based approaches, MDLMs are not explicitly conditioned on a timestep, raising a natural question: do these models internally represent denoising progress, and how is such information used downstream? In this work, we show that MDLMs do in fact encode a latent representation related to the diffusion timestep within their residual streams. We find that this signal can be reliably extracted using probes across layers, indicating that denoising progress is decodable from internal activations. We further demonstrate that steering the model along a low-dimensional subspace associated with the inferred timestep allows us to systematically modulate its notion of denoising progress, leading to controlled and predictable changes in model confidence and entropy. Finally, we analyse the geometry of the identified representation, showing that it exhibits structured and interpretable properties in activation space, and shedding light on how such a signal is manipulated by these models.

Subliminal Clocks: Latent Time Modelling in Diffusion Language Models / Rulli, M., Fontanari, T.V., Petruzzi, S., Alvetreti, F., Strano, G., Crisostomi, D., Nikolaou, G., Mencattini, T., Santilli, A., Rodolà, E., Scardapane, S., Devoto, A.. - (2026), pp. 39904-39927. (Empirical Methods in Natural Language Processing Budapest; Hungary ) [10.18653/2026.emnlp-main.1749].

Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

Rulli, Maximo;Petruzzi, Simone;Alvetreti, Federico;Strano, Giorgio;Crisostomi, Donato;Mencattini, Tommaso;Santilli, Andrea;Scardapane, Simone;Devoto, Alessio
2026

Abstract

Masked Diffusion Language Models (MDLMs) have recently emerged as a promising alternative to autoregressive models. Unlike standard diffusion-based approaches, MDLMs are not explicitly conditioned on a timestep, raising a natural question: do these models internally represent denoising progress, and how is such information used downstream? In this work, we show that MDLMs do in fact encode a latent representation related to the diffusion timestep within their residual streams. We find that this signal can be reliably extracted using probes across layers, indicating that denoising progress is decodable from internal activations. We further demonstrate that steering the model along a low-dimensional subspace associated with the inferred timestep allows us to systematically modulate its notion of denoising progress, leading to controlled and predictable changes in model confidence and entropy. Finally, we analyse the geometry of the identified representation, showing that it exhibits structured and interpretable properties in activation space, and shedding light on how such a signal is manipulated by these models.
2026
Empirical Methods in Natural Language Processing
masked diffusion language models; interpretability; probing; activation steering; machine learning;
04 Pubblicazione in atti di convegno::04b Atto di convegno in volume
Subliminal Clocks: Latent Time Modelling in Diffusion Language Models / Rulli, M., Fontanari, T.V., Petruzzi, S., Alvetreti, F., Strano, G., Crisostomi, D., Nikolaou, G., Mencattini, T., Santilli, A., Rodolà, E., Scardapane, S., Devoto, A.. - (2026), pp. 39904-39927. (Empirical Methods in Natural Language Processing Budapest; Hungary ) [10.18653/2026.emnlp-main.1749].
File allegati a questo prodotto
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1781217
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact