Absorbing Dynamics and Uniform-Mixing Kernels

Combining Absorbing and Uniform-Mixing Kernels

The paper introduces a new design space by combining absorbing and uniform-mixing kernels with region-dependent rates. This approach allows missing positions to follow absorbing dynamics, while context undergoes low-rate random substitution during training. At inference, a heuristic sampler re-injects noise into suspicious context under an annealed intensity and freezes context updates late in the trajectory. This method aims to improve the reliability of diffusion models by detecting and mitigating corruption in generated text.

Training and Inference

The authors fine-tune a pretrained 170M-parameter masked diffusion model for 8,000 steps on a 6 GB laptop GPU. The base model assigns roughly 50% probability to observed tokens even when they are uniform-random garbage. For uniform-random corruption, finetuning detects corrupted context with 0.96 recall and 0.91 precision. The method also improves infill token-F1 by 3.8 points at 10% corruption and 6.4 points at 20%. Clean conditioning shows no measurable regression, with a false-edit rate below 0.4%. Gains for plausible corruptions are weaker, with about 0.5 F1 points at 10% corruption.

Trust as a Design Space

The paper interprets absorbing diffusion as the infinite-contrast limit of a trust axis, analogous to the signal-to-noise gap between known and unknown regions in diffusion inpainting. In their setting of a 170M model and a single text domain, intermediate trust is reached by changing forward dynamics rather than architecture.

Limitations and Future Work

The authors note that their results suggest that detectability and damage depend jointly on departure from the data conditional. They suggest that future work could explore different architectures or incorporate additional techniques to further improve the reliability of diffusion models.