Visual backward masking is a versatile tool for understanding principles and limitations of visual information processing in the human brain. However, the mechanisms underlying masking are still poorly understood. In the current contribution, the authors show that a structurally simple mathematical model can explain many spatial and temporal effects in visual masking, such as spatial layout effects on pattern masking and B-type masking. Specifically, the authors show that lateral excitation and inhibition on different length scales, in combination with the typical time scales, are capable of producing a rich, dynamic behavior that explains this multitude of masking phenomena in a single, biophysically motivated model.