Glossary
Activation function — Glossary
A nonlinearity applied in neural networks so stacked layers can model complex functions.
Full definition
Common activations include ReLU variants and gating functions in transformers’ feed-forward blocks. Without nonlinearities, deep stacks collapse to linear maps.