Attention Deep Learning; Natural Language Processing
noun
Definition: A neural-network mechanism that dynamically assigns different weights to different parts of the input or context so that the model can focus on the most relevant information when producing an output. In sequence modeling, attention allows the decoder or a token representation to selectively use information from other positions instead of relying on a single fixed-size representation [ISO; Bahdanau et al. 2016].
Example in context: “Attention is an ultimate mechanism that allows neural networks to focus on any parts of the input, emphasizing its specific ones that are the most relevant to the task at hand.” [Gordeev et al. 2025]
Synonyms: attention mechanism
Related terms: self-attention; cross-attention; Transformer; encoder–decoder model; alignment