Attention Quiz (Hard)

Which type of attention mechanism is often used in sequence-to-sequence tasks to align the elements of the source and target sequences during translation?

In the context of the Transformer model, which component uses multi-head self-attention?

Which mathematical operation is performed during the computation of attention scores in a typical attention mechanism?

What is the primary challenge of using self-attention mechanisms in very long sequences?

In multi-head attention, how are the attention weights typically combined from different heads?

How are attention weights typically calculated in an attention mechanism?

In the context of self-attention mechanisms, what is the purpose of the query, key, and value vectors?

In a multi-head attention mechanism, what do different attention heads learn?