(2015. 8) Luong Attention

• published in 2015. 8

• Minh-Thang Luong, Hieu Pham and Christopher D. Manning

Simple Summary

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.

images

• concat is Bahdanau Attention mechanism

• local attention mechanism selectively focuses on a small window of context (avoiding the expensive computation)

images

• The location-based function does not learn good alignments.

• It is interesting to observe that dot works well for the global attention and general is better for the local attention.