# (2017. 3) Self Attn Sentence Embed

* Submitted on 2017. 3
* Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou and Yoshua Bengio

## Simple Summary

> Proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding.

![images](/files/-LRazKd3gR6bkP5sibGs)

* A self-attention mechanism for these sequential models to replace the max pooling or averaging step.
  * allows extracting different aspects of the sentence into multiple vector representations.
  * interpreting the extracted embedding becomes very easy and explicit.
* Model
  * bidirectional LSTM -> concat -> self-attention
  * the final sentence embedding to directly access previous LSTM hidden states via the attention summation.
  * `a = softmax(w_s2 tanh (W_s1 H^T))`
  * `M = AH` M: r-by-2u embedding matrix, A: annotation matrix, H: LSTM hidden states
  * Penalization Term: `P = ||(AA^T - I)||_F^2`

![images](/files/-LRazKd5G7MDjBhXggPt)

* Experiments
  * The Author Profiling dataset
  * Sentiment Analysis: Yelp and Age Dataset
  * Textual Entailment: the SNLI corpus
* able to encode any sequence with variable length into a fixed size representation, without suffering from long-term dependency problems.
* not able to train it in an unsupervised way.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://humanbrain.gitbook.io/notes/notes/natural-language-processing/self_attn_sentence_embed.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.