Do we need to use smoothing and transform in Theano if we use an index matrix?

Question

I am trying to understand the implementation of Theano LSTM (at the moment, the link does not work for any reason, but I hope that it will return soon).

In the code, I see the following part:

emb = tparams['Wemb'][x.flatten()].reshape([n_timesteps, n_samples, options['dim_proj']]) 

To make it "context independent", I rewrite it as follows:

 e = W[x.flatten()]].reshape([n1, n2, n3]) 

where the dimension x is (n1, n2) and the dimension W is (N, n3) .

So my assumption is that the code can be rewritten to be shorter. In particular, we can simply write:

 e = W[x] 

Or, if we use the original notation, it should be:

 emb = tparams['Wemb'][x] 

I'm right?

To provide a little more context, x is a 2D array containing integers representing words (for example, 27 means "word number 27"). W in my notation (or tparams['Wemb'] ) in the original notation is a 2D matrix in which each row corresponds to a word. Thus, it is a word embedding matrix (Word2Vec) that maps each word into a real vector.

Answer 1

Yes you are right.

W[x.flatten()]] gives W strings (that is, words) defined by x values. The result is shape = (n1*n2,n3) . Let me call this "list of words" (not a list of pythons, but just a general list of speeches). Reformatting then gives the desired size, where the word list is suspended on n1 pages of n2 words.

You get the same with W[x] since each of the lines n2 x gives you one of the n1 pages of the result.

Here's an example program that shows that both expressions are equivalent:

 import numpy as np N = 4 n3 = 5 W = np.arange(n3*N).reshape((N,n3)) print("W = \n", W) n1 = 2 n2 = 3 x = np.random.randint(low=0, high=N,size=(n1,n2)) print("\nx = \n", x) print("\ne = \n", W[x.flatten()].reshape([n1, n2, n3])) print("\nalternativeE = \n", W[x])