*Memos:
- My post explains Embedding Layer.
- My post explains manual_seed().
- My post explains requires_grad.
Embedding() can get the 1D or more D tensor of the zero or more elements computed by Embedding from the 0D or more D tensor of one or more elements(indices) with or without the 2D tensor of zero or more elements(weights) as shown below:
*Memos:
- The 1st argument for initialization is
num_embeddings
(Required-Type:int
). *It must be1 <= x
. - The 2nd argument for initialization is
embedding_dim
(Required-Type:int
). *It must be0 <= x
. - The 3rd argument for initialization is
padding_idx
(Optional-Default:None
-Type:int
). - The 4th argument for initialization is
max_norm
(Optional-Default:None
-Type:float
). - The 5th argument for initialization is
norm_type
(Optional-Default:2.0
-Type:float
). *It must be0 <= x
when returning an empty tensor. - The 6th argument for initialization is
scale_grad_by_freq
(Optional-Default:False
-Type:bool
). - The 7th argument for initialization is
sparse
(Optional-Default:False
-Type:bool
). - The 8th argument for initialization is
_weight
(Optional-Default:None
-Type:tensor
offloat
): *Memos:- If
None
,weight
is randomly generated. - It must be the 2D tensor of zero or more elements.
- Its size must be same as
num_embeddings
andembedding_dim
.
- If
- The 9th argument for initialization is
_freeze
(Optional-Default:False
-Type:bool
). *If it'sFalse
,requires_grad
isTrue
while if it'sTrue
,requires_grad
isFalse
. - The 10th argument for initialization is
device
(Optional-Defalut:None
-Type:str
,int
or device()): *Memos:- If it's
None
, get_default_device() is used. *My post explainsget_default_device()
and set_default_device(). -
device=
can be omitted. -
My post explains
device
argument.
- If it's
- The 11th argument for initialization is
dtype
(Optional-Default:None
-Type:dtype): *Memos:- If it's
None
, get_default_dtype() is used. *My post explainsget_default_dtype()
and set_default_dtype(). -
dtype=
can be omitted. -
My post explains
dtype
argument.
- If it's
- The 1st argument is
input
(Required-Type:tensor
ofint
): *Memos:- It's indices.
- Indices must be less than
num_embeddings
. - It must be the 0D or more D tensor of one or more elements.
- Its
device
must be same asEmbedding()
's.
-
embedding.device
andembedding.dtype
don't work.
import torch
from torch import nn
tensor1 = torch.tensor([6, 0, 2, 5]) # Indices
tensor1.requires_grad
# False
torch.manual_seed(42)
embedding = nn.Embedding(num_embeddings=7, embedding_dim=3)
tensor2 = embedding(input=tensor1)
tensor2
# tensor([[0.8034, -0.6216, -0.5920],
# [1.9269, 1.4873, 0.9007],
# [0.8008, 1.6806, 0.3559],
# [0.8599, -0.3097, -0.3957]], grad_fn=<EmbeddingBackward0>)
tensor2.requires_grad
# True
embedding
# Embedding(7, 3)
embedding.num_embeddings
# 7
embedding.embedding_dim
# 3
embedding.padding_idx
# None
embedding.max_norm
# None
embedding.norm_type
# 2.0
embedding.scale_grad_by_freq
# False
embedding.sparse
# False
embedding.weight
# Parameter containing:
# tensor([[1.9269, 1.4873, 0.9007],
# [-2.1055, 0.6784, 1.0783],
# [0.8008, 1.6806, 0.3559],
# [-0.6866, -0.4934, 0.2415],
# [-1.1109, 0.0418, -0.2516],
# [0.8599, -0.3097, -0.3957],
# [0.8034, -0.6216, -0.5920]], requires_grad=True)
torch.manual_seed(42)
embedding = nn.Embedding(num_embeddings=7, embedding_dim=3,
padding_idx=None, max_norm=None, norm_type=2.0,
scale_grad_by_freq=False, sparse=False,
_weight=None, _freeze=False,
device=None, dtype=None)
embedding(input=tensor1)
# tensor([[0.8034, -0.6216, -0.5920],
# [1.9269, 1.4873, 0.9007],
# [0.8008, 1.6806, 0.3559],
# [0.8599, -0.3097, -0.3957]], grad_fn=<EmbeddingBackward0>)
weight = torch.tensor([[4., 9., 1.],
[-2., 0., 3.],
[0., 5., 7.],
[8., -6., 0.],
[1., 3., -9.],
[-3., 1., 2.],
[-5., 7., -4.]])
embedding = nn.Embedding(num_embeddings=7, embedding_dim=3,
_weight=weight)
embedding(input=tensor1)
# tensor([[-5., 7., -4.],
# [4., 9., 1.],
# [0., 5., 7.],
# [-3., 1., 2.]], grad_fn=<EmbeddingBackward0>)
my_tensor = torch.tensor([[6, 0], # Indices
[2, 5]])
torch.manual_seed(42)
embedding = nn.Embedding(num_embeddings=7, embedding_dim=3)
embedding(input=my_tensor)
# tensor([[[0.8034, -0.6216, -0.5920],
# [1.9269, 1.4873, 0.9007]],
# [[0.8008, 1.6806, 0.3559],
# [0.8599, -0.3097, -0.3957]]], grad_fn=<EmbeddingBackward0>)
my_tensor = torch.tensor([[[6], [0]], # Indices
[[2], [5]]])
torch.manual_seed(42)
embedding = nn.Embedding()
embedding(input=my_tensor)
# tensor([[[[0.8034, -0.6216, -0.5920]],
# [[1.9269, 1.4873, 0.9007]]],
# [[[0.8008, 1.6806, 0.3559]],
# [[0.8599, -0.3097, -0.3957]]]], grad_fn=<EmbeddingBackward0>)
Embedding.from_pretrained()
can get the 1D or more D tensor of the zero or more elements computed by Embedding from the 0D or more D tensor of one or more elements(indices) with the 2D tensor of zero or more elements(weights) as shown below:
*Memos:
- The 1st argument for initialization is
embeddings
(Required-Type:tensor
ofint
,float
orcomplex
): *Memos:- It's weight.
- It must be the 2D tensor of zero or more elements.
- The 2nd argument for initialization is
freeze
(Optional-Default:True
-Type:bool
). *If it'sFalse
,requires_grad
isTrue
while if it'sTrue
,requires_grad
isFalse
. - The 3rd argument for initialization is
padding_idx
(Optional-Default:None
-Type:int
). - The 4th argument for initialization is
max_norm
(Optional-Default:None
-Type:float
). *It's must beNone
, ifembeddings
is an empty tensor andnorm_type
is negative. - The 5th argument for initialization is
norm_type
(Optional-Default:2.0
-Type:float
). - The 6th argument for initialization is
scale_grad_by_freq
(Optional-Default:False
-Type:bool
). - The 7th argument for initialization is
sparse
(Optional-Default:False
-Type:bool
). - The 1st argument is
input
(Required-Type:tensor
ofint
): *Memos:- It's indices.
- Indices must be less than
embeddings
of the number of the 1st dimension. - It must be the 0D or more D tensor of one or more elements.
import torch
from torch import nn
weight = torch.tensor([[4., 9., 1.],
[-2., 0., 3.],
[0., 5., 7.],
[8., -6., 0.],
[1., 3., -9.],
[-3., 1., 2.],
[-5., 7., -4.]])
tensor1 = torch.tensor([6, 0, 2, 5]) # Indices
tensor1.requires_grad
# False
embedding = nn.Embedding.from_pretrained(embeddings=weight)
tensor2 = embedding(input=tensor1)
tensor2
# tensor([[-5., 7., -4.],
# [4., 9., 1.],
# [0., 5., 7.],
# [-3., 1., 2.]])
tensor2.requires_grad
# False
embedding
# Embedding(7, 3)
embedding.num_embeddings
# 7
embedding.embedding_dim
# 3
embedding.padding_idx
# None
embedding.max_norm
# None
embedding.norm_type
# 2.0
embedding.scale_grad_by_freq
# False
embedding.sparse
# False
embedding.weight
# Parameter containing:
# tensor([[4., 9., 1.],
# [-2., 0., 3.],
# [0., 5., 7.],
# [8., -6., 0.],
# [1., 3., -9.],
# [-3., 1., 2.],
# [-5., 7., -4.]])
embedding = nn.Embedding.from_pretrained(embeddings=weight, freeze=True,
padding_idx=None, max_norm=None, norm_type=2.0,
scale_grad_by_freq=False, sparse=False)
embedding(input=tensor1)
# tensor([[-5., 7., -4.],
# [4., 9., 1.],
# [0., 5., 7.],
# [-3., 1., 2.]])
my_tensor = torch.tensor([[6, 0], # Indices
[2, 5]])
embedding = nn.Embedding.from_pretrained(embeddings=weight)
embedding(input=my_tensor)
# tensor([[[-5., 7., -4.],
# [4., 9., 1.]],
# [[0., 5., 7.],
# [-3., 1., 2.]]])
my_tensor = torch.tensor([[[6], [0]], # Indices
[[2], [5]]])
embedding = nn.Embedding.from_pretrained(embeddings=weight)
embedding(input=my_tensor)
# tensor([[[[-5., 7., -4.]],
# [[4., 9., 1.]]],
# [[[0., 5., 7.]],
# [[-3., 1., 2.]]]])
weight = torch.tensor([[4, 9, 1],
[-2, 0, 3],
[0, 5, 7],
[8, -6, 0],
[1, 3, -9],
[-3, 1, 2],
[-5, 7, -4]])
embedding = nn.Embedding.from_pretrained(embeddings=weight)
embedding(input=my_tensor)
# tensor([[[[-5, 7, -4]],
# [[4, 9, 1]]],
# [[[0, 5, 7]],
# [[-3, 1, 2]]]])
weight = torch.tensor([[4.+0.j, 9.+0.j, 1.+0.j],
[-2.+0.j, 0.+0.j, 3.+0.j],
[0.+0.j, 5.+0.j, 7.+0.j],
[8.+0.j, -6.+0.j, 0.+0.j],
[1.+0.j, 3.+0.j, -9.+0.j],
[-3.+0.j, 1.+0.j, 2.+0.j],
[-5.+0.j, 7.+0.j, -4.+0.j]])
embedding = nn.Embedding.from_pretrained(embeddings=weight)
embedding(input=my_tensor)
# tensor([[[[-5.+0.j, 7.+0.j, -4.+0.j]],
# [[4.+0.j, 9.+0.j, 1.+0.j]]],
# [[[0.+0.j, 5.+0.j, 7.+0.j]],
# [[-3.+0.j, 1.+0.j, 2.+0.j]]]])
weight = torch.tensor([[True, False, True],
[False, True, False],
[True, False, True],
[False, True, False],
[True, False, True],
[False, True, False],
[True, False, True]])
embedding = nn.Embedding.from_pretrained(embeddings=weight)
embedding(input=my_tensor)
# tensor([[[[True, False, True]],
# [[True, False, True]]],
# [[[True, False, True]],
# [[False, True, False]]]])
Top comments (0)