[Question]: How the StackedEmbeddings function actually works?

[ijazul-haq]

Question

How the StackedEmbeddings function actually works? Is it concatinating the two word embeddings, i.e. using torch.cat([emb1, emb2]).

I want to concatinate BytePairEmbeddings with TransformerWordEmbeddings, so i'm doing like this:

bert_emb = TransformerWordEmbeddings( model='xlm-roberta-base', layers="-1", subtoken_pooling="mean", fine_tune=True, use_context=True, )

bpe_emb = BytePairEmbeddings('en') stacked_embeddings = StackedEmbeddings([bert_emb , bpe_emb])

So the resultant word embeddings (stacked_embeddings) will be a concatenation of the two embeddings, or is it element-wise mean embedding, or anything else?

Thank you

[helpmefindaname]

Hi @ijazul-haq

StackedEmbeddings use the stacking/concatination operator as their name implies. Note that element-wise pooling would not be possible, as the embeddings usually don't have the same embedding length.

[ijazul-haq]

Does it mean that StackedEmbeddings can not be used for sequence tagging, it's only for sequence classification? right?

[helpmefindaname]

Sorry, I cannot follow how you come to that conclusion, but I can asure you that the StackedEmbeddings work for both, TokenEmbeddings and DocumentEmbeddings.