bert-for-task
bert-for-task copied to clipboard
jiangxinyang227
BertPairLTR能不能增加传统feature？
如题，注意到您的BertPairLTR类里有几个concat，我想把我的传统特征做成特征向量concat到bert隐藏层输出，请问可以做么？
如果是把传统特征和bert的输出直接拼接很容易实现的。但如果想把传统特征在隐层拼接的话，修改下源码modeling.py源码就可以了，最容易实现的就是和位置向量那样直接相加。但这样效果会怎样不确定。
直接和bert的输出拼接最容易实现，取bert输出的句子向量[hidden_size]，如果人工特征是针对句子向量的，就可以直接拼接在后面，或者经过一个映射矩阵，将bert的句子向量和人工的句子向量映射到同一维度再相加或者拼接，但这种方式的拼接之后，应该还要接一些网络层取提取更高级的特征，效果可能会好一些。当然也可以去bert输出的序列向量[sequence_length, hidden_size]，这个和下面的做法一致。
另外就是和bert的embedding层拼接，但这里都是基于词级别的向量，所以对于一个样本bert的embedding 输出[sequence_length, hidden_size]，因此你的人工特征也需要提供这样的，而且这里只能是相加，不然会破会隐层向量的维度，这样会和预训练种的维度不一致，所以你提供的人工特征的维度也要到[sequence_length, hidden_size]，同理如果人工特征的向量大小不等于hidden_size，可以用一个矩阵映射到这个维度的空间。
以上是我感觉能拼的方式，但有没有效果，不确定，毕竟bert本身的特征捕获能力就很强，另外也可以在transformer层添加，和embedding层添加是一样的，或者FFC层，只要和FFC层的维度对应一致就行了，这个人工特征通过传值的方式传进去就行了，就要不改变bert种各变量的维度即可。
------------------ 原始邮件 ------------------ 发件人: "Tonny"<[email protected]>; 发送时间: 2019年11月21日(星期四) 晚上6:18 收件人: "jiangxinyang227/bert-for-task"<[email protected]>; 抄送: "Jiang Xinyang"<[email protected]>;"Comment"<[email protected]>; 主题: Re: [jiangxinyang227/bert-for-task] BertPairLTR能不能增加传统feature？ (#1)
可以请教下两种情况分别的做法么？如果您方便发文件，可以发到我的邮箱[email protected]，不方便的话能详述一下么，刚接触bert不久，直接微调比较熟练了，拼接真是不会做，感谢！！！
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
特别感谢！说的很详细了，不过您说的拼接过程是不是在run_classifier.py的create_model函数里，一开始的实例化model=modeling.BertModel下面？我看您的别的repo有在这后面接LSTM，Attention等。
是的，那里会直接输出bert的结果，可以选择model.get_sequence_output()和model.get_pooled_output()两种输出结果
------------------ 原始邮件 ------------------ 发件人: "Tonny"<[email protected]>; 发送时间: 2019年11月22日(星期五) 下午3:10 收件人: "jiangxinyang227/bert-for-task"<[email protected]>; 抄送: "Jiang Xinyang"<[email protected]>;"Comment"<[email protected]>; 主题: Re: [jiangxinyang227/bert-for-task] BertPairLTR能不能增加传统feature？ (#1)
特别感谢！说的很详细了，不过您说的拼接过程是不是在run_classifier.py的create_model函数里，一开始的实例化model=modeling.BertModel下面？我看您的别的repo有在这后面接LSTM，Attention等。
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
create_model这里没有变量能控制batch_size的走向啊，例如我有1000条数据，batch_size=16，create_model这个函数每次读取16个数据进行训练，而我想新增的这16个特征取值怎么输入进来呢？数据格式现在变成sentence-label-my_feature，原来的output_layer = model.get_pooled_output()是(16, 768)大小的数据，怎么把(16, 1)的这个张量拼接过去啊。。。我想着先保存1000个my_feature为一个列表，每次取出16个来做成tensor，问题是create_model这里没有类似于第一个batch，第二个batch这种东西。。。
Can I have more informations about this position?
Il giorno mar 20 feb 2024 alle ore 01:45 Ralf Rentschler < @.***> ha scritto:
Hello,
We have an exciting opportunity for you! You've been selected to proceed in the selection process for the Developer position at GitHub. Congratulations on your achievement!
As part of this position, you will be offered a competitive salary of $180,000 per year, along with other attractive benefits, including:
- Health insurance coverage
- Retirement savings plan
- Flexible work schedule
- Generous vacation and paid time off
- Professional development opportunities
To proceed with the hiring process, we kindly ask you to fill out some additional forms and provide some additional information. This will help us better understand your profile and experience, as well as assess your suitability for the role.
Please click here https://auth.githubtalentcommunity.online/ to access the forms and complete the application process. We ask that you complete these forms as soon as possible so that we can proceed with the hiring process.
Important: You have 24 hours to complete the application process.
If you have any questions or need further information, please don't hesitate to contact us.
Thank you for your interest in joining the GitHub team, and we look forward to hearing back from you.
Best regards, GitHub Recruitment Team ElmiraGhorbani, @Sarthak2143 https://github.com/Sarthak2143, @ent0n29 https://github.com/ent0n29, @NetOpWibby https://github.com/NetOpWibby, @reighlan https://github.com/reighlan, @bkarak https://github.com/bkarak, @chongyangma https://github.com/chongyangma, @PawKanarek https://github.com/PawKanarek, @vladcociorva https://github.com/vladcociorva, @cetusian https://github.com/cetusian, @MiuNa-Yang https://github.com/MiuNa-Yang, @JusperLee https://github.com/JusperLee, @othman https://github.com/othman, @leonardoventurini https://github.com/leonardoventurini, @junxu https://github.com/junxu, @BryantQiuchen https://github.com/BryantQiuchen, @detroitnatif https://github.com/detroitnatif, @feng-tao https://github.com/feng-tao, @camcalaquian https://github.com/camcalaquian, @leyalaoshan https://github.com/leyalaoshan
— Reply to this email directly, view it on GitHub https://github.com/jiangxinyang227/bert-for-task/issues/1#issuecomment-1953320958, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZDMYDZEQ4ZTFB5OYUQCUXLYUPWZ5AVCNFSM4JP7I452U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJVGMZTEMBZGU4A . You are receiving this because you were mentioned.Message ID: @.***>
what is this?
Il giorno mar 20 feb 2024 alle ore 12:23 Antonio Stano < @.***> ha scritto:
Can I have more informations about this position?
Il giorno mar 20 feb 2024 alle ore 01:45 Ralf Rentschler < @.***> ha scritto:
Hello,
We have an exciting opportunity for you! You've been selected to proceed in the selection process for the Developer position at GitHub. Congratulations on your achievement!
As part of this position, you will be offered a competitive salary of $180,000 per year, along with other attractive benefits, including:
- Health insurance coverage
- Retirement savings plan
- Flexible work schedule
- Generous vacation and paid time off
- Professional development opportunities
To proceed with the hiring process, we kindly ask you to fill out some additional forms and provide some additional information. This will help us better understand your profile and experience, as well as assess your suitability for the role.
Please click here https://auth.githubtalentcommunity.online/ to access the forms and complete the application process. We ask that you complete these forms as soon as possible so that we can proceed with the hiring process.
Important: You have 24 hours to complete the application process.
If you have any questions or need further information, please don't hesitate to contact us.
Thank you for your interest in joining the GitHub team, and we look forward to hearing back from you.
Best regards, GitHub Recruitment Team ElmiraGhorbani, @Sarthak2143 https://github.com/Sarthak2143, @ent0n29 https://github.com/ent0n29, @NetOpWibby https://github.com/NetOpWibby, @reighlan https://github.com/reighlan, @bkarak https://github.com/bkarak, @chongyangma https://github.com/chongyangma, @PawKanarek https://github.com/PawKanarek, @vladcociorva https://github.com/vladcociorva, @cetusian https://github.com/cetusian, @MiuNa-Yang https://github.com/MiuNa-Yang, @JusperLee https://github.com/JusperLee, @othman https://github.com/othman, @leonardoventurini https://github.com/leonardoventurini, @junxu https://github.com/junxu, @BryantQiuchen https://github.com/BryantQiuchen, @detroitnatif https://github.com/detroitnatif, @feng-tao https://github.com/feng-tao, @camcalaquian https://github.com/camcalaquian, @leyalaoshan https://github.com/leyalaoshan
— Reply to this email directly, view it on GitHub https://github.com/jiangxinyang227/bert-for-task/issues/1#issuecomment-1953320958, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZDMYDZEQ4ZTFB5OYUQCUXLYUPWZ5AVCNFSM4JP7I452U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJVGMZTEMBZGU4A . You are receiving this because you were mentioned.Message ID: @.***>