oneDNN
oneDNN copied to clipboard
oneapi-src
Some unimplemented unary expressions?
Hi Here are some unary ops currently not implemented in 1dnn (or I missed them):
- "fast sigmoid": out = (in / (1 + abs(in)))
https://stackoverflow.com/questions/10732027/fast-sigmoid-algorithm
https://en.wikipedia.org/wiki/File:Gjl-t(x).svg
- cos and sin : must have for transformers positional encoding
- reciprocal squareroot: 1 / sqrt(x)
Best
@WilliamTambellini,
Isn't reciprocal square root supported by eltwise pow algorithm with alpha =1 and beta = -1/2?
Will adding sin/cos algorithms to eltwise primitive be enough to cover embeddings case?
- recip sqrt: I ll check, but not a high priority, mainly used by layer norm and hopefully 1dnn already have it implemented internally
- sin/cos is not directly to compute embeddings but only position encoding: https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/ https://arxiv.org/pdf/2104.09864.pdf Adding support for cos/sin element wise should be enough to cover standard position encoding.
Thanks
+1 for adding cos/sin elementwise! I would also greatly appreciate it!
@WilliamTambellini @lucasalavapena Thank you for the request, regarding sin/cos: ML frameworks have their own implementation and C++ applications have access to the standard math library with cos/sin functions, so what is the benefit of having an additional implementation under oneDNN API?
@WilliamTambellini I see that positional encoding does not depend on the actual data, is it pre-computed prior to inference/training and remains constant during the process?
@igorsafo I am still relatively new to oneDNN, so hopefully what I am saying makes sense :smile: .
I was more thinking of having it as an op for dnnl::graph, so you can get some benefits you normally can get with the graph extension (memory layout and use of graph compiler?).
Aren't some of the existing element wise functions part of the standard math library too?