a mathematical problem ..

[tyfeng1997]

I derived Equation 12, but the result is not the same as Equation 13 in your paper. In my derivation, I didn't get the first item in Equation 13, I don't know where it is wrong. can you help me..？

[haarnoja]

Thanks for the question. You mean Equation 13 in this paper? It is the total derivative of J(\phi) with respect to the policy parameters \phi. Note that both \pi_\phi and a_t = f_\phi depend on these parameters, so we'll need to differentiate with respect to both, and use the chain rule for the latter. It's is exactly analogous to this example.