rl icon indicating copy to clipboard operation
rl copied to clipboard
verified publisher icon pytorch

[WIP] Release docs

Open vmoens opened this issue 2 years ago • 1 comments

vmoens avatar Sep 27 '23 13:09 vmoens

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}23$. Worsened: $\large\color{#d91a1a}9$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1173s 0.1140s 8.7690 Ops/s 8.7268 Ops/s $\color{#35bf28}+0.48\%$
test_sync 66.4057ms 62.8807ms 15.9031 Ops/s 16.2361 Ops/s $\color{#d91a1a}-2.05\%$
test_async 0.1139s 61.6634ms 16.2171 Ops/s 16.6764 Ops/s $\color{#d91a1a}-2.75\%$
test_simple 1.0402s 0.9688s 1.0322 Ops/s 1.0286 Ops/s $\color{#35bf28}+0.34\%$
test_transformed 1.3299s 1.2494s 0.8004 Ops/s 0.8081 Ops/s $\color{#d91a1a}-0.96\%$
test_serial 2.9457s 2.7921s 0.3581 Ops/s 0.3647 Ops/s $\color{#d91a1a}-1.80\%$
test_parallel 2.4841s 2.3823s 0.4198 Ops/s 0.4152 Ops/s $\color{#35bf28}+1.10\%$
test_step_mdp_speed[True-True-True-True-True] 2.0752ms 57.6863μs 17.3351 KOps/s 16.4380 KOps/s $\textbf{\color{#35bf28}+5.46\%}$
test_step_mdp_speed[True-True-True-True-False] 3.3471ms 32.7685μs 30.5171 KOps/s 28.8444 KOps/s $\textbf{\color{#35bf28}+5.80\%}$
test_step_mdp_speed[True-True-True-False-True] 4.2849ms 41.1190μs 24.3196 KOps/s 23.3704 KOps/s $\color{#35bf28}+4.06\%$
test_step_mdp_speed[True-True-True-False-False] 1.3256ms 23.5561μs 42.4519 KOps/s 40.4083 KOps/s $\textbf{\color{#35bf28}+5.06\%}$
test_step_mdp_speed[True-True-False-True-True] 0.4989ms 60.2065μs 16.6095 KOps/s 16.4204 KOps/s $\color{#35bf28}+1.15\%$
test_step_mdp_speed[True-True-False-True-False] 5.2570ms 35.6457μs 28.0539 KOps/s 26.9749 KOps/s $\color{#35bf28}+4.00\%$
test_step_mdp_speed[True-True-False-False-True] 0.6420ms 44.2218μs 22.6133 KOps/s 21.7076 KOps/s $\color{#35bf28}+4.17\%$
test_step_mdp_speed[True-True-False-False-False] 0.4218ms 25.3951μs 39.3776 KOps/s 37.8927 KOps/s $\color{#35bf28}+3.92\%$
test_step_mdp_speed[True-False-True-True-True] 2.1706ms 63.4186μs 15.7682 KOps/s 15.9550 KOps/s $\color{#d91a1a}-1.17\%$
test_step_mdp_speed[True-False-True-True-False] 0.9722ms 37.7263μs 26.5067 KOps/s 25.8084 KOps/s $\color{#35bf28}+2.71\%$
test_step_mdp_speed[True-False-True-False-True] 7.6892ms 43.4535μs 23.0131 KOps/s 21.6271 KOps/s $\textbf{\color{#35bf28}+6.41\%}$
test_step_mdp_speed[True-False-True-False-False] 5.8677ms 25.9781μs 38.4940 KOps/s 37.1117 KOps/s $\color{#35bf28}+3.72\%$
test_step_mdp_speed[True-False-False-True-True] 3.1985ms 65.1458μs 15.3502 KOps/s 14.4745 KOps/s $\textbf{\color{#35bf28}+6.05\%}$
test_step_mdp_speed[True-False-False-True-False] 0.5694ms 40.3812μs 24.7640 KOps/s 24.3083 KOps/s $\color{#35bf28}+1.87\%$
test_step_mdp_speed[True-False-False-False-True] 1.2195ms 46.0758μs 21.7034 KOps/s 20.6431 KOps/s $\textbf{\color{#35bf28}+5.14\%}$
test_step_mdp_speed[True-False-False-False-False] 0.4847ms 28.4136μs 35.1944 KOps/s 31.2178 KOps/s $\textbf{\color{#35bf28}+12.74\%}$
test_step_mdp_speed[False-True-True-True-True] 2.1456ms 62.3523μs 16.0379 KOps/s 15.5629 KOps/s $\color{#35bf28}+3.05\%$
test_step_mdp_speed[False-True-True-True-False] 0.6193ms 38.1379μs 26.2206 KOps/s 25.7197 KOps/s $\color{#35bf28}+1.95\%$
test_step_mdp_speed[False-True-True-False-True] 2.4903ms 48.8088μs 20.4881 KOps/s 20.7741 KOps/s $\color{#d91a1a}-1.38\%$
test_step_mdp_speed[False-True-True-False-False] 1.2308ms 29.9845μs 33.3505 KOps/s 33.5967 KOps/s $\color{#d91a1a}-0.73\%$
test_step_mdp_speed[False-True-False-True-True] 3.8087ms 65.5073μs 15.2655 KOps/s 15.1302 KOps/s $\color{#35bf28}+0.89\%$
test_step_mdp_speed[False-True-False-True-False] 1.6124ms 40.3039μs 24.8115 KOps/s 23.7946 KOps/s $\color{#35bf28}+4.27\%$
test_step_mdp_speed[False-True-False-False-True] 1.7774ms 49.7318μs 20.1078 KOps/s 19.4991 KOps/s $\color{#35bf28}+3.12\%$
test_step_mdp_speed[False-True-False-False-False] 1.8721ms 29.6350μs 33.7439 KOps/s 30.4479 KOps/s $\textbf{\color{#35bf28}+10.83\%}$
test_step_mdp_speed[False-False-True-True-True] 4.1982ms 67.5382μs 14.8064 KOps/s 14.1602 KOps/s $\color{#35bf28}+4.56\%$
test_step_mdp_speed[False-False-True-True-False] 2.3061ms 41.7031μs 23.9790 KOps/s 22.6037 KOps/s $\textbf{\color{#35bf28}+6.08\%}$
test_step_mdp_speed[False-False-True-False-True] 1.4076ms 49.1383μs 20.3507 KOps/s 19.5721 KOps/s $\color{#35bf28}+3.98\%$
test_step_mdp_speed[False-False-True-False-False] 1.3795ms 30.4901μs 32.7975 KOps/s 32.7659 KOps/s $\color{#35bf28}+0.10\%$
test_step_mdp_speed[False-False-False-True-True] 8.7952ms 69.2056μs 14.4497 KOps/s 14.7167 KOps/s $\color{#d91a1a}-1.81\%$
test_step_mdp_speed[False-False-False-True-False] 4.4625ms 44.4151μs 22.5148 KOps/s 22.3656 KOps/s $\color{#35bf28}+0.67\%$
test_step_mdp_speed[False-False-False-False-True] 2.3978ms 52.9813μs 18.8746 KOps/s 19.3603 KOps/s $\color{#d91a1a}-2.51\%$
test_step_mdp_speed[False-False-False-False-False] 1.3759ms 32.2683μs 30.9902 KOps/s 30.3967 KOps/s $\color{#35bf28}+1.95\%$
test_values[generalized_advantage_estimate-True-True] 24.7740ms 21.1391ms 47.3057 Ops/s 42.0904 Ops/s $\textbf{\color{#35bf28}+12.39\%}$
test_values[vec_generalized_advantage_estimate-True-True] 96.2024ms 78.2987ms 12.7716 Ops/s 12.5531 Ops/s $\color{#35bf28}+1.74\%$
test_values[td0_return_estimate-False-False] 2.4299ms 0.7207ms 1.3876 KOps/s 1.0772 KOps/s $\textbf{\color{#35bf28}+28.81\%}$
test_values[td1_return_estimate-False-False] 27.3584ms 20.2760ms 49.3194 Ops/s 44.0450 Ops/s $\textbf{\color{#35bf28}+11.97\%}$
test_values[vec_td1_return_estimate-False-False] 92.1794ms 78.2376ms 12.7816 Ops/s 12.7238 Ops/s $\color{#35bf28}+0.45\%$
test_values[td_lambda_return_estimate-True-False] 64.8764ms 53.2176ms 18.7908 Ops/s 17.2564 Ops/s $\textbf{\color{#35bf28}+8.89\%}$
test_values[vec_td_lambda_return_estimate-True-False] 0.1105s 80.3181ms 12.4505 Ops/s 12.6986 Ops/s $\color{#d91a1a}-1.95\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 19.5406ms 16.6327ms 60.1224 Ops/s 55.1521 Ops/s $\textbf{\color{#35bf28}+9.01\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 13.2724ms 5.7095ms 175.1480 Ops/s 168.2566 Ops/s $\color{#35bf28}+4.10\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 6.6072ms 0.9655ms 1.0357 KOps/s 1.0033 KOps/s $\color{#35bf28}+3.23\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 83.0217ms 74.3828ms 13.4440 Ops/s 12.7881 Ops/s $\textbf{\color{#35bf28}+5.13\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 12.5800ms 6.9432ms 144.0248 Ops/s 144.5142 Ops/s $\color{#d91a1a}-0.34\%$
test_dqn_speed 9.1625ms 4.0433ms 247.3219 Ops/s 239.5029 Ops/s $\color{#35bf28}+3.26\%$
test_ddpg_speed 13.2563ms 7.8849ms 126.8250 Ops/s 121.9509 Ops/s $\color{#35bf28}+4.00\%$
test_sac_speed 29.5017ms 19.9447ms 50.1387 Ops/s 45.6884 Ops/s $\textbf{\color{#35bf28}+9.74\%}$
test_redq_speed 43.0009ms 33.6535ms 29.7146 Ops/s 29.5940 Ops/s $\color{#35bf28}+0.41\%$
test_redq_deprec_speed 39.2301ms 32.6531ms 30.6250 Ops/s 31.1110 Ops/s $\color{#d91a1a}-1.56\%$
test_td3_speed 26.3119ms 21.4262ms 46.6718 Ops/s 48.1589 Ops/s $\color{#d91a1a}-3.09\%$
test_cql_speed 91.0656ms 75.5092ms 13.2434 Ops/s 14.3451 Ops/s $\textbf{\color{#d91a1a}-7.68\%}$
test_a2c_speed 23.5288ms 15.4495ms 64.7272 Ops/s 70.2399 Ops/s $\textbf{\color{#d91a1a}-7.85\%}$
test_ppo_speed 27.0479ms 15.0574ms 66.4124 Ops/s 64.9002 Ops/s $\color{#35bf28}+2.33\%$
test_reinforce_speed 40.9175ms 14.3883ms 69.5010 Ops/s 80.7982 Ops/s $\textbf{\color{#d91a1a}-13.98\%}$
test_iql_speed 76.5419ms 65.9827ms 15.1555 Ops/s 15.9574 Ops/s $\textbf{\color{#d91a1a}-5.03\%}$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.6261ms 3.5071ms 285.1339 Ops/s 256.5746 Ops/s $\textbf{\color{#35bf28}+11.13\%}$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.1923s 4.3970ms 227.4266 Ops/s 271.5050 Ops/s $\textbf{\color{#d91a1a}-16.23\%}$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.1839s 4.4465ms 224.8955 Ops/s 266.3704 Ops/s $\textbf{\color{#d91a1a}-15.57\%}$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.2240ms 3.6880ms 271.1522 Ops/s 295.0601 Ops/s $\textbf{\color{#d91a1a}-8.10\%}$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 7.9608ms 3.6866ms 271.2542 Ops/s 266.7077 Ops/s $\color{#35bf28}+1.70\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 6.3300ms 3.7411ms 267.3002 Ops/s 246.6780 Ops/s $\textbf{\color{#35bf28}+8.36\%}$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.0161ms 3.5289ms 283.3766 Ops/s 292.6768 Ops/s $\color{#d91a1a}-3.18\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 7.9139ms 3.7139ms 269.2613 Ops/s 215.8140 Ops/s $\textbf{\color{#35bf28}+24.77\%}$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 11.6127ms 3.6801ms 271.7347 Ops/s 267.6322 Ops/s $\color{#35bf28}+1.53\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.1549ms 3.3196ms 301.2420 Ops/s 200.8942 Ops/s $\textbf{\color{#35bf28}+49.95\%}$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 6.7328ms 3.6706ms 272.4345 Ops/s 265.7415 Ops/s $\color{#35bf28}+2.52\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 10.2225ms 3.8182ms 261.9065 Ops/s 253.9379 Ops/s $\color{#35bf28}+3.14\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.0657ms 3.3544ms 298.1162 Ops/s 272.7815 Ops/s $\textbf{\color{#35bf28}+9.29\%}$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 8.8258ms 3.6106ms 276.9658 Ops/s 270.9181 Ops/s $\color{#35bf28}+2.23\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 6.4242ms 3.5838ms 279.0326 Ops/s 279.0673 Ops/s $\color{#d91a1a}-0.01\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.5541ms 3.3340ms 299.9411 Ops/s 287.4035 Ops/s $\color{#35bf28}+4.36\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 8.6897ms 3.6832ms 271.5002 Ops/s 270.0424 Ops/s $\color{#35bf28}+0.54\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 5.7479ms 3.5908ms 278.4866 Ops/s 274.4979 Ops/s $\color{#35bf28}+1.45\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.3125s 43.1943ms 23.1512 Ops/s 22.6765 Ops/s $\color{#35bf28}+2.09\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 0.1997s 38.6842ms 25.8503 Ops/s 26.9827 Ops/s $\color{#d91a1a}-4.20\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 0.1804s 39.7404ms 25.1633 Ops/s 24.3671 Ops/s $\color{#35bf28}+3.27\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1744s 36.7087ms 27.2415 Ops/s 26.0788 Ops/s $\color{#35bf28}+4.46\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 0.1993s 40.3607ms 24.7766 Ops/s 23.9370 Ops/s $\color{#35bf28}+3.51\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 0.1811s 36.2987ms 27.5492 Ops/s 24.0276 Ops/s $\textbf{\color{#35bf28}+14.66\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1969s 42.3400ms 23.6184 Ops/s 25.8022 Ops/s $\textbf{\color{#d91a1a}-8.46\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 0.1780s 36.5661ms 27.3478 Ops/s 24.8402 Ops/s $\textbf{\color{#35bf28}+10.09\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 0.1865s 40.4314ms 24.7333 Ops/s 27.0196 Ops/s $\textbf{\color{#d91a1a}-8.46\%}$

github-actions[bot] avatar Sep 27 '23 14:09 github-actions[bot]