Lodestar key import takes too long

[barnabasbusa]

Currently loading in lodestar validator keys one by one takes too long ~ 0.3 sec/key on an M1 macbook pro.

This means that on an average kurtosis run where the default validator will have 64 validator keys, each validator process will add an additional 20 seconds to start up.

Once the key loaded in once, lodestar stores it in a cache at <dataDir>/cache (default to ~/.local/share/lodestar/<network_name>).

Example local_keystores.cache:

  {
    "version": 4,
    "uuid": "402a9ffa-8c4b-443d-947f-473fefdc1729",
    "path": "/root/.local/share/lodestar/testnet/cache/local_keystores.cache",
    "pubkey": "81ef1d058664e94bcdd4876d049fc40e9a2d55a104a2fbcd33b63774a55bf994d72e6c7dc641af29bfa57362413db70581f9ac9f60825d682d5ab33098117dbdcf3c5245116c8c03a8c0493a5d441ba578a0b3d069d745cfdff70122c65e421a8257c261afa77e79086b503de88dad720443a1d135cbc14f8a6de408a03ac5b9c4263731d6693bf843f0a9657aa3c4e8837d8e7320247799d20afbddb410fda1bb7fcd31e36ccc841012e4dac0d643e8e5e12d2467d8d3219aab33b64280eaea84c1b3ec3752b9d7e3ddfe87c68c59443b5fca9d18d78e5441c80e4fb6c0ecec27f8074811c2e5f823364a71e056739484e8581cf13f7df6a96835ad1330593bf5d1f4e5bd6341f7a54063b0233e921250d0c09a48849155953e293cf635d7b6859155dd5a22f116ae8f61b1516770f8ff41ec0ea24b8b745171b4cf34981bb7d235e7e1a739a0589e7c7ff69ede9b15863fb35bdce0573031210c5a6d3521b5a3e11cc99356de9e8adccf3accb4c4d387f954b202d0db647e65c331b5019226870342ee85d1d3eda564de4126f20880d59164e2f88652d9dcf3dc93d0bf19e22ca3a11305f1cba1cadaf2d11702893688ce4d8fee80abef17438d6e1dee0da0087b3fde540c7cd157e169866c603af0af26fe3c3d2ba517dead10376d60df898afc5f7128e998c2a59b9e2dd1805ccda56220d1bb25ea94c13d4f9abb4ab55a522518aabface4f10d54f45d08dffa8d8bf6583d6de04a89b9ca61c69977de4ca440c4c9b13c7a1c65e205979849376e239d03cb46d8bcdf58afca96a8e0fcfa8c3f73416c86d93ae2dbc2a468b5d6cc39d42c2ea7e20cc215212d3dd3b8efbed324bcd4a28de941fe855c54a92c69738cb3b628e5ec89b1cae8fffb946a1277613deec1f3f0e7d75708fe3f6b15c5efb75a0ec8dce885812cf14f3d589b6f918d19ef6c96e7ea917640ea3ad0c6c6a9ac8320456c3ec046cebaa625d415476f71d854d186a1758c79843925b2d268ac8d1df9e9132058de96486f102ff4ef6e34c988b6dd42a9462954218b8728310bdc25a4251c092eec8128bfdde893049e8de14b70a834e78b74d6bd46bd0f5f92d878a1b19f9081808dae86391aeab2a048368f0ae5b93ab412561db70482dba08f14d19885ec7d1bf7b6c4668d505283d68d1af78588532a15aa80ecf96a2d8bbeb813f63c25c15cd8f04c10fca4c49f8faadcfded5c85beb36f1cdc234f0cfd8ebb0820a18899445accf3e6e35efc0ee34419eeb0cfb097dff5fd6462b9ed908fe444e4b5610d3583a667dcf23af26a9f686db05e7de9ebc03f8ea0a756cc96b4f55611f5788d61f26e5bda1df70f688ff3e5eb9c905e42b88cd8fd8593bde4d44e22cbf296d47917a4b0f144fbbe2ab69a4421b2c0ff3500141dcf88a0b007902b0898a017b3e98d334cdf49d5411e507f6043fca0624e937945c7dfb1829a3a2e1d0bdba654aef7d2e3c14c76b48d914dc0bb6af111bc9021887ca4dfb27cc2063c9ebd2be133754d30ca18fa8698787005538f556bd94867cc1c5f7b817a9188ee0eb50d3a88a27a2ce4334ea8cac1662593526f4c280cb3049bc91afd8381ddbda124b9fab871aacf378ab5380a9218096756d3ccec228caddd27979b9050b58d7304f2be4f7d4aafb6ae19fdab0bb9149ce29517a383f59facaacf6d39933d88e34601df31b7a66bae1d49079548a2ae2d72037c7c3b2fb8925631989c1ae353311c36fe325b2f1fdc1f648194947e056beba42f6d7fec6712de16458c85ae513391dc4d42120e5877ed21a0681820a6150cb45af4f0cbf98c1d25c8cc94ef22c9183e15da2e4ad8e05a75c5b9201d52e9ad7f66cb0061b0c68779ba5a1fd0f11b9c365c1721d00199d287923c9592c95f5c1574b8c510e546759f383779a0364fefa009f84dfcebb2efc4e86b909a5fffb90ed79b5f66980b420db7e495e8e9e5c389be338759c40d2e408b1f0b78ebc0ffbafae360e33b683ada3638414338b83490e9ab1ff067425d25d78598249520dcdc8be36b8afed019336e2eb478d44a0f5ebb6d7c994710856abf32ed25313867e060f83b205ea5ebe9f0fc98eeb5417011d88a2924cbe7ff7ed616a6b7dfe273187360757c667f27349f525ac60665f9f7ba2d07d91d2f94566f1e995c28e8767e8677ef93cf6aa49453a8ad6c279622820e321ff9d352ceafffaa7ec082713e4c85e50cf9f7e11439cc16999664783bb5eb59491a99142392f020fb4e3b607203c3f9bec7ccb3537e8f7856d7ddfbede9dc3332cf40bdd4334a64a03f7aa96fd1dcb385cf9f4f29f9f3ef4c25a47efdb090c18763a51608e92267daa98695f765705a7630269561f9086ca0833e9ca3181e33a9b51d7c31722cd07c8f0a18a34ba083eaeb091529c2a454c3950ec449ab7f6f237995bf1ec0a802a1c646e753dd9d9811cd75f58c371e74ce83f05606bc076d64bdc77adb313b1bfd8ceceae38d8a8b7d4795f10ac68d18a567c07d1f258a7dc4f685b9c45c3217e9e640d8cbb3fde3a875e31b0212df6d48985f8524922205aaf6917a5b577d89a608c06384bc606f723cde2a6f7d64a29de9cd987c0626dabea2414bd5b653647b31953c1156803191484d5ecf4630f1a6094298efdad03170daefba1bf92ee8691aa86fb753afcc252090c74c5f97c7d6cdfbbe0017f57e25d406d4ccebdb0ea62b23a8a25355c20cf4ebf93bc43a7b4076ab247ac7bff133ddaa7cfb9588e598d1022285b0e2f2fdd0ee3fd51a6f39a757657d95e795d3460d5454b0f3987885f0a8138c4de92e08d2843709f808adb191c9d2b22399bb9445dbb94e190382a83c93593c32b08e89f0089e1a0892dcf121a856199b751ec40959433b4e64ee7fea260a16f4929261c5e2ebc148042ca8f3c2f7371294ca58c85e7b1513c155a39e5bd76eedd90f50a4fb50d6ae55b40623d5ba0c5d85c40def89d6c67d5fc3a94d36839c1557727aa55fcca24faaab0d1e5f0ecff1dd709e22f0c55408b8b74d22173bc312d27baaa8688ae1b4be2ca9f291de2f415ab3a4002206769493d82a094e1934b78d98b79c93f70ffff7389d8ea0962b34b6df04ba999442a0fda5aa228e16f801dfbde52d5f57b44e7e9a613fef33f893fad979ce003eeed012c8f45e9050db814e241be87ae94c0c4011aaafd2bc633a130d0798c8f398ebd00f5c1b131c2b4d48cbaddde1c8ed59eae6af8290d27700852df2b3cf23d630d807adc1c39301fa1fe99678a7b7887e895c8df24e15546b13d2237ab2795cc7004e6b68e69724f2c0922f119d5af8819bc9adcb081ad4dd8f1acfdf1a71360c6d5655bdd58d9bb1f09e4de43b7ff8a6da60b61df9ce65f9ffc951740dfc69812667adfdf512e8ac8e3f01ab9a03733c6d0bd0a5403778eebf691d2906380bc01591e93b43cae86c94f657c92c29d5698a06ae2bb170260de9afb23f49d5770474a65b4d380d904aa41cab7c2852074ed8ad9fb94f6d7e5a5a1f6f71c0dce0bddd12b05cafec5912f22dbd6f15677f25f13d93ecd5ec6f957fddd7cf27d73521b34aaaf6a219f77b21128d18321c2c8d679bb06cb2010c1167c72840b3149ba92b326799375b1b05a1c0ca38ace5e8f61cba48c1b350a7220938e9c2c0fe6b6c2881b0862b7f0788739e315de558384d6f95531088016f18330ab572fe89f3600098284f7a08a1ea8e7e33666904b918b17cb0db46ced0115b365df2d2c1e29ef3333b0bd4ed297288f7a09ca9c1de5e702ef8f2cbeb89d0d70a584cfe991cf7bb65b1748deeb17775232e5b53f55bc2d1b08c494a80cb727a2f2361af61464afa85a1afcae978e5d802e6c75e3be60965dab7c37e964efdbe3916e611daa4a1241bad3aeedf6a57a21c87c23bec872452c05816ee9179d021c3cf36843063c687d0b7ec2f481129da715b78d3c6bc1ab2e04bedadb937812497196409c08d7837d133c8bf52aec70689b1b180d8eef2676cb7f216edfaf073f84d71ed41b7376d8fe85c88b40297636a2870278d00b452fd37b41af5e357a2dbe1297b53fb027e9ab83d8622fe3180d6e8fd95b59d5e5bc6b7c0451040ec3b14588b1405410bbb249c15d449dd9afa4d9b42650b26a58fccb869ba1794050a014193ad467452efa3a54bfe6c6d1689bf7de9576eac2d2c2dfd4383a219e2d450b00cd9a70fc5e2a7b8a434ac1fa9c6a2c2c37da065576261d981d870923222a50b225422097a3e59598f548be8c6bc0b4764c02546f11b7cb956f93f164469f3bb6b2e95c7d3db7d979b03457be68c6b227a9bda9b4be639198e89c9a62452162640cf95a00ad339",
    "crypto": {
      "kdf": {
        "function": "pbkdf2",
        "params": {
          "dklen": 32,
          "c": 262144,
          "prf": "hmac-sha256",
          "salt": "cf4cd3d5e53974d54450ab228e4c4afbfb31ec35f169da2b39b7e115fd285e78"
        },
        "message": ""
      },
      "checksum": {
        "function": "sha256",
        "params": {},
        "message": "fe3f157ffcbe91120fd60cde10bf1c6dad27d7693b859fdf15a806baf7fa41d1"
      },
      "cipher": {
        "function": "aes-128-ctr",
        "params": {
          "iv": "9c1043f98617ad44af506887835e3dc2"
        },
        "message": "f65a70134da92d495cf305890b76fd39eb74210c305a9e157f3f4a3f2bc7d4fabd663a8df2503d5afe5d2ac6ff5a242b774ccbd1557255160038c93130a9ad3d4bef60be5b1b88bfa44fc368643252e5627523d4800d959e6c34e7477597b9226051ddd570e96fe1a4347ea0a61cfc8af9a37c536ffc84f6da6ba2541947405397bdd1154a53aec7cef40dbdb9bff2227362e1b5d896d47bbdf48dace9413384f04b828fad0f893d3a941b6cbfc75fff1a860a80d89331103e5402b9ca749464f2b007fd21e9fee134ba915c0d09310ba3b09675e3d7e856f372d6c36826a3798cbfb46e05a240ae79bab6e7151921e1407c5def5a07892a1fc8dcff3f05eb320fcb025f225897972de64e8a5580f92de015b4e3b2bfe4c525ec18c7a58e9b780a674428cc15117d5f824bba43fa642c333062380419b761662d3c1961b2858d85830b97b1a7fb49346108bc6957f5292b7bf92091f7e180cc901383d3ffa6861aa7e6bfabdc9e157a42914c1be049ce896f97210f4baa7934f85a6c7123f53393df2a2a64b05ffe81e9568c616f4433eb0eeb81d18c0882567482bc3308b1c2a051c9dfa986200c4f86e64dabe3c5581a76d93c4a2f3a0dc3ea4d3fcd62df3ec612d45a4d2241949808f1c9a14343a77b560725b619cc6de1afd8175ca7a8c0f829faa02e03995aebb25db91f9c0252227981b0af7d9dbb2220b34aa32d7d6304e32d1d08de3f662e5b723791714370939a181adeb5dcb2490db142942fc165b369b65f44c7247ae4ca0f68d1a552fc15b5f33147f0580b70599d585c3aaeccbc7b6b38386397fe733dc34131a84c3c7d52be0ad8cb7c077c1cde879ef2f98d121572874cdce22e8a7e628a73d7e54ccde2406bfd726d471ef7d7f48931e3acb0875f43bbb59607c4c833d9efdd1ec8a6b5d7c1c7777680535fbfad5c8df872d54622a0781357d210ef1908ddd05719c479e8fed29daa6919e55aefbee6ee8202324c00585622cb1d304fb75c80b07f4f0529b4fddecb2b4876ac3be0a09041427f2bcb14d9ea57f6984028bc7252f7beef3de5d0a6c906896daa64509b02fb1507230d183b1b890e5d2be86dccac10dbc06ab700b26ae3b15507969cbfadea25f41f4f88d60714f329547015f27ceee5b27b3115e93401f3547ca40880aed57ec49f01390188f450b24fff38002db810f4a9b378d23f75d82ff96e96ad8fa428ed0124767a4dae885819148bd2845ee09b97ef0836110af16f95a0cf9058ac779e197877e65cbdce09a7e5798db1b324e4d7ba6367bcf7da4d400ac59ceec71be4683eadfcc434eb06dbd1f5382190067cb30a2f3b4c54181b81d35087c03289b5307685e0fb12d625dee46466e4236a817a2ccc8e522dbcef5929cf1b5d00f20d56aa4e8d2db09d7145c4465adf029c9911efcf1634a41bdcf2284dee6893c9efc15ee97dddb6caf6748bb1ea1404f8f682e319bea327be96fadd0aef1f3d58029593734905faccbdb6093dc92b5398f43c397e69e30e47377cee9fe41bd7432a94c457bef39307cdd7c3b904776267d893f0704e8a878727c9c3e414469c5628771c9773699a9869f0ab4d43a7b2c08a87a20b8a6f082034bb51e638b94f38ef7f70a4151603823c8fb8a4af2237680560c4f518faa1c1e03157fdda3520ae3863ca548ee6cc767b18750eeadf869382e8d231a5b29cd1c6af2e0cb2d1fe154dee71feaee60c4547da431e916956cfc01bac9e4708d3fb72016bb8ba9436094f83ab178f75f9a78df1159f77461aba8c1ace1a84e80f84a8df45fa37e504a17651732636fbd3a37d6fd78be851b8db40ba414823bbd5671d079393d9fe34e8b73b9e472ba81840adf833cf9c9ec5014bc5c40d39e0c970819122e85b4a3d117417e7d1ca510dd79673c84bdcc86d751993019028cdcb6a3b45d7ad71f59805e828bac9c978086ad014f2173369ae40ce6c064139fbbd48e8454781152ef5546035de6a50a58c57e8a7e1105227f15cc0b6148d1d79b15c8aa9fa6be48d5dea040acc238c1adb2c1c3ebe8db8e6a2f13798e75b01bc7cce4f4d53c6c5771a54b07ff9cf5046f3ea920a219449f3c188c6027226751f08c6d0ec0f36ed0329b85bd81cc857f01b32c338706243b4a85ff8b95ed7b5635d0ca91ae16acb2b79eabea62c8a6e8e5814fbcd3ef17bb7f547ae77a60b6a3426495e80acfdb6a2e1da74972a5b53f32972bdff0a17b051336ab0a2e65a654dfb1c79902a5d593f15d75d2dc546274bdaa83d5d041be7e53d973865a207e9193b81886220b342ab113968d6414794435824842c159e5911f44774b0744e2fa88394f50dcd4eb46a3b8899ed50faee7412038264654814613fa50b0727281e8f8df2e107990635312226294c01407a1ff1cfd8f0867e8a3413b5bcda6183c189013d70e45def8ff41e720468ae9c816ccecc9b2df4e0de539597a8da7e55b9f5081471062e7b5716ab2b345d21737fcb5f5b42e72ca44960d12026ea653a7755549b5f413c9b9aafef2dda81e44b6c51f8ab44d064705ffdee94d5298eeb7face9fbb35fb26d40712be73a58cc59584fec17976ce5ec564ecd062a1deee737fa74e904e27cd137a69392abc9e46052fc46f57eb3cf6cedf2ef39cc0f685cd32a9cfaade27375aa6ea563ccf1c28b0c1f842312b6a5219df8a4222aaa3ba86a1445b036178a70874bdb4ab0d6b2a1818f64a103086933d47ebd9abe772c05d364417a67d3de44d4512c2231db6c45ad7d374b2d7446fa1a14aac44f21f7c883c1bbd0db3b66cee1d34754eb49fad04143a636e81b34ed06c49fde453b58087822ffe21e551f5bd3bd2de1664c3a03ddbfb550ad4facefdc452def1c96a16ecd0fde934e86fe14b246e9"
      }
    }
  }

The keyfetching from this cache file takes a constant time of ~4 seconds (even if the number of keys is increased significantly). - Majority of the time still spent on decrypting this file, as this cached file is still enrypted.

Would it be possible to add functionality to generate this cache file during key generation? To derive this format this code bit is used: https://github.com/ChainSafe/lodestar/blob/unstable/packages/cli/src/cmds/validator/keymanager/keystoreCache.ts#L65-L85