magnetico icon indicating copy to clipboard operation
magnetico copied to clipboard

Published 20 hours ago • verified publisher icon boramalper

Unable decode metadata name, so metadata lost

Open ad-m opened this issue 7 years ago • 6 comments

Hello,

I would like debug https://github.com/boramalper/magnetico/blob/master/magneticod/magneticod/persistence.py#L70 and seen a lot of metadata losts due unable decode metadata name. The string is hex string.

Can we improve encoding parsing?

2017-05-01 08:23:22,259  DEBUG     Unable decode metadata name '5548534d2d30303443' of ca4f01ba1ef5e4dc13b756d7c6fb14b3fc9c55a4.
2017-05-01 08:23:38,666  DEBUG     Unable decode metadata name '5641202d204e6172636f732c20566f6c2e203120616e642032205b323031365d204d50332c20333230' of 315d5d3f165fd8d12b67e2ad5685df35b28a24c5.
2017-05-01 08:29:35,748  DEBUG     Unable decode metadata name '54696e7920546f75726973742077697468204855474520546974732067657473204772616262656420696e746f20612056616e20616e642044657374726f7965642062792042696720426c61636b20436f636b73' of c57ed7e087e27a0c9180d7b65668c59c8126adb6.
2017-05-01 08:29:38,653  DEBUG     Unable decode metadata name '4c656f6e6172646f202d20546865204162736f6c757465204d616e20283230303129' of 5fed97244fd3fc2f5ff61388e5ba7cf0f6a55a53.
2017-05-01 08:30:37,237  DEBUG     Unable decode metadata name '2888ea94ca8fac90e029205b8384837d834f8360836d837b838b5d20835b838d82cc8e6782a296822091e630312d3136' of 4d47f156676537d72f8ad722a9396dd8446d996b.
2017-05-01 08:31:09,349  DEBUG     Unable decode metadata name '436f646520476561737320426f756b6f6b75204e6f20416b69746f' of be8b17ff42c27a2076f326a8824fc909dadbe534.
2017-05-01 08:32:09,929  DEBUG     Unable decode metadata name 'd1ebe0e4eae0ff20e6e8e7edfc202853617452697029' of f2ff4438ba9f93d17efc568f8906736ad396638d.
2017-05-01 08:36:15,424  DEBUG     Unable decode metadata name '5b416e696d65202d204954415d202d204e656f6e2047656e65736973204576616e67656c696f6e' of da5f79cf2e15ad7338ee60f2a953bcc295c0cafc.
2017-05-01 08:37:33,667  DEBUG     Unable decode metadata name '3036323531365f313132' of 7433586db78983b69806f202fd1d6541f8ca8c06.
2017-05-01 08:38:23,306  DEBUG     Unable decode metadata name '82ab82e182f182ab82e182f1836f836a815b82552069a56d61696c' of 2613943ca92c0358d0baf693cc3b72ab7d3c5cc9.
2017-05-01 08:39:02,494  DEBUG     Unable decode metadata name 'cac2cd20323031332e20caf3e1eeea20ecfdf0e020cceef1eae2fb2028323031332c20deeceef02c205341545269702920284d5944494d4b41292e617669' of 358a3affcae7b070f50d8eaff7ab13950a5e59f2.
2017-05-01 08:40:07,964  DEBUG     Unable decode metadata name '2831388bd68351815b8380295b3039303432345d5b414c636f742083568367838983585d8e80905f82cc90da95ab82cd95ca97a382cc96a120286d64662b6d64732b7272292e726172' of ea211d7126a86bc70d76a3478f4b84bb5a81e9ae.
2017-05-01 08:41:05,967  DEBUG     Unable decode metadata name 'd0e0e7e2e8e2e0fef9e0ff20e0f3e4e8eefdedf6e8eaebeeefe5e4e8ff' of 0be6966ac062d15bd89d6dbef0aa3a3fe4e44ee1.
2017-05-01 08:41:11,108  DEBUG     Unable decode metadata name '75736f66614039396262732e636f6d40c8a4cff282989853b2bfa1eda3b1c5aed7d3e5bca4cea5c8a5a4a5ecb1cbc5aea2da' of 546409e11be246fd9d286492c07ec8b97e5d1a48.
2017-05-01 08:41:40,338  DEBUG     Unable decode metadata name 'c0ede4f0e5e920c7e2eeedeaeee2202d20cfeeeae020e5e4e5f220abd1eaeef0e0ffbb2028c2e0ebe5f0e8e920d1ece5eae0ebeee229' of b442a588685219bd42abd7a0334dce6105acd6ed.
2017-05-01 08:42:29,153  DEBUG     Unable decode metadata name 'caf3f5edff2035202d20f1e5f0e8ff2030322e617669' of e6dbb60f194d1742de709065bc0432e5fed79515.
2017-05-01 08:42:42,358  DEBUG     Unable decode metadata name '58562d373839' of 15ac80639339974462dd5e0e32b34e7633b94b77.
2017-05-01 08:42:49,757  DEBUG     Unable decode metadata name '4665646f736565765f47725f2d5f5a6c6f795f6475685f59616d627579615f2844756e6165765f477229' of 72257a5e29be15318210169ba6bcc7e4ff89b2ee.
2017-05-01 08:43:27,047  DEBUG     Unable decode metadata name '3036323331365f353936' of 951cb4d9eb7001cbb97e4b44205f5eb7fd1c30dc.
2017-05-01 08:43:43,255  DEBUG     Unable decode metadata name '416c706861204672616e6365' of f9a6bf527a0e12d1ad80fc8ff5cb1a0375e5f2d4.
2017-05-01 08:44:04,684  DEBUG     Unable decode metadata name 'c3eef0ede8f7ede0ff20285745422d444c52697029' of 803e6e9265492ceb9d1bbaf6202533fef9dd04ed.
2017-05-01 08:44:37,183  DEBUG     Unable decode metadata name 'd1eee1e0f7fce820cce0f1eae8' of e80bb3b1ec65b8a1ae84deebbdc482fd512f0d86.
2017-05-01 08:44:39,196  DEBUG     Unable decode metadata name 'd1e5f0e3e5e920cbfbf1e0ea202d20cef220c3f3e4e7eeede020e4ee20cbe02dcfebe0f2fb2028cce0eaf1e8ec20c4eef0eeede8ed29' of 70ba67f0146d3da274d503d06f45d023c785ffb4.
2017-05-01 08:44:42,599  DEBUG     Unable decode metadata name '546f74616c2e5761722e524f4d452e49492e48616e6e6962616c2e61742e5468652e47617465732e696e636c2e444c432e4d756c7469322e762e312e31312e302e537465616d2d526970202d204f726967696e73' of dcfbcba77e08fa56d3f89aeb25185d33798f93b3.
2017-05-01 08:46:43,501  DEBUG     Unable decode metadata name 'd1e5f0e3e5e920d2e0f0ece0f8e5e2202d20c0f0e5e0eb2e20cee1f0e5f7e5ededfbe52028c8e2e0ed20cbe8f2e2e8edeee229' of 717b81cd59962f87434c3fc5860cba8d12cb6d37.
2017-05-01 08:47:00,859  DEBUG     Unable decode metadata name 'a650a8c6a454a4c0bfcb' of 1c1e78bfb3c80d9e8a26c579b6c05676af711d9b.
2017-05-01 08:47:18,511  DEBUG     Unable decode metadata name '506561726c20486172626f72202853636f726529' of 1e35dcf3b86fd5905aae05cec90d4db60eaedc0b.
2017-05-01 08:47:48,173  DEBUG     Unable decode metadata name 'ced7cac820c1c0cacac820c1d3c3c2c8cdc0' of edca61accf8d2aaefac013665c44ca31cb7bfc8f.
2017-05-01 08:47:55,860  DEBUG     Unable decode metadata name 'd1e0ed2dd0e5ecee2032303133202853616e72656d6f2032303133292028323031332c20506f702c20484454562920284d5944494d4b4129' of cfcc595e9e26369a5d2070710f46c61d68f7e3b5.
2017-05-01 08:48:10,401  DEBUG     Unable decode metadata name 'caebe0f1f1ede0ff2ef8eaeeebe02e323031332e587669442e5341545269702e5468756e646572' of 0b6171b7da40cc53b2f2745db6ad81f1204d53de.
2017-05-01 08:48:24,940  DEBUG     Unable decode metadata name '3133cbead0a1c3c0c5aebbb9c3bbd3d0b7a2d3fdcdeac8abb5c4d3d7c5aebdd3cadcbca6b0cdb5c4b4ddb2d0' of 733d6c5e25f198a096477d3f3e135d7d1f1de3e8.
2017-05-01 08:48:40,590  DEBUG     Unable decode metadata name '444554454354495659' of b9e4c641198cac0ed29a9b521cbedef7d54d56b3.
2017-05-01 08:49:46,893  DEBUG     Unable decode metadata name '313131313131313131' of 01f911de5c71257ed3021b0604fb42b11c45915a.
2017-05-01 08:50:16,003  DEBUG     Unable decode metadata name '3032313731372d333735' of d92863ffca7431ba0eaedd12c52beed28e10434f.
2017-05-01 08:50:22,811  DEBUG     Unable decode metadata name 'd8e5f0ebeeea5fd5eeebecf15f5361745269705f62795fc4f0e5ea5f522e4720d1e5f0e8e0ebfb' of 9e5a4b46e395261dba08029d64cda32eee2d139c.
2017-05-01 08:50:58,859  DEBUG     Unable decode metadata name '31362e4d79204669727374205365782054656163686572203533' of 0252a5de79f3dafc34688c352a0ecffb91580b14.
2017-05-01 08:51:25,660  DEBUG     Unable decode metadata name '6d7572616d7572612e74762d3032313031315f3337362d48442d414c4c' of 4dca25b6639278b3c167a867149bfdf24c6a3ebe.
2017-05-01 08:51:39,258  DEBUG     Unable decode metadata name '5652544d2d30303152' of 2e8a93e2ab17983ac3efc7a6092ff5165ab3a706.
2017-05-01 08:51:46,293  DEBUG     Unable decode metadata name '3132313230392d323431c9dfc9e0c5aebdcc8e9f20c7b0be8e20d0c7839ec4cb' of 3ed84ccd268c7b898206956ad5ea4ce77d648ea2.
2017-05-01 08:52:47,991  DEBUG     Unable decode metadata name '6a756b756a6f2d636c75622d36353538' of c8940efbc4715eb8511a2293d205d995a36a1480.
2017-05-01 08:52:55,610  DEBUG     Unable decode metadata name 'cbfee1eee2fc20f120eff0e8e2e8ebe5e3e8ffece85f4456425f62795f434c49504d414e2e6d7067' of e4738d61e9fe9a0804e9ff9f3ebac104ba3ba946.
2017-05-01 08:53:08,332  DEBUG     Unable decode metadata name '367468204e6f2e38342041594140686b706c617a6120627920617273656e616c2d66616e' of 546bace76297ae290543e643f59e82a5b2122a4b.
2017-05-01 08:54:25,570  DEBUG     Unable decode metadata name 'c0edf2eeebeee3e8ff205761726372616674' of ab09f7ba43fb31ff867c71021007dd0203763eb1.
2017-05-01 08:54:34,197  DEBUG     Unable decode metadata name 'cce5f2eee420d4f0e5e9e4e02032202853617452697029' of a873348cbf13b2252f797e64803ab78d88480c6c.
2017-05-01 08:54:41,278  DEBUG     Unable decode metadata name '41472d303839' of 3ca89864da5e64eed8564be3bcf6abc71d3bcbc2.
2017-05-01 08:55:11,561  DEBUG     Unable decode metadata name 'c0ede4f0e5e920cbe5e2e8f6eae8e92c20c0ebe5eaf1e5e920c1eee1eb202d20c2e0f0e2e0f0fb20caf0fbece02028cae8f0e8ebeb20d0e0e4f6e8e329' of cc7b5c4b1440320bb93236b95267f08abdf84145.
2017-05-01 08:55:30,974  DEBUG     Unable decode metadata name '4856592d30313543' of f78329db99808fbbed2ae7fc1fbfa41bdfee28d9.
2017-05-01 08:56:52,717  DEBUG     Unable decode metadata name '46502d30303420445644' of 9155d167d49eddb287e15895557b766b52259477.
2017-05-01 08:56:53,352  DEBUG     Unable decode metadata name 'cce0f2e020d5e0f0e820284844545652697029' of 1e20118440b88206673f5dc9162835f92b30dc96.
2017-05-01 08:56:58,216  DEBUG     Unable decode metadata name '42656c2043616e746f202d20372e20416c62756d732028313938372d3230303329' of 3731ee073987fe555fea55c3d798eab2d1c88cd0.
2017-05-01 08:57:04,668  DEBUG     Unable decode metadata name '416c636f686f6c203132302520322e302e322e353632392b52655061636b206279204421616b6f762b2052655061636b206279204b706f4a49754b' of 858d0f73bf0e8bf50a4338e0f8630b968a83a609.
2017-05-01 08:57:15,649  DEBUG     Unable decode metadata name 'c8f1f2eef0e8ff20f0f3f1f1eaeee920ebe8f2e5f0e0f2f3f0fb20323020e22e202832302d393020e3e3292e20caeef0ece8ebeee22e646f63' of 2ce963bf8e3a26a228d22448b71ed6637bdea2f1.
2017-05-01 08:57:59,807  DEBUG     Unable decode metadata name '44617669642042697362616c205b446973636f6772616669615d5b506f705d5b323030312d323030375d5b7777772e7063747265636f7264732e636f6d5d' of 6b063e6b363b4e7d5a4f0725633979d557f969d2.
2017-05-01 08:58:00,973  DEBUG     Unable decode metadata name '4f78666f7264204865726f65732031' of bf90aa17e24b491e2b881ab190ddc93e6ad3a20e.
2017-05-01 08:58:06,744  DEBUG     Unable decode metadata name 'd1eff0eef1e8f2e520efeee2e0f0e0202d2032303130' of 162aebb51e5ad0f4711e3a4a92e239d39fde87f5.
2017-05-01 08:58:14,324  DEBUG     Unable decode metadata name 'd1e2e5f2ebe0ede020caeeeffbebeee2e0206d7033' of 397650f79fd56ef7ca56cca5a6d683dabb3c0155.
2017-05-01 08:58:34,776  DEBUG     Unable decode metadata name '48303933302d6f726931343430' of ac08ade13512e7e9864704eb59594c475afd1a2c.
2017-05-01 08:58:48,688  DEBUG     Unable decode metadata name 'c9f1ccbdb5d2c8cabddc4949c8ab3430bcaf2e323030382eb9fad3efd6d0d7d6a1eacaa5b3c7b4bacaf7' of 08110b73374d27c4b7a9b83bab34aec5ed98039e.
2017-05-01 08:58:54,675  DEBUG     Unable decode metadata name '6865797a6f2d3037323020627920617273656e616c2d66616e' of 1011fac8a1e7f043396de52cc3392d0ed8af92ce.
2017-05-01 08:59:51,257  DEBUG     Unable decode metadata name '6c6a7a74793535354031387032704050656570696e6720486f6c65732dcdd1d2c2cbf9cfb5c1d0' of b78bb4a0b69468f1cac9d0819795b71b8d108dd6.
2017-05-01 08:59:58,261  DEBUG     Unable decode metadata name '64696c69383239407777772e736578696e7365782e6e65744049204861766520612057696665202d20416c69736f6e2054796c6572' of 5558256a9a7e0ac75f5e6a517e070437370dbb80.
2017-05-01 09:00:23,408  DEBUG     Unable decode metadata name '53444d552d35343652' of ac4dac918357a529e81d96d5cea11db2c5c519d0.
2017-05-01 09:01:05,583  DEBUG     Unable decode metadata name '47532d30323652' of 939080fa14c5c51796cee30f79f9ceead928c8b9.
2017-05-01 09:01:56,819  DEBUG     Unable decode metadata name 'c0ebe5eaf1e0ede4f020cff3f8eae8ed202d20cae0efe8f2e0edf1eae0ff20e4eef7eae02028cce0eaf1e8ec20cfe8edf1eae5f029' of e732b956a55388a03005bcfc8b5c50bdd5a5eabd.
2017-05-01 09:02:18,471  DEBUG     Unable decode metadata name '4465795f4b696e' of aee4f0f5aeeb09f3d3a67868b573fc11120379cb.
2017-05-01 09:03:15,485  DEBUG     Unable decode metadata name 'd0eee0ebfce420c4e0ebfc202d20d1eef7e8ede5ede8ff20e4ebff20e4e5f2e5e9' of f04392b74f88f98ed1c6a7d8c0d28a8790417579.
2017-05-01 09:03:20,227  DEBUG     Unable decode metadata name '42303032a5f5b1e6c646a4732d2db944c3d2aa6bae76b67da5dc2e69736f' of 439359e3f09c5960e38424eb1ec94e3793c19dbc.
2017-05-01 09:03:34,241  DEBUG     Unable decode metadata name '2831388bd68351815b8380295b3035303431355d5b82bd82c182bf815b5d2869643d31353236363129968290a29286836e8f49838f838983698343' of 618c72daa4181ff252aeec2d2fe76083c4fd54d5.
2017-05-01 09:03:35,686  DEBUG     Unable decode metadata name 'ddede5f0e3e5f2e8eae020e820f6e5ebe8f2e5ebfcf1f2e2ee' of 7e67ef954f5b5e8af19a3d75d6a76e92897c46a3.
2017-05-01 09:03:43,804  DEBUG     Unable decode metadata name '6b696e3874656e676f6b752d31343638' of 17e43c5945e8555471ec4616f8a108afc1d59861.
2017-05-01 09:03:59,281  DEBUG     Unable decode metadata name 'c2e5ebe8eae0ff20e820e7e0e1fbf2e0ff' of bc00c496d31bd9140c0dfc3b0730999fc47469ce.
2017-05-01 09:04:06,607  DEBUG     Unable decode metadata name 'cfe5f1203220285745422d444c52697029' of c303e8f0396509906bd1f1c0bb72a79bfb7dda50.
2017-05-01 09:04:48,901  DEBUG     Unable decode metadata name 'd8e0edf1eeed20e3eee4e020323031362e20d7e0f1f2fc2030312028323031362c205368616e736f6e2c20506f702c20484454562031303830692920284d5944494d4b41292e7473' of 4eef7c71b531710d6c8a8ec87d11329a61bbf2ba.
2017-05-01 09:05:25,184  DEBUG     Unable decode metadata name 'cae0f2ff20cce0f2fef8eae8ede02c20cae0f2ff20ceeaeee2e8f2e0ff202d20cbe0effb20e2e2e5f0f52028c5ebe5ede020d5f0f3f1f2e0ebe5e2e02c20d1e5ece5ed20cce5ede4e5ebfcf1eeed29' of 1aacccabd8c10d75916d3862dedcca517fea5c25.
2017-05-01 09:06:55,245  DEBUG     Unable decode metadata name '53656c656e61' of cfc4218521d4dab3023f9fb784062d58b992dd93.
2017-05-01 09:07:23,431  DEBUG     Unable decode metadata name '32382e416e616c204265617574792035205b484431303830505d' of 22200bfbf52f95dd70793302d11aed487ca76e73.
2017-05-01 09:08:22,399  DEBUG     Unable decode metadata name 'c8f1f2eef0e8ff20d0eef1f1e8e8203230e2e5ea' of 7f59dd329c1ae4dd62581f7c9a14dc4fdc6c00c3.
2017-05-01 09:08:26,593  DEBUG     Unable decode metadata name '5bb9fab2fa5d5bcaaeb6fec9fad0a4b4abc6e65d5bb9fad3ef3334bcaf5d5b4456442d524d56425d' of 5df1619e4cbba69b28ff9c0d6d773959b6337e30.
2017-05-01 09:08:28,360  DEBUG     Unable decode metadata name '676b2d3030353220627920617273656e616c2d66616e' of 3923517322945a0b49b42493a6fe6af6c97ff714.
2017-05-01 09:08:40,517  DEBUG     Unable decode metadata name '974496d882dc82a882dd202d208f838c89834c836c837d' of 58548e66cbdb4200b4c12c289b3aab3a5473ff74.
2017-05-01 09:08:58,455  DEBUG     Unable decode metadata name '5bbaabc3d4d7d6c4bbd7e97777772e313137686d2e636f6d5d5bced2c3c7bcd7cbb35d5bb5da3530bcaf5d5bbaabd3efd6d0d7d65d5b373230705d2e726d7662' of 5b8654cb1fab9402ca419836d2ab1ae6484be3d3.
2017-05-01 09:09:17,441  DEBUG     Unable decode metadata name 'b344ab48bb50a6dec0f420536e616b657320616e642045617272696e677320b344c752c7d0c77cc7b5' of 03f4c7e5fbfb6b685ffced0d6f2fb1244aeca91c.
2017-05-01 09:09:20,151  DEBUG     Unable decode metadata name 'd0e5ede0f020cbe0f0e8f1e0202d20caf0f3e320e6e5edf1eaeee920f1e8ebfb2e617669' of 608dbd6d52acf0372a7f1c38f0bf5aa535dc99c4.
2017-05-01 09:09:43,507  DEBUG     Unable decode metadata name '4254ced2b0aec4e34073656c616e672e6e657440a5e2a5c7a5eba5aaa960a5c7a5a3a5b7a5e7a5f32033a5aba5e1bacdcabda5c8a5a4a5ec20766f6c2e3131' of 9856ca36f045d42e0509e9e348c4d3f1a4cea7e2.
2017-05-01 09:10:51,739  DEBUG     Unable decode metadata name 'c6e6c4b1c3eebcc6cee5b8a3d0c72e42442e373230702eb9fad4c1cbabd3efd6d0d7d6' of 5d0c3a6b113d5a5d2759ee1a5b12fc3def79e12d.
2017-05-01 09:10:58,224  DEBUG     Unable decode metadata name '727573616d61746575722d782e636f6d20686f6d6520766964656f' of 2dc011e86e18ebc1429e4ed425249eb524002f6a.
2017-05-01 09:12:47,733  DEBUG     Unable decode metadata name '67616368697070762d31303731' of 1bce86099315497e5b9deb1bb0cc4433ce5b1f9b.
2017-05-01 09:13:52,569  DEBUG     Unable decode metadata name '3230313030343234' of 794d5bfcfb8cd64ea8a0ac512e7348c1930a5913.
2017-05-01 09:14:09,772  DEBUG     Unable decode metadata name 'cbe0e9ece5ed20d42e20c1e0f3ec202d20cee7ece020e8e720d1f2f0e0edfb20cee72028e0f3e4e8eef1efe5eaf2e0eaebfc29' of 2c2375381e8c904660a3c040ac8c47d264f49b47.
2017-05-01 09:14:11,446  DEBUG     Unable decode metadata name 'cbe8e4e8ff20d7e0f0f1eae0ff202d20c7e0efe8f1eae820ece0ebe5edfceaeee920e3e8ecede0e7e8f1f2eae82028deebe8ff20d5e0ece8f2eee2e029' of 608d5c1be6b2e87fc8f5ee4d428eab0143b804d5.
2017-05-01 09:14:13,850  DEBUG     Unable decode metadata name '456c207365637265746f206465206d6920e97869746f2e617669' of e271153636ec627896ee5c92a9cfc85dd5945b87.
2017-05-01 09:14:18,490  DEBUG     Unable decode metadata name '6a756b756a6f2d636c75622d36373331' of f3a0a1b8686b7f4d40919bcd57195add871fa8db.
2017-05-01 09:14:37,767  DEBUG     Unable decode metadata name 'd1e5edf7f3eaeee220de2ede2e2dcff0ffeceee920f3e4e0f02e2d323031362e706466' of 1cb1c013d3f9462b324c18dc739428285d2fca7d.
2017-05-01 09:15:05,255  DEBUG     Unable decode metadata name 'c0ede4f0e5e920c2e0f1e8ebfce5e2202d20cff3f2e820c2eef1f2eeeae02028c0f0eae0e4e8e920c1f3f5ece8ed29' of 545cf412ad541f24741dbb28f72d2b9cf81951f9.
2017-05-01 09:15:10,180  DEBUG     Unable decode metadata name '446976582b6e696b652853444d5331313129' of dc68b9dd3c95c5f229da5b502d399736b926bee1.
2017-05-01 09:15:10,827  DEBUG     Unable decode metadata name '31372e486f744c656773416e6446656574202d20417468696e612c20496e6e6120536972696e61202d20416674657220576f726b6f75742052656c61786174696f6e2041205374756427732053757270726973696e6720466f6f74204675636b' of 9fcb754229c8481e6c265393f59786d01adca991.
2017-05-01 09:15:15,457  DEBUG     Unable decode metadata name 'c0eaf3ede8ed20c1eef0e8f1202d20cfebe0ede5f2e020c2eee4e02028c7e0e9f6e5e220c2ebe0e4e8ece8f029' of 8c531c0dc6c6d7aa5035aa6f7a3b7750c2d29c04.
2017-05-01 09:15:20,745  DEBUG     Unable decode metadata name '4175737472616c69617320546f70203430204d7573696320566964656f2043686172742030382d30382d32303135' of 0cecef27105087a90db9478253914f2ea675cb60.
2017-05-01 09:15:38,006  DEBUG     Unable decode metadata name '287264323831292b5370656564506c757373' of 4d6c152a9f196938b367ab42b9b9273eb07b557c.
2017-05-01 09:15:42,541  DEBUG     Unable decode metadata name 'c3eeebeef12e20cfe5f0e2fbe920f1e5e7eeed2e20c2f1e520f7f2ee20eef1f2e0ebeef1fc20e7e020eae0e4f0eeec2e2028323031332c20c4eeeaf3ece5edf2e0ebfcedfbe92c20484454562031303830692920284d5944494d4b41292e7473' of f999f806465e2b98d52129c6349ce0a39e54d9d8.
2017-05-01 09:15:53,209  DEBUG     Unable decode metadata name '7769666520322e3238' of fcd986da7ba004ff7a7f3ac569fb579d0034c1d6.
2017-05-01 09:16:20,041  DEBUG     Unable decode metadata name '3031313431375f3031' of 4c5b91454c43ffccb0e6721dbb511cfc1c770547.
2017-05-01 09:16:46,219  DEBUG     Unable decode metadata name '414220526f6164202d20436f6d706c65746520507572706c6520436869636b205365742076312e3120776974682070617463686573' of 90fa6e527629821922cd9255c178b637d4fcde55.
2017-05-01 09:16:55,265  DEBUG     Unable decode metadata name '44494f4e2026205448452042454c4d4f4e54532028caeeebebe5eaf6e8ff20333020e4e8f1eaeee229' of 9b563610c512cd31c59c8fa8f0f84b84d3f03bc6.
2017-05-01 09:17:19,821  DEBUG     Unable decode metadata name '5665726469202d204f74656c6c6f202d204361726c6f73204b6c6569626572' of 2132e74fea1e82196b2a7d03d24afc80c855ed80.
2017-05-01 09:17:58,870  DEBUG     Unable decode metadata name 'cfeeebe8f6e5e9f1eae0ff20e0eae0e4e5ece8ff203420c3f0e0e6e4e0ede520e220e4eee7eef0e5' of 062b7bfc6e3563d7244660404e97f20b90ea1069.
2017-05-01 09:18:17,967  DEBUG     Unable decode metadata name '5b4541435d20a3a8c3c8a4a8a3a920ccd2beaea4cfa4eba4b3a3af53756e646179206561726c79206d6f726e696e67205bb3f5bbd8b1505d286170652b6375652b424d5029727233252e726172' of badb24322f199a53c125c8d72d6a3ff331c9da73.
2017-05-01 09:18:38,737  DEBUG     Unable decode metadata name '446976582b6e696b652853444d5333393129' of f853f8c40f139036b1b103c7c59bbf25373e5d7c.
2017-05-01 09:19:16,061  DEBUG     Unable decode metadata name 'cce0f2e020d5e0f0e8202853617452697029' of 9f6bfec58f0739cb45bb9ef65c47d6d86f1dcc83.
2017-05-01 09:19:18,274  DEBUG     Unable decode metadata name '7a65706879722e7a68616f40736578696e7365782e6e6574404641442d31353937' of 53e0152c69381605345e551557c614d46b1b6a10.
2017-05-01 09:19:27,149  DEBUG     Unable decode metadata name 'cef0ebeee220cc2e202d20d1e2e5f0e4ebeee2f1eae8e920f0eeea20efe0ecfff2ede8ea20ece8f4f3202028d3f0e0eb20d5d520e2e5ea29202d20323030302e706466' of 121e0c0c47cbe99bdcb9b020dc14ac69735d0d2f.
2017-05-01 09:20:03,638  DEBUG     Unable decode metadata name 'd4e0edf2e0f1f2e8f7e5f1eae8e520f2e2e0f0e820e820e3e4e520eeede820eee1e8f2e0fef22e323031362e42445269702e31303830702e6d6b76' of dbb4f6d337ac78e510b45507307c4102f1dfa2f9.
2017-05-01 09:20:46,518  DEBUG     Unable decode metadata name 'cff3e7e8eaeee220c22ec32e202d20d2e5f5edeeebeee3e8ff20e2e5e4e5ede8ff20f2f0e5ede8ede3e02028cff1e8f5eeebeee3e8f7e5f1eae8e920f2f0e5ede8ede329202d20323030372e646f63' of 040d65629cc2e8f729d820ffe53373dafe382195.
2017-05-01 09:22:02,445  DEBUG     Unable decode metadata name 'cde520f5eef7f320e6e5ede8f2fcf1ff5f4456425f62795f434c49504d414e2e6d7067' of 0a6a3bfcfa055e02b81a6531c8f55d2f6962be16.
2017-05-01 09:22:06,116  DEBUG     Unable decode metadata name '546f6b796f20486f74206b30303630' of bdcc22ebb63bd2c7c911bc579d89e168c6f88a98.
2017-05-01 09:22:08,297  DEBUG     Unable decode metadata name '4d5847532d39353252' of 6281e74e206c3d1c124c2e915877303b86d01f8e.
2017-05-01 09:22:22,851  DEBUG     Unable decode metadata name '4e6176616a65726f732e4456445269702e5b7777772e44697678546f74616c2e636f6d5d736973696275746f' of 29a5cb2e0954095e48ed15afe421533fba122bf4.
2017-05-01 09:22:37,538  DEBUG     Unable decode metadata name 'd0d0d4cbb3acc8cb312d3530bcafcdead4c1d3ef4259d3e6c8cbd6aebcd240d6c1b0aed4c1d3ef' of 4e45c60be58c9eb272e76ed7ab09cd054b7f2cba.
2017-05-01 09:22:47,894  DEBUG     Unable decode metadata name '284d49474431383429' of 376ff1488587de528dd5cd6da22d47ddfb22716f.
2017-05-01 09:22:58,234  DEBUG     Unable decode metadata name 'c8e3f0eeece0ede8ff20b920303520283233362920323031372e69736f' of 290d1b23076f46e7296f2d92739bccce372bdd46.
2017-05-01 09:23:22,504  DEBUG     Unable decode metadata name 'cde5efeef1e5e4e020c7f32e323031332e587669442e534154526970' of e8bc6a219c42a85e017118facb1d46226fecfd77.
2017-05-01 09:23:50,338  DEBUG     Unable decode metadata name '4c6962657274e172696f73' of f06167cfb102c4accfabea706f2b9ab4356fa39e.
2017-05-01 09:23:51,036  DEBUG     Unable decode metadata name '4450482d303832' of c42fd5e8cd18e5c94a482f0b139c0e1b2f663470.
2017-05-01 09:23:59,513  DEBUG     Unable decode metadata name '5072657474792e576f6d616e2e4456445269702e5b7777772e44697678546f74616c2e636f6d5d736973696275746f' of 67c64c080fc32a4d7c7cf1ef20d7d0875870216b.
2017-05-01 09:24:20,161  DEBUG     Unable decode metadata name 'd02e20c7ebeef2ede8eaeee22c20d12e20ccf3f1e0ede8f4202d20c2f1e5ebe5edede0ff20ede5f3e4e0f7ede8eaeee22028def0e8e920cbe0e7e0f0e5e229' of 76d425c722ce4e7b573342010a396384daf6390b.

ad-m avatar May 01 '17 14:05 ad-m

I think we can use chardet to detect the character encoding, and then try decoding it.

boramalper avatar May 01 '17 16:05 boramalper

I was looking at the log messages you appended, but there are also some false-negatives, that is, an UTF-8 encoded string is reported as not-decodable. Try (from the 3rd line):

>>> name = bytes.fromhex("54696e7920546f75726973742077697468204855474520546974732067657473204772616262656420696e746f20612056616e20616e642044657374726f7965642062792042696720426c61636b20436f636b73")
>>> name.decode("utf-8")
'Tiny Tourist with HUGE ...'

How did you generate the logs? If you just added a logging statement right under the exception statement in line 83, then you might have also logged (mistakenly) for other kinds of errors.

I implemented chardet support and trying now to see the results. As it also slows it down a bit, I would test it first to see whether it is an improvement or not performance-wise.

boramalper avatar May 09 '17 08:05 boramalper

@boramalper , here is it: https://github.com/ad-m/magnetico/blob/broken-test/magneticod/magneticod/persistence.py#L88

ad-m avatar May 09 '17 08:05 ad-m

@ad-m Interesting, try running the code-snippet I sent and see.

boramalper avatar May 09 '17 08:05 boramalper

@boramalper , there is might some errors in filename too. Try download metadata by info_hash fron DHT network.

ad-m avatar May 09 '17 09:05 ad-m

Try changing line 74:

https://github.com/ad-m/magnetico/blob/broken-test/magneticod/magneticod/persistence.py#L74

Replace .decode("utf-8") with .decode("utf-8", "replace") (or .decode("utf-8", "ignore").

Example:

>>> b'\x37\x39\xcf\xf1\x45'.decode("utf8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 2: invalid continuation byte
>>> b'\x37\x39\xcf\xf1\x45'.decode("utf8", "replace")
'79��E'
>>> b'\x37\x39\xcf\xf1\x45'.decode("utf8", "ignore")
'79E'

richardkiss avatar May 16 '17 20:05 richardkiss