ShadowFinder icon indicating copy to clipboard operation
ShadowFinder copied to clipboard

CLI doesn't have a caching strategy

Open GalenReich opened this issue 1 year ago • 3 comments

At the moment the CLI recomputes the grid of timezones (the most expensive part of the computation) every time it is executed. Which makes the process very slow if a user calls shadowfinder multiple times.

Instead it would be good if a user could provide a path to a timezone json file.

It would also be good if there was a subcommand to generate the timezone file.

The end result would be something like:

shadowfinder generate_timezone_grid # creates timezone_grid.json in the working directory (or the given path?)
shadowfinder find 10 5 2024-02-29 13:59:59 --time_format=utc --grid=timezone_grid.json

GalenReich avatar Jun 21 '24 15:06 GalenReich

I'm new here, but I can work on this. Does this entail just adding a CLI command to do essentially what is in the README (and already written in Python, just not the CLI)?

borisnezlobin avatar Jul 20 '24 04:07 borisnezlobin

There are a few parts to this:

  • [ ] Add a new CLI subcommand called generate_timezone_grid that calls finder.generate_timezone_grid() and finder.save_timezone_grid() as in the Notebook implementation
  • [ ] Modify the existing find CLI subcommand to take an optional argument grid that loads the generated file before running the search
  • [ ] Modify the quick_find function (or refactor it away)https://github.com/bellingcat/ShadowFinder/blob/e5f636b3f4496ffd77cdc597380d3a531b71f019/src/shadowfinder/shadowfinder.py#L96-L99 to avoid generating the grid if it has been loaded by the new grid argument

Hope that helps! Let me know if you run into any issues when getting set up.

GalenReich avatar Jul 22 '24 08:07 GalenReich

Thanks Galen! I've done these things (correctly, I hope) and made a PR (#21). Could you look over it and let me know if it needs changes?

borisnezlobin avatar Jul 22 '24 17:07 borisnezlobin

Hello. Is this issue still open? I see the referenced PR was merged.

steve-bate avatar Aug 08 '25 08:08 steve-bate

Hi Steve, sorry I just saw this. I think the issue is closed

borisnezlobin avatar Aug 11 '25 00:08 borisnezlobin

Hi Steve, sorry I just saw this. I think the issue is closed

Hi. Maybe I didn't understand (closed versus should be closed), but the issue appears to still be open.

Image

I found it because I was looking for open issues that I might be able to work on.

steve-bate avatar Aug 11 '25 04:08 steve-bate

Yes, sorry, I meant that it was resolved... not closed though. Maybe @GalenReich can close it?

borisnezlobin avatar Aug 11 '25 04:08 borisnezlobin

Yes, thank you! Apologies for the confusion!

GalenReich avatar Aug 11 '25 09:08 GalenReich