Cannot sort discoveries by "category" nor "snippet"

[alaabenfatma]

It enters an infinite loop.

Tested client: sqlite

Reproduce the bug: While on discoveries view page (http://0.0.0.0:5000/discoveries?url=https://github.com/user/repo), any attempt to sort discoveries via "category" or "snippet" fails on the server side resulting in an infinite loop on the UI (no matter the number of discoveries).

[marcorosa]

Can you give more context, please? Is it an issue that happens with both the clients? How can we reproduce it (e.g., a script to test this bug)? Did you test it with a repo giving only a few discoveries (so that it's very easy to see that it enters an infinite loop)?

[alaabenfatma]

Can you give more context, please? Is it an issue that happens with both the clients? How can we reproduce it (e.g., a script to test this bug)? Did you test it with a repo giving only a few discoveries (so that it's very easy to see that it enters an infinite loop)?

Hi @marcorosa, I updated the description

[marcorosa]

Ok, but can you be more specific, please? 😄 At first I though it was related to the data structure we are using (could have been not sortable). Yet, I sorted manually some discoveries and it works fine

conn = sqlite3.connect('data.db')
cursor = conn.cursor()
repo_url = '...'
cursor.execute('select * from discoveries where repo_url=?', (repo_url,))
discoveries = cursor.fetchall()
sorted(discoveries, key=lambda d:d[4])  # 4 is the snippet position

So, where is this issue coming from? what methods are you using? are you trying to do this sort with the python library or directly in-browser with javascript? are you trying to sort all the discoveries or a repo or just the paginated ones?

[alaabenfatma]

Thank you for your comment @marcorosa No, it is not related to the data structure nor the database.

So, where is this issue coming from?

The issue is coming from the /get_discoveries route/function. Precisely this line: https://github.com/SAP/credential-digger/blob/c4b000d6994fffc7e9de2ce48d3fbb47516d4be2/ui/server.py#L393

are you trying to do this sort with the python library or directly in-browser with javascript?

The bug occurs when we try to sort the discoveries on the browser (javascript).

The bug can be reproduced manually without writing a script, by sorting the discoveries via the UI using these built-in buttons:

are you trying to sort all the discoveries or a repo or just the paginated ones?

I tried both 'all discoveries' and only the paginated ones, but none of them has worked.

Fix

Problem

Sorting discoveries (manually, from the UI, using the built-in sorting buttons), results in this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2088, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2073, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/root/apps/new_cdigger/credential-digger-main/ui/server.py", line 396, in get_discoveries
    'data': sorted([{'snippet': keys[0],
  File "/root/apps/new_cdigger/credential-digger-main/ui/server.py", line 408, in <lambda>
    key=lambda i: States[i[order_by]].value,
  File "/usr/lib/python3.8/enum.py", line 349, in __getitem__
    return cls._member_map_[name]
KeyError: 'crypto_key'

crypto_key in our case is one of the categories we have at hand. One of the discoveries happens to be a crytpo_key.

Fix

The problem comes from line: https://github.com/SAP/credential-digger/blob/c4b000d6994fffc7e9de2ce48d3fbb47516d4be2/ui/server.py#L393 We replaced States[i[order_by]].value by order_by and it works just fine as for our first experiments, we will test it further more.

[marcorosa]

Ordering discoveries by category still breaks the UI

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/i355397/git/credential-digger/ui/server.py", line 447, in get_discoveries
    'data': sorted([{'snippet': keys[0],
  File "/Users/i355397/git/credential-digger/ui/server.py", line 460, in <lambda>
    key=lambda i: States[i[order_by]].value,
  File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/enum.py", line 432, in __getitem__
    return cls._member_map_[name]
KeyError: 'password'