shell-history icon indicating copy to clipboard operation
shell-history copied to clipboard

Use dichotomy instead of "one by one" when db commit fails

Open pawamoy opened this issue 7 years ago • 1 comments

Write a recursive function using dichotomy to insert objects in the database.

  • If inserting all objects fails, split the list in two, recurse on both parts.
  • Stop condition: when there is only one object in the list.
  • Sum up results: return size of (successfully) inserted list.

This will allow faster database updates from big history files.

Boost priority

  • Boost priority in our backlog through Polar.sh. Higher pledge, higher priority.
  • Minimum pledge by user/organization is $5, minimum amount for boost is $30.
  • View all issues with pledges.
  • We receive the funds once the issue is completed and confirmed by you.
  • Features with the insiders label are released to sponsors first, and tied to a funding goal.
Fund with Polar

pawamoy avatar Dec 07 '17 13:12 pawamoy

We could also improve import exec time by looking at UUID and start fields:

for session in data.sessions:
  if session not in db:
    insert(session.commands)
  else:
    session_last_command = get_session_last_command(db, session)
    # with filter, might be slow, esp. with lambda
    insert(filter(session.commands, lambda c: c.start > session_last_command.start))
    # with sort, might be much faster
    session_commands = sorted(session.commands)
    pivot = None
    for i, command in enumerate(session_commands):  # FIXME: use dichotomy instead of naive iteration
      if command.start > session_last_command.start:
        pivot = i
        break
      if pivot is not None:
        insert(session_commands[pivot:])

Finally we could end up with a fast session_last_command import function, and a slower but "100% sure not to miss any command" dichotomy import function.

pawamoy avatar Dec 07 '17 14:12 pawamoy