Setting up HPI - config file

[AtomicNess123]

Hi, struggling to install orger for which I need HPI :)

I read:

After installing HPI, run hpi config create.

This will create an empty config file for you (usually, in ~/.config/my), which you can edit. Example configuration:

import pytz # yes, you can use any Python stuff in the config

class emfit:
    export_path = '/data/exports/emfit'
    tz = pytz.timezone('Europe/London')
    excluded_sids = []
    cache_path  = '/tmp/emfit.cache'

class instapaper:
    export_path = '/data/exports/instapaper'

class roamresearch:
    export_path = '/data/exports/roamresearch'
    username    = 'karlicoss'

(1) I have no my/config file under ~/.config (2) Is the config file /Users/user/HPI/my/config.py? (3) What exactly should I add in here? I don't see where to include my personal user data to access hypothesis.

Thanks so much!

[karlicoss]

Right, have you run hpi config create? It will print out a path like /path_to_home_dir/.config/my/my/config/__init__.py -- this will be your personal config which you can edit.

In the config, you'll need to add the corresponding snippets, e.g. for hypothesis (https://github.com/karlicoss/HPI/blob/master/doc/MODULES.org#myhypothesis), or instapaper (https://github.com/karlicoss/HPI/blob/master/doc/MODULES.org#myinstapaper)

[karlicoss]

If you want to setup hypothesis, the first thing you'd need to do is to setup hypexport and try to export the data, let me know if the instructions are clear enough!

[AtomicNess123]

Yes, I did ran hpi config create (as explained in my ORGER issue).

I also tried hypexport (without secrets.py, just running python3 -m hypexport.export --username <username> --token <token> with my data and I got my annotations (hurray for that!)

Let's continue: I copypasted this into the config file:

class hypothesis:
    '''
    Uses [[https://github.com/karlicoss/hypexport][hypexport]] outputs
    '''

    # paths[s]/glob to the exported JSON data
    export_path: Paths

So the file /Users/gahis/HPI/my/config.py should not be touched? I got confused thinking this was the config file. It contains for instance:

class hypothesis:
    # expects outputs from https://github.com/karlicoss/hypexport
    # (it's just the standard Hypothes.is export format)
    export_path: Paths = '/path/to/hypothesis/data'

so I imagined I had to paste my path there...

[karlicoss]

Well done with hypexport!

So the file /Users/gahis/HPI/my/config.py should not be touched

Ah! Is it the same file as this? https://github.com/karlicoss/HPI/blob/master/my/config.py Then no, it's basically there just for documentation & tests. I should probably make it more clear in the header, good catch!

Yeah, you should only modify the (currently empty) config created by hpi config create, just paste there something like:

class hypothesis
    export_path = '/path/to/exported/data'

(you don't need the :Paths thing like in the examples, it's optional)

Then you can run hpi doctor my.hypothesis to check if it works!

[AtomicNess123]

hpi doctor my.hypothesis

Ok, did all mantioned above (big thanks), and ran doctor getting the following

✅ config file: /Users/hagis/Library/Application Support/my/my/config/__init__.py
❗ mypy not found, so can't check config with it. See https://github.com/python/mypy#readme if you want to install it and retry
❌ FAIL: my.hypothesis                                      loading failed
   Traceback (most recent call last):
     File "/Users/hagis/HPI/my/core/__main__.py", line 204, in modules_check
       mod = importlib.import_module(m)
     File "/Users/hagis/opt/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
       return _bootstrap._gcd_import(name[level:], package, level)
     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
     File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
     File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
     File "<frozen importlib._bootstrap_external>", line 728, in exec_module
     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
     File "/Users/hagis/HPI/my/hypothesis.py", line 32, in <module>
       config = make_config(hypothesis)
     File "/Users/hagis/HPI/my/core/cfg.py", line 19, in make_config
       return cls(**params) # type: ignore[call-arg]
   TypeError: __init__() missing 1 required positional argument: 'export_path'

Too complex for me to decode :/

[karlicoss]

Not problem, it's a bit cryptic indeed :) Basically means it can't construct the config because export_path is missing, which shouldn't be the case if you added it 🤔

Can you paste the /Users/hagis/Library/Application Support/my/my/config/__init__.py here so I could check?

[AtomicNess123]

Wait! What a stupid error! Sorry to have sent the previous message. I had "export_path:" and noticed in your last message you did not have : but = (export_path =). After running doctor now I get:

✅ import order: ['/Users/hagis/HPI/my']
✅ config file: /Users/hagis/Library/Application Support/my/my/config/__init__.py
❗ mypy not found, so can't check config with it. See https://github.com/python/mypy#readme if you want to install it and retry
✅ OK  : my.hypothesis                                     
  /Users/hagis/HPI/my/core/common.py:182: UserWarning: /Users/hagis/HPI/my/hypothesis.py: no paths were matched against []. This might result in missing data.
    warnings.warn(f'{caller()}: no paths were matched against {paths}. This might result in missing data.')
  File "/Users/hagis/.local/bin/hpi", line 33, in <module>
    sys.exit(load_entry_point('HPI', 'console_scripts', 'hpi')())
  File "/Users/hagis/HPI/my/core/__main__.py", line 310, in main
    func(args)
  File "/Users/hagis/HPI/my/core/__main__.py", line 263, in doctor
    modules_check(args)
  File "/Users/hagis/HPI/my/core/__main__.py", line 220, in modules_check
    res = stats()
  File "/Users/hagis/HPI/my/hypothesis.py", line 74, in stats
    **stat(highlights),
  File "/Users/hagis/HPI/my/core/common.py", line 386, in stat
    fr = func()
  File "/Users/hagis/HPI/my/hypothesis.py", line 59, in highlights
    return sort_res_by(_dal().highlights(), key=key)
  File "/Users/hagis/HPI/my/hypothesis.py", line 52, in _dal
    sources = get_files(config.export_path)
  File "/Users/hagis/HPI/my/core/common.py", line 183, in get_files
    traceback.print_stack()
❗      - stats:                      computing failed
   Traceback (most recent call last):
     File "/Users/hagis/HPI/my/core/__main__.py", line 220, in modules_check
       res = stats()
     File "/Users/hagis/HPI/my/hypothesis.py", line 74, in stats
       **stat(highlights),
     File "/Users/hagis/HPI/my/core/common.py", line 386, in stat
       fr = func()
     File "/Users/hagis/HPI/my/hypothesis.py", line 59, in highlights
       return sort_res_by(_dal().highlights(), key=key)
     File "/Users/hagis/HPI/my/core/error.py", line 60, in sort_res_by
       for i in items:
     File "/Users/hagis/.local/lib/python3.7/site-packages/hypexport/dal.py", line 63, in highlights
       for i in self._iter_raw():
     File "/Users/hagis/.local/lib/python3.7/site-packages/hypexport/dal.py", line 53, in _iter_raw
       last = max(self.sources)
   ValueError: max() arg is an empty sequence

So the import is fine although there are millions of warnings.

[karlicoss]

I had "export_path:" and noticed in your last message you did not have : but = (export_path =

Huh, this is actually something mypy would have caught (note the line mypy not found, so can't check config with it).

Anyway, this is better! It's just long because it prints the whole stacktrace; but when you are a programmer, you appreciate it, because it helps you understand what's happening faster!.

What it means here is that it tried to max(self.sources) (max just means 'latest export' here), and sources happened to be empty, so it couldn't get the last either. Probably means your export directory is empty. (I guess also another place where it's worth to add a more humane warning :) ).

What you need to do is to run something like:

python3 -m hypexport.export --username username --token token > /path/to/export_dir/hypothesis.json

(the > thing redirects the output in the file, otherwise it's only printed in the console and not saved on the disk)

After that, try running hpi doctor my.hypothesis again!

[AtomicNess123]

There we go, sir! Yes, that mypy package comes handy indeed. This is fantastic. Here the output from hpi doctor:

✅ import order: ['/Users/hagis/HPI/my'] ✅ config file: /Users/hagis/Library/Application Support/my/my/config/init.py ❗ mypy not found, so can't check config with it. See https://github.com/python/mypy#readme if you want to install it and retry ✅ OK : my.hypothesis
✅ - stats: {'highlights': {'count': 5080, 'last': datetime.datetime(2020, 11, 23, 3, 14, 58, 530750, tzinfo=datetime.timezone.utc)}, 'pages': {'count': 2801}}

[AtomicNess123]

I do now: python hypothesis.py And it works! It creates an ORG file with my hypothesis annotations! Thank you a million!

[karlicoss]

Excellent, glad I could help, especially to a non-programmer :)

The last thing you might want to do is to make sure the export command (python3 -m hypexport.export --username username --token token > /path/to/export_dir/hypothesis.json) runs periodically, so you get new annotations in. Usually the easiest is to use cron for this, not sure if you used it before, but if you haven't, here's a guide: https://www.whoishostingthis.com/resources/cron/#examples

Alternatively, you can just run the export command manually if you don't need to sync that often.

[AtomicNess123]

Yes, that was going to be my next question! I already have a crontab set up so this will be easy I think 👍

[AtomicNess123]

One last (hopefully) question: you said to run python3 -m hypexport.export --username username --token token > /path/to/export_dir/hypothesis.json to keep the annotations up-to-date in the disk. How about running python hypothesis.py instead as this would do the same but in ORG format? or maybe I am missing something?

[AtomicNess123]

Ok, I get it. First I need to run the hypexport and then have another line to run python hypothesis.py. Confirm it this is correct, thanks!

[karlicoss]

Ok, I get it. First I need to run the hypexport and then have another line to run python hypothesis.py

Yep! Just space it apart in time, e.g. run hypexport at 10:00, and orger at 11:00, and it should be OK.

[AtomicNess123]

Ok, I get it. First I need to run the hypexport and then have another line to run python hypothesis.py

Yep! Just space it apart in time, e.g. run hypexport at 10:00, and orger at 11:00, and it should be OK.

Thanks, I have now give some time between them two (I had them both at the same specified time—export first, orger second). I would have thought that cron runs them sequentially (?).

[karlicoss]

Ah yes, possibly sequential by default. I guess it's more of my general habit -- with some schedulers it's parallel, or sometimes the process forks off and runs in the background, etc.

[AtomicNess123]

Understood, thanks!

[redthing1]

I had "export_path:" and noticed in your last message you did not have : but = (export_path =

Huh, this is actually something mypy would have caught (note the line mypy not found, so can't check config with it).

Could you explain this? When following the instructions here I got an error in my config using: export_path: Paths and had to change it to export_path = '/path/to/thing'.

[karlicoss]

When following the instructions here

Yeah, perhaps I need to add a short explanation; I guess I kind of assumed people would be familiar with python and figure out that you need to replace the type (: Path) with the actual value ( = '/path/to/export_dir')

Could you explain this?

What I mean was: if you had something like this in the config (note the wrong : instead of =):

class reddit:
     export_path: '/path/to/export_dir'

, and then ran the doctor (and also had mypy installed), it would have complained, because a string is used 'as a type'.

[redthing1]

Oh, that was a type specifier!! I was up late last night and didn't think about that. Thanks for clarifying, it makes sense to me now!

[AtomicNess123]

I was wondering why my config file is in config file: /Users/hagis/Library/Application Support/my/my/config/__init__.py and not in ~/.config. I think we never tackled this issue!

[seanbreckenridge]

HPI uses appdirs to figure out where to put your config. See preinit for the code that figures that out

You can set the MY_CONFIG environment variable to override that, so in your .bashrc or .zshrc set:

export MY_CONFIG="${HOME}/.config/my"