query_padmapper throwing error

[robstrange]

Are there any parameters I need to pass into the script I have set the Lat/Long for DC and get the following error.

Traceback (most recent call last): File "query_padmapper.py", line 84, in start(*sys.argv[1:]) TypeError: start() takes exactly 2 arguments (0 given)

Many thanks for doing this really great project.

[jeffkaufman]

I recently needed to rewrite that script heavily after padmapper updated their UI. The new way to call it is:

python query_padmapper.py padmapper-download-file.txt processed-apts.txt

The first argument is where to put the results of querying padmapper, the second file is a stripped down version of the first ready to be fed into draw_heatmap.py

[jeffkaufman]

Use chrome to follow the link it gives you, open developer tools, go to the networking tab, scroll the viewport until you get see an entry in the networking tab show up that says "pins", right click on that entry, choose "copy as curl", paste into query_padmapper.

(I'm sorry this is such a pain, I'd like to fully automate it at some point but I'm not sure when that would be.)

[robstrange]

Please don't worry, I am sorry to keep asking so many questions :)

I found the pins, it wasn't there when I first looked for some reason, hence the deleted question, I didn't want you to be bothered with it. It seems to be taking a while (~10 mins and counting) is that normal?

[jeffkaufman]

The download is taking 10+ minutes?

It's probably too much for the padmapper server. You could break your query up into multiple areas and then concatenate the output.

[robstrange]

Thanks for all your help, I don't seem to be able to get it to complete. It keeps hanging, only once have I got to "waiting...." and it hung there for ~15 mins before I killed it. I will keep trying, thank you again for sharing the code.

[lusanaturae]

My pin curl seems to be too long to paste. Any suggestions? Here's the curl I'm getting from the pins request:

curl 'https://www.padmapper.com/api/t/1/pins' -H 'Origin: https://www.padmapper.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8,fr;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36' -H 'Content-Type: application/json;charset=UTF-8' -H 'Accept: application/json, text/plain, /' -H 'Referer: https://www.padmapper.com/apartments/toronto-on/leslieville?box=-79.3756547907,43.6451828324,-79.3040719965,43.6910641976&property-categories=apartment' -H 'X-CSRFToken: Uujzn5u62W7HX3OFfuVoWY4GGFEYw3mT' -H 'Cookie: optimizelyEndUserId=oeu1487201727791r0.9229861962613399; optimizelySegments=%7B%227290380047%22%3A%22false%22%2C%227229180825%22%3A%22gc%22%2C%227290390041%22%3A%22direct%22%7D; csrftoken=Uujzn5u62W7HX3OFfuVoWY4GGFEYw3mT; _ga=GA1.2.227856134.1487201728; _gat=1; optimizelyBuckets=%7B%227961191885%22%3A%227956051421%22%7D; optimizelyPendingLogEvents=%5B%5D' -H 'Connection: keep-alive' -H 'X-Zumper-XZ-Token: 16ht3wwaf59.f82ellf7' --data-binary '{"limit":100,"maxLat":43.72558,"minLat":43.68164,"maxLng":-79.27734,"minLng":-79.36523,"propertyCategories":["apartment"]}' --compressed

Not sure if I need to paste all that in?

[jeffkaufman]

Yes, you need to paste the whole thing.

If that still gives you the error, can you paste the error here?

[lusanaturae]

Thanks for your help! Here's what I keep getting (note the curl gets cut off):

Corinnas-MacBook-Pro:Test corinnaprior$ python query_padmapper.py padmapper-download-file.txt processed-apts.txt Visit: https://www.padmapper.com/apartments/toronto-on/leslieville?box=-79.3533173539,43.655053042,-79.3450990655,43.6662609136&property-categories=apartment Inspect the networking, find a pins request, copy request as curl and paste here.

curl 'https://www.padmapper.com/api/t/1/pins' -H 'Origin: https://www.padmapper.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8,fr;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36' -H 'Content-Type: application/json;charset=UTF-8' -H 'Accept: application/json, text/plain, /' -H 'Referer: https://www.padmapper.com/apartments/toronto-on/leslieville?box=-79.3533173539,43.655053042,-79.3450990655,43.6662609136&property-categories=apartment' -H 'X-CSRFToken: Uujzn5u62W7HX3OFfuVoWY4GGFEYw3mT' -H 'Cookie: optimizelyEndUserId=oeu1487201727791r0.9229861962613399; _gat=1; optimizelySegments=%7B%227290380047%22%3A%22false%22%2C%227229180825%22%3A%22gc%22%2C%227290390041%22%3A%22direct%22%7D; _ga=GA1.2.227856134.1487201728; csrftoken=Uujzn5u62W7HX3OFfuVoWY4GGFEYw3mT; optimizelyBuckets=%7B%227961191885%22%3A%227956051421%22%7D; optimizelyPendingLogEvents=%5B%5D' -H 'Connection: keep-alive' -H
Traceback (most recent call last): File "query_padmapper.py", line 82, in start(*sys.argv[1:]) File "query_padmapper.py", line 70, in start download(fname_download) File "query_padmapper.py", line 40, in download raise Exception("Something looks wrong. Was that the curl version of a pins request?") Exception: Something looks wrong. Was that the curl version of a pins request? Corinnas-MacBook-Pro:Test corinnaprior$

[lusanaturae]

I'll add too, that if I cut out bits of that curl (particularly the Accept-Encoding, Accept-Language, User-Agent, and Content-Type), I can get the thing to run, but then it hangs at "waiting..." forever.

[jeffkaufman]

When it's on "waiting" it's expecting you to go run the curl command it printed out. After you've run that command in another window, press enter in the window that says "waiting".

Sorry!

[anthonybob]

Hello, I'm having a similar issue with my curl command, except it's not even executing when I paste it into the python input prompt.