Total Earnings Calculation (AccountStats Plugin)

[praeluceo]

The "BTC in total" line of the total earnings appears to only go back 221 days from current. The code appears to be in www/lendingbot.js but I can't find a way to configure it, it seems to default to 5,000 days?

https://github.com/BitBotFactory/poloniexlendingbot/pull/349/files#diff-b1937da93efaa6e81673d3420a9a0b43R71

It doesn't seem to be reflective of that, I can compare the output of the Lifetime Earnings to my own spreadsheet, and it matches almost perfectly going back to 10/13/2016, but nothing before that. Is that as far back as the Poloniex API goes? I can manually pull older than that, but I'm not sure if the API allows it.

[rnevet]

Sadly Poloniex doesn't actually return all history in one call, that's why the bot loops back trying to fetch all. https://github.com/BitBotFactory/poloniexlendingbot/pull/349/files#diff-b1937da93efaa6e81673d3420a9a0b43R58

Can you delete the history sqlite db created under market_data, restart the bot and follow the log... you should be able to see which dates where fetched. I suspect the loop is somehow missing loans or Polo isn't returning them.

[praeluceo]

I'll try removing the history db when I get home tonight. Another insight into this that I'm seeing, is that the bot runs its "total earnings" report whenever it's launched, and again each 24 hours it seems like. However Poloniex "closes" their day at midnight GMT. So my total earnings report line doesn't actually represent the earnings, and it should probably say "Previous Day's Earnings" since I don't think Poloniex will actually give you a "last 24 hours" number. You could do "Yesterday through today" but that's not as useful, I'd much prefer just to see what I generated yesterday, vs. whatever the revenue has been so far today, based on whenever I started the bot relative to 12am GMT.

[rnevet]

"is that the bot runs its "total earnings" report whenever it's launched, and again each 24 hours it seems like. " - that is correct. I don't follow the rest of your logic though, but basically you would like to see Yesterdays earnings GMT?

[praeluceo]

Correct, so for example, this morning my bot listed:

Earnings:
----------
0.00037296 BTC in last 24hrs

But that's not correct, that was only the earnings between midnight GMT and when the bot started, a few hours later. More accurately would be to list that as "BTC since yesterday" as it isn't actually going to cover a 24 hour period. However, that information isn't super relevant as a daily report, it'd be more useful in an hourly graph (so you could see daily earnings by hour).

More useful though, would be a true "Previous Day's Earnings" entry, as that can be compared over time more easily, and is an absolute and unchanging value.

[praeluceo]

The message I received after moving the db out of the way is here:

$ mv loan_history.sqlite3 loan_history.sqlite3.old
$ python lendingbot.py
Welcome to Poloniex Lending Bot
Started WebServer, lendingbot status available at http://127.0.1.1:8000/lendingbot.html, http://127.0.0.1:8000/lendingbot.html
2017-06-07 18:18:36 Error: Connection timed out. Please try again. Requesting returnLendingHistory
Timed out, will retry in 30.0sec

And tailing www/botlog.json just shows what the web interface outputs:

2017-06-07 18:23:33 Error: The read operation timed out Requesting returnLendingHistory 2017-06-07 18:22:52 Downloaded 0 loans history 2017-06-08 01:08:47 to 2017-06-08 01:22:51

[rnevet]

"More accurately would be to list that as "BTC since yesterday" as it isn't actually going to cover a 24 hour period." - no, the bot will always cover a 24hr period relative to the time you started it. I can change it to yesterday (GMT). "More useful though, would be a true "Previous Day's Earnings" entry" - I agree.

As for the log, when the bot fails to get the lending history it tries to recover it in later cycles, is the total earning corrected in the "morning report"?

[praeluceo]

It's been about 8 hours since I restarted the bot, the 24 number is now correct, as you said it would be. However the lifetime earnings value is still under-reported and my log output shows:

2017-06-08 04:18:12 Error: Connection timed out. Please try again. Requesting returnBalances Timed out, will retry in 30.0sec Error in MarketAnalysis: list index out of range Error in MarketAnalysis: list index out of range Error in MarketAnalysis: list index out of range Error in MarketAnalysis: list index out of range Error in MarketAnalysis: list index out of range Error in MarketAnalysis: list index out of range

I just restarted the bot again in an attempt to get the fetch to manually restart (although from appearances, that isn't necessary).

[rnevet]

Error in MarketAnalysis: list index out of range This might mean that the CSV files are corrupt.

[utdrmac]

Can the bot format numbers correctly and not in scientific notation?

2017-06-08 08:55:14 Earnings: ---------- 9.56e-06 BTC in last 24hrs 0.39269841 BTC in total

[utdrmac]

Can the bot please be less chatty? I have PAGES of "downloaded 0". Very unnecessary .

[rnevet]

@utdrmac please update to latest version, it only tries fetching history every 24h. scientific notation was also fixed already AFAIK.

Reduce spam https://github.com/BitBotFactory/poloniexlendingbot/commit/ac7318adbd4afed5e3b73e5e0ade562226caad58

Fix output https://github.com/BitBotFactory/poloniexlendingbot/commit/b9ef3b9257384858044feede99bc141c76cd7383

[kevinlekiller]

Here's what mine says : 2017-06-11 13:26:09 Earnings: | ---------- | 0.00008678 BTC in last 24hrs | 0.00016549 BTC in total |

But if I get the total from the csv file it's 0.00146602 BTC : TOTALINTEREST=0; CURRENCY=BTC; while read line; do if [[ $line =~ ^$CURRENCY ]]; then EARNED=$(echo "$line" | cut -d, -f7) TOTALINTEREST=$(echo $TOTALINTEREST + $EARNED | bc); fi; done < lendingHistory.csv; if [[ $TOTALINTEREST =~ ^\. ]]; then echo -n 0; fi; echo $TOTALINTEREST $CURRENCY

[rnevet]

@kevinlekiller, you can delete the sqlite3 file to restart the AccountStats dd (& restart the bot). If that doesn't help try to dig into the sqlite3 and try to figure out why is there such a difference...

btw "EARNED=$(echo "$line" | cut -d, -f7)" which column is that?

[kevinlekiller]

Thanks @rnevet, deleting the DB and restarting worked.

The column is called "Earned" in the csv file, it is the interest minus the 15% fee (or if the question was whether it should be the 6th index, cut's field option starts at 1).

[rnevet]

Happy to hear that, the Poloniex API is problematic and it was very hard to detect if all history is retrieved.

[rnevet]

@kevinlekiller I have a PR for some improvements for the AccountStats plugin if you wish to test it: https://github.com/BitBotFactory/poloniexlendingbot/pull/378

[kevinlekiller]

@rnevet Replied on the PR.