Use UTC datetimes everywhere

[rubik]

What changed

• All datetime math is now done in UTC. DateTimeFields are guaranteed to be already UTC since when USE_TZ = True Django will set PostgreSQL's timezone config option to UTC, as explained here: https://docs.djangoproject.com/en/1.9/ref/databases/#optimizing-postgresql-s-configuration;
• created_on and modified_on do not rely anymore on get_time, everything is done automatically with auto_now=True and auto_now_add=True. Those two options use internally timezone.now();
• a new migration has been generated to alter the aforementioned fields;
• ~~created_on_str is now saved in UTC. In this way the DB is not tied to a specific timezone. However, it's still not perfect, since the old data is still in MST and a migration could be necessary.~~ ~~created_on_str remained in MST.~~ Thanks to django-chartit2, created_on_str and date_str have been removed.

Migration

Since PostgreSQL stores all the datetimes as UTC, no DB migration is required. However, some Django fields have been altered, so a python manage.py migrate is necessary to get things working.

Notes

created_on and modified_on are no longer shown in the admin. This is due to the fact that those two attributes are now managed automatically by Django.

For reference, see #9.

[owocki]

:clap: thanks!

i want to

1. verify theres no custom datetime mutation logic that this PR missed
2. test locally

before merging.

[rubik]

Sure, let me know if anything goes wrong.

[rubik]

Maybe I can open a PR with the linting, get it merged and then rebase my branch on master so that only the relevant changes are left in this PR? That would make it easier to review.

[owocki]

I still see a little bit of timezone mutation on this branch

Searching 71 files for "hours=7"

/Users/kevinowocki/pytrader/history/views.py:
  555      data = {}
  556      for t in Trade.objects.filter(symbol=symbol,status='fill').order_by('-created_on').all():
  557:         date = datetime.datetime.strftime(t.created_on-datetime.timedelta(hours=7),'%Y-%m-%d')
  558          if not date in data.keys():
  559              data[date] = { 'buyvol' : [], 'sellvol' : [], 'buy' : [], 'sell': [], 'bal' : 0.00 }

/Users/kevinowocki/pytrader/history/management/commands/compare_perf.py:
   56                               directionally_same=directionally_same,
   57                               directionally_same_int=1 if directionally_same else 0,
   58:                              created_on_str=(tr_timerange_end - datetime.timedelta(hours=7)).strftime('%Y-%m-%d %H:%M'))
   59          pc.save()
   60  

2 matches across 2 files

[owocki]

While I agree that storage in UTC is the right way to go, I'd really like to be able to display data in my local timezone. This prevents me from having to do timezone math in my head everytime I look at some data. Any thoughts on the best way to do that?

[owocki]

@rubik merged your previous PR.

[owocki]

However, it's still not perfect, since the old data is still in MST and a migration could be necessary.

Is it worth writing a data migration for this? I know there's 5-10 others from slack that are running the repo, so I know it's not just me that has old data in the DB.

[rubik]

Is it worth writing a data migration for this? I know there's 5-10 others from slack that are running the repo, so I know it's not just me that has old data in the DB.

Fair enough, I don't think there's a good solution to this. I'll just revert the created_on_str changes. I also think that this is temporary. Either #35 comes up with a good solution (which maybe does not use django-chartit), or we'll look into some alternatives (perhaps for performance reasons). The problem is that django-chartit is not really maintained anymore, so the issue relevant to us (https://github.com/pgollakota/django-chartit/issues/2) is never going to be fixed.

[rubik]

I suggest you test locally, since now I'm not able to. If everything works, I think it's simpler to merge this one after #72.

[owocki]

this will need a rebase before merging.

[rubik]

There you go. It should be good now.

[rubik]

From my experiments, 'categories': 'created_on' works fine. The real bottleneck was 'terms': 'created_on' which didn't work in django-chartit. With django-chartit2 it's possible to specify a function to handle the data. I wasn't able to test the code with the real data because I only have the prices (I never traded).

This PR needs #74.

[owocki]

I'm good with merge here. @igorpejic @Snipa22 @t0mk ?

[owocki]

would like to get another to sign off on merge here... @Snipa22 @igorpejic @t0mk ?

[owocki]

@rubik could you describe how you tested this? does your personal instance of pytrader have data flowing through the admin?

[rubik]

@owocki I had price data in the admin and the chart was displaying correctly. Then I generated fake data in another Django project to test PivotChart and it was working correctly. If you could test it with trade data it would be even better. Locally I an only run predict_many_sk.py and it appears that c_chart works as intended.

What do you think about adding deposit_created_on? (See above.)