Partial incrementalization of AnswerDistribution via Hive

[tobz]

This is a redux of Brian's work, and then Alex's work, on working towards making AnswerDistribution incremental, by partially achieving incremental-ness by generating Hive partitions for each day's data.

[codecov-io]

Codecov Report

Merging #433 into master will decrease coverage by 0.31%. The diff coverage is 31.9%.

@@            Coverage Diff             @@
##           master     #433      +/-   ##
==========================================
- Coverage   78.55%   78.23%   -0.32%     
==========================================
  Files         188      202      +14     
  Lines       20117    22825    +2708     
==========================================
+ Hits        15803    17858    +2055     
- Misses       4314     4967     +653

Impacted Files	Coverage Δ
edx/analytics/tasks/insights/answer_hive.py	30.32% <30.32%> (ø)
edx/analytics/tasks/util/datetime_util.py	70.37% <33.33%> (-4.63%)	:arrow_down:
...alytics/tasks/tests/acceptance/test_answer_hive.py	44.73% <44.73%> (ø)
...sts/acceptance/test_lms_courseware_link_clicked.py	33.33% <0%> (-8.98%)	:arrow_down:
...cs/tasks/tests/acceptance/test_problem_response.py	30.37% <0%> (-4.63%)	:arrow_down:
...ics/tasks/tests/acceptance/test_database_import.py	57.5% <0%> (-4.27%)	:arrow_down:
edx/analytics/tasks/common/vertica_load.py	62.75% <0%> (-3.66%)	:arrow_down:
...ceptance/test_internal_reporting_course_catalog.py	26.15% <0%> (-3.4%)	:arrow_down:
edx/analytics/tasks/insights/problem_response.py	85.27% <0%> (-3.32%)	:arrow_down:
...ytics/tasks/tests/acceptance/test_course_blocks.py	32% <0%> (-2.89%)	:arrow_down:
... and 37 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e655919...cc2edcf. Read the comment docs.

[tobz]

Silly bot! Also.. hi! 👋

[azafty468]

Ha, label changes result in re-triggering builds. Also hi! 👋

[mduboseedx]

@edx/analytics Is this PR still worth pursuing?

[brianhw]

This should not be an OSPR, and I'm not quite sure how it turned into one. This was a background project, and Toby was the last person working on this when he was here. It is on the back burner again while we work on higher priority stories. It probably has some context in its reviews that might be useful, but those would still persist even if I closed it.

But in general, I would be curious to know what we should do in future to prevent a long-running PR created by someone that then leaves the organization from becoming an OSPR.

On Fri, Apr 20, 2018 at 11:39 AM, mduboseedx [email protected] wrote:

@edx/analytics https://github.com/orgs/edx/teams/analytics Is this PR still worth pursuing?

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub https://github.com/edx/edx-analytics-pipeline/pull/433#issuecomment-383136400, or mute the thread https://github.com/notifications/unsubscribe-auth/ACaqz4Kkamav1rF5xsgXhtV7NW067FuNks5tqgFEgaJpZM4PshOB .

[mduboseedx]

@brianhw We have OPEN-2111 to have work done on the bot to prevent former employee PRs from triggering an OSPR. I've deleted the OSPR and will leave his PR open.