edx-analytics-pipeline
edx-analytics-pipeline copied to clipboard
Python 2/3 compatibility for finance tasks
This does not change any infrastructure to actually run anything under python 3, it only represents the results of my testing of the finance unit tests and acceptance tests under python 3 and modernizing the code to become both python 2 and 3 compatible.
Maybe half of the changes were caused by python-modernize (and manually audited), the other half are manual changes.
Testing requirements
The following four suite of tests are needed to pass before merging this PR:
suite | status |
---|---|
Python 2 unit tests | Passing. |
Python 2 acceptance tests | Failed: build #1795 |
Python 3 unit tests (financial tasks only) | Passing. |
Python 3 acceptence tests (test_financial_reports only) | In Progress |
Analytics Pipeline Pull Request
Make sure that the following steps are done before merging:
- [ ] If you have a migration please contact data engineering team before merging.
- [ ] Before merging run full acceptance tests suite and provide URL for the acceptance tests run.
- [ ] A member of data engineering team has approved the pull request.
Here's the current python 3 luigi execution summary:
20:40:00 ===== Luigi Execution Summary =====
20:40:00
20:40:00 Scheduled 65 tasks of which:
20:40:00 * 3 present dependencies were encountered:
20:40:00 - 2 ExternalURL(url=s3+https://edx-analytics-test-data/credentials/acceptance-db.json,s3+https://edx-analytics-test-data/credentials/acceptance-vertica.json)
20:40:00 - 1 PaymentTask(...)
20:40:00 * 51 ran successfully:
20:40:00 - 1 FullOttoOrderTableTask(...)
20:40:00 - 1 FullShoppingcartOrderTableTask(...)
20:40:00 - 1 ImportAuthUserTask(...)
20:40:00 - 1 ImportCouponVoucherIndirectionState(...)
20:40:00 - 1 ImportCouponVoucherState(...)
20:40:00 ...
20:40:00 * 3 failed:
20:40:00 - 1 LoadInternalReportingFullShoppingcartOrdersToWarehouse(...)
20:40:00 - 1 OrderTableTask(...)
20:40:00 - 1 ReconcileFullOrdersAndTransactionsTask(...)
20:40:00 * 8 were left pending, among these:
20:40:00 * 6 had failed dependencies:
20:40:00 - 1 BuildEdServicesReportTask(...)
20:40:00 - 1 BuildFinancialReportsTask(...)
20:40:00 - 1 LoadInternalReportingEdServicesReportToWarehouse(...)
20:40:00 - 1 LoadInternalReportingFullOrderTransactionsToWarehouse(...)
20:40:00 - 1 ReconcileOrdersAndTransactionsTask(...)
20:40:00 ...
20:40:00 * 2 was not granted run permission by the scheduler:
20:40:00 - 1 LoadInternalReportingOrderTransactionsToWarehouse(...)
20:40:00 - 1 TransactionReportTask(...)