edx-analytics-pipeline icon indicating copy to clipboard operation
edx-analytics-pipeline copied to clipboard

Python 2/3 compatibility for finance tasks

Open pwnage101 opened this issue 5 years ago • 1 comments

This does not change any infrastructure to actually run anything under python 3, it only represents the results of my testing of the finance unit tests and acceptance tests under python 3 and modernizing the code to become both python 2 and 3 compatible.

Maybe half of the changes were caused by python-modernize (and manually audited), the other half are manual changes.


Testing requirements

The following four suite of tests are needed to pass before merging this PR:

suite status
Python 2 unit tests Passing.
Python 2 acceptance tests Failed: build #1795
Python 3 unit tests (financial tasks only) Passing.
Python 3 acceptence tests (test_financial_reports only) In Progress

Analytics Pipeline Pull Request

Make sure that the following steps are done before merging:

  • [ ] If you have a migration please contact data engineering team before merging.
  • [ ] Before merging run full acceptance tests suite and provide URL for the acceptance tests run.
  • [ ] A member of data engineering team has approved the pull request.

pwnage101 avatar Jul 17 '19 19:07 pwnage101

Here's the current python 3 luigi execution summary:

20:40:00 ===== Luigi Execution Summary =====
20:40:00 
20:40:00 Scheduled 65 tasks of which:
20:40:00 * 3 present dependencies were encountered:
20:40:00     - 2 ExternalURL(url=s3+https://edx-analytics-test-data/credentials/acceptance-db.json,s3+https://edx-analytics-test-data/credentials/acceptance-vertica.json)
20:40:00     - 1 PaymentTask(...)
20:40:00 * 51 ran successfully:
20:40:00     - 1 FullOttoOrderTableTask(...)
20:40:00     - 1 FullShoppingcartOrderTableTask(...)
20:40:00     - 1 ImportAuthUserTask(...)
20:40:00     - 1 ImportCouponVoucherIndirectionState(...)
20:40:00     - 1 ImportCouponVoucherState(...)
20:40:00     ...
20:40:00 * 3 failed:
20:40:00     - 1 LoadInternalReportingFullShoppingcartOrdersToWarehouse(...)
20:40:00     - 1 OrderTableTask(...)
20:40:00     - 1 ReconcileFullOrdersAndTransactionsTask(...)
20:40:00 * 8 were left pending, among these:
20:40:00     * 6 had failed dependencies:
20:40:00         - 1 BuildEdServicesReportTask(...)
20:40:00         - 1 BuildFinancialReportsTask(...)
20:40:00         - 1 LoadInternalReportingEdServicesReportToWarehouse(...)
20:40:00         - 1 LoadInternalReportingFullOrderTransactionsToWarehouse(...)
20:40:00         - 1 ReconcileOrdersAndTransactionsTask(...)
20:40:00         ...
20:40:00     * 2 was not granted run permission by the scheduler:
20:40:00         - 1 LoadInternalReportingOrderTransactionsToWarehouse(...)
20:40:00         - 1 TransactionReportTask(...)

pwnage101 avatar Jul 17 '19 20:07 pwnage101