lightning: Adding Base 64 encoding support to lightning

[ankita25]

What problem does this PR solve?

Issue Number: close #44893

Problem Summary:

What is changed and how it works?

In lightning.toml, a config param can be added as follows [mydumper.csv] base64-encoded = true

This means all columns in csv files are base64 encoded. So lightning will decode it before storing it in the database.

Check List

Tests

• [ ] Unit test
• [ ] Integration test
• [x] Manual test (add detailed scripts or steps below)
• [ ] No code

Side effects

• [ ] Performance regression: Consumes more CPU
• [ ] Performance regression: Consumes more Memory
• [ ] Breaking backward compatibility

Documentation

• [ ] Affects user behaviors
• [ ] Contains syntax changes
• [ ] Contains variable changes
• [ ] Contains experimental features
• [ ] Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign lance6716 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot[bot]]

Welcome @ankita25!

It looks like this is your first PR to pingcap/tidb 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/tidb. :smiley:

[ti-chi-bot[bot]]

Hi @ankita25. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tiprow[bot]]

Hi @ankita25. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hawkingrei]

/ok-to-test

[hawkingrei]

/test tiprow_fast_test

[ti-chi-bot[bot]]

@hawkingrei: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test build
• /test canary-scan-security
• /test check-dev
• /test check-dev2
• /test mysql-test
• /test pingcap/tidb/canary_ghpr_unit_test
• /test pull-integration-br-test
• /test pull-integration-mysql-test
• /test unit-test

Use /test all to run the following jobs that were automatically triggered:

• pingcap/tidb/ghpr_build
• pingcap/tidb/ghpr_check
• pingcap/tidb/ghpr_check2
• pingcap/tidb/ghpr_mysql_test
• pingcap/tidb/ghpr_unit_test

In response to this:

/test tiprow_fast_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ankita25]

/retest

[codecov[bot]]

Codecov Report

:x: Patch coverage is 25.00000% with 9 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 78.8057%. Comparing base (5369209) to head (ae65131). :warning: Report is 6186 commits behind head on master.

:x: Your patch check has failed because the patch coverage (12.5000%) is below the target coverage (85.0000%). You can increase the patch coverage or adjust the target coverage. :x: Your project check has failed because the head coverage (73.5204%) is below the target coverage (85.0000%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #44892        +/-   ##
================================================
+ Coverage   73.2388%   78.8057%   +5.5669%     
================================================
  Files          1240       3692      +2452     
  Lines        386007    1072849    +686842     
================================================
+ Hits         282707     845467    +562760     
- Misses        85169     186062    +100893     
- Partials      18131      41320     +23189     

Flag	Coverage Δ
integration	30.5209% <12.5000%> (?)
unit	73.5204% <25.0000%> (+0.2816%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	78.4071% <ø> (+23.7229%)	:arrow_up:
parser	∅ <ø> (∅)
br	74.1979% <25.0000%> (-0.5468%)	:arrow_down:

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[tiprow[bot]]

@ankita25: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tiprow_fast_test	ae651312dc25a17070dea711693333dd98ab457b	link	true	/test fast_test_tiprow

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[ti-chi-bot[bot]]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[D3Hunter]

@ankita25 hi, do you have time to push this PR forward to be ready to be merged?

if no, i would like to file another PR with the same idea, but we would like to make the field named by fields_encoded_by='base64' to make it more flexible, and implements it for IMPORT INTO too, and add more test.

[D3Hunter]

new pr https://github.com/pingcap/tidb/pull/59396

[ti-chi-bot[bot]]

@ankita25: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
idc-jenkins-ci-tidb/unit-test	ae651312dc25a17070dea711693333dd98ab457b	link	true	/test unit-test
pull-integration-realcluster-test-next-gen	ae651312dc25a17070dea711693333dd98ab457b	link	true	/test pull-integration-realcluster-test-next-gen
pull-unit-test-next-gen	ae651312dc25a17070dea711693333dd98ab457b	link	true	/test pull-unit-test-next-gen

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.