TEZ-4666: Migrate tez-tools python scripts to python3

[Aggarwal-Raghav]

BEFORE FIX:

AFTER FIX:

[abstractdog]

this is awesome @Aggarwal-Raghav , thanks! +1

[Aggarwal-Raghav]

The goal was to keep the changes at minimum and convert to python3. Otherwise there is scope of improvement in scripts by using argparser instead of sys.argv, pathlib instead of os.path.

[abstractdog]

The goal was to keep the changes at minimum and convert to python3. Otherwise there is scope of improvement in scripts by using argparser instead of sys.argv, pathlib instead of os.path.

yeah, small refactors are always welcome, absolutely fit this PR

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	31m 27s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	shelldocs	0m 0s		Shelldocs was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
			_ master Compile Tests _
-1 :x:	pylint	0m 1s	/branch-pylint-stderr.txt	Error running pylint. Please check pylint stderr files.
			_ Patch Compile Tests _
+1 :green_heart:	codespell	0m 4s		No new issues.
-1 :x:	blanks	0m 0s	/blanks-eol.txt	The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-1 :x:	blanks	0m 0s	/blanks-tabs.txt	The patch 84 line(s) with tabs.
+1 :green_heart:	markdownlint	0m 2s		The patch generated 0 new + 45 unchanged - 1 fixed = 45 total (was 46)
-1 :x:	pylint	0m 2s	/patch-pylint-stderr.txt	Error running pylint. Please check pylint stderr files.
+1 :green_heart:	pylint	0m 2s		No new issues.
+1 :green_heart:	shellcheck	0m 0s		No new issues.
			_ Other Tests _
-1 :x:	asflicense	0m 51s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		33m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-444/1/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/444
Optional Tests	dupname asflicense codespell detsecrets pylint markdownlint shellcheck shelldocs
uname	Linux d201e749b075 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-444/src/.yetus/personality.sh
git revision	master / 92119cb1fd9b4225a5a610ced1d7707bee006cee
Max. process+thread count	52 (vs. ulimit of 5500)
modules	C: tez-tools U: tez-tools
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-444/1/console
versions	git=2.43.0 maven=3.8.7 codespell=2.0.0 markdownlint=0.23.2 shellcheck=0.7.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

[maheshrajus]

LGTM @Aggarwal-Raghav can you please check the above failures in yetus report. thanks !

[Aggarwal-Raghav]

LGTM @Aggarwal-Raghav can you please check the above failures in yetus report. thanks !

thanks for the review @maheshrajus , sure, I'll fix the failures.

[Aggarwal-Raghav]

@maheshrajus @abstractdog , it seems that the pylint is not working in yetus based on /branch-pylint-stderr.txt , As i ran the pylint in local, there are lot of issues unrelated to the change. I can surely check/fix them along with shellcheck, blanks, asfheader etc.

Also, 1 major flaw in python scripts are, tabs are not converted to spaces and has un-even indentation.

Python's official style guide, PEP 8, strongly recommends using spaces for indentation, specifically four consecutive spaces per indentation level. While Python technically allows both tabs and spaces for indentation, and even disallows mixing them within the same file, spaces are the widely preferred and recommended convention

Scripts are old and requires some decent changes. Personally, I'm ready and committed to fix them. Will also accomodate that pathlib and argparser changes as well. I'll use both https://github.com/astral-sh/ruff and pylint to ensure scripts are of good quality.

[abstractdog]

@maheshrajus @abstractdog , it seems that the pylint is not working in yetus based on /branch-pylint-stderr.txt , As i ran the pylint in local, there are lot of issues unrelated to the change. I can surely check/fix them along with shellcheck, blanks, asfheader etc.

Also, 1 major flaw in python scripts are, tabs are not converted to spaces and has un-even indentation.

Python's official style guide, PEP 8, strongly recommends using spaces for indentation, specifically four consecutive spaces per indentation level. While Python technically allows both tabs and spaces for indentation, and even disallows mixing them within the same file, spaces are the widely preferred and recommended convention

Scripts are old and requires some decent changes. Personally, I'm ready and committed to fix them. Will also accomodate that pathlib and argparser changes as well. I'll use both https://github.com/astral-sh/ruff and pylint to ensure scripts are of good quality.

thanks a lot for taking a look at this! it's fine to fix all these things in follow-up tickets, mainly because they tend to fall into different categories:

1. python problems with our code (a lot)
2. pylint problems that needs yetus configuration maybe

[Aggarwal-Raghav]

i have removed the tabs with space (just indentation change). that should fix the /blanks-eol.txt /blanks-tabs.txt and requirements.txt for now

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	shelldocs	0m 0s		Shelldocs was not available.
+1 :green_heart:	@author	0m 1s		The patch does not contain any @author tags.
			_ master Compile Tests _
-1 :x:	pylint	0m 1s	/branch-pylint-stderr.txt	Error running pylint. Please check pylint stderr files.
			_ Patch Compile Tests _
+1 :green_heart:	codespell	0m 4s		No new issues.
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	markdownlint	0m 3s		The patch generated 0 new + 45 unchanged - 1 fixed = 45 total (was 46)
-1 :x:	pylint	0m 2s	/patch-pylint-stderr.txt	Error running pylint. Please check pylint stderr files.
+1 :green_heart:	pylint	0m 3s		No new issues.
+1 :green_heart:	shellcheck	0m 0s		The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
			_ Other Tests _
-1 :x:	asflicense	0m 49s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		2m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-444/2/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/444
Optional Tests	dupname asflicense codespell detsecrets pylint markdownlint shellcheck shelldocs
uname	Linux 875f66e6fb1c 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-444/src/.yetus/personality.sh
git revision	master / d317d55f89ce7f3b6c17234623910f873e28df9c
Max. process+thread count	51 (vs. ulimit of 5500)
modules	C: tez-tools U: tez-tools
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-444/2/console
versions	git=2.43.0 maven=3.8.7 codespell=2.0.0 markdownlint=0.23.2 shellcheck=0.7.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

[Aggarwal-Raghav]

Earlier pylint rating:

logsplit.py: Your code has been rated at 6.85/10
swimlane.py: Your code has been rated at 0.00/10
counter-diff.py: Your code has been rated at 0.00/10
amlogparser.py: Your code has been rated at 0.00/10

pylint rating after reformatting:

logsplit.py: Your code has been rated at 6.99/10
swimlane.py: Your code has been rated at 6.17/10
counter-diff.py: Your code has been rated at 8.35/10
amlogparser.py: Your code has been rated at 6.71/10

[abstractdog]

patch looks good to me now, thanks @Aggarwal-Raghav, I'm merging this soon can you please create a follow-up for the pylint issue?

[Aggarwal-Raghav]

patch looks good to me now, thanks @Aggarwal-Raghav, I'm merging this soon can you please create a follow-up for the pylint issue?

Thanks for merging this @abstractdog . I'll create the follow-up JIRA to fix pylint errors and maybe further improving the scripts.