Python Utilities

Different python utility scripts to help automate mundane/repetitive/specific performance testing tasks. The readme page will continue to get updated as and when I add a new utility to the repo.

Note: Most of these utilities are focused around performance testing/engineering. However, with some modifications(/in some cases none) can be used in other fields/areas too.

1: Merge Files Column

This script merges columns from multiple files and generates a new file.

2: Unique and Sorted

This script extracts data from a file, removes rows with no data, sorts and saves unique data into a file.

3: Histogram

This script generates response time distribution graph (histrogram) from the JMeter result CSV. Can be extended for others tools too.

4: Extract Matched Data

This script extracts response time data that matches a specific text in a column and saves into a new file.

5: Swap Columns

This script swaps columns in a file. Useful when trying to rearrange columns for easy of use/readability. For demonstration purpose, the script rearranges JMeter result csv columns as I like them to view. However can be extended for other use too.

6: Randomize Data

This script randomizes data in a file. Useful when you want to have a random order of data in file for testing purpose.

7: Unique Occurrence Count

This script saves total number of occurrence of each unique item in a file. Useful for designing the test data distribution for testing.

8: Split File By Text

Extract data that matches a text in the data file and save it to a new file.Useful when you have a single large data file containing all of the data and wish to create different data for each test run.

For example, suppose you have a single large data file with images, java, and CSS URLs. And you want to build distinct data files for CSS, JS, and pictures. This script will assist you in doing so.

In the screenshot below, team was not part of the search text, hence no csv was created for it. The name is the file's column name.

9: Heatmap

There may be instances when using a line chart, it is difficult to detect slight swings in data. This script creates a heatmap for the data when you are attempting to observe patterns over minutes/hours but for a longer period of time (i.e.30 days). For example, there may be a specific hour of the day when there is increased load, but it is not greater than the peak load for the day. As a result, a line chart for a longer period of time (e.g., 30 days) may hide or obscure that pattern.

The same thing can be done in Excel using a pivot table, but it will take some manual effort.

10: Pivot Table

This script converts extensive data (two/three column data) into a summairzed pivot table format. Useful when you want to have a summarized view of the requests/error/response time over long period.

11: Generate ABN And ACN

Generate random ABN and ACN numbers. Useful for performance/functional test scenario's that require valid ABN and/or ACN data for testing.

12: Network Conversation Heatmap

Create a heatmap from the network conversation obtained in the trace file. When you have a large number of conversations captured, this is really useful. If you are not comfortable reading/trolling the Wireshark network conversation view, this script will help you visualise the conversations. It also gives the option of generating a graph. However, the technology used to produce the graph will need to be tweaked somewhat to accommodate too many conversations. The current code should suffice for less than 40 conversations in a trace file.

13: Creditcard Generator

Generate Mastercard/Visa creditcard numbers. Useful when dummy creditcard numbers are needed for Testing purpose ONLY. They are useless without the valid owner name, an expiration date and a valid CVV code. Therefore they CAN NOT be used for REAL transactions.

14: TFN Generator

Generate a list of valid Australian Tax File Numbers (TFN). It is useful when you need TFN numbers for testing purposes.

15: IRD Generator

Generate a list of valid New Zealand Inland Revenue Department Numbers (IRD). It is useful when you need IRD numbers for testing purposes.

16: Dollar Format

Format the data file with a monetary value. When you need to transmit a dollar value parameter from a file in a payload instead of a number, this is useful. The majority of the time, it should be handled in code. If you are passing it through a file, this script can help you save time.

17: File Splitter

Split a big file into multiple smaller size files. Useful when you want to have unique data for the same script running across multiple injectors.

18: Locate File

Search for a file in a folder and all of its subfolders recursively. Print the locations where that file can be found. When you don't want to scroll through all the folders and files to get the file you need, this feature comes in handy.

19: Websphere Verbosegc

Save the high-level websphere verbosegc metrics to a csv file. When you wish to import data into a load test tool for analysis and correlation with other metrics, this is useful. Assumption is that you have other tools (e.g., apm tools, sitescope, etc.) to view this data alongside other application and system metrics.

20: Formatted Server Metrics To Excel

In some cases, you will receive metrics data for all servers in an excel file in a column format. You would prefer, however, to have metrics data for each server on a separate sheet. Also, the data is appropriately formatted and pivoted, making it simple to analyse and build graphs from it. This script will assist you in doing so.

It accepts data in column format and stores pivoted data from each system/application to a separate sheet.

21: Merge Columns

This script merges multiple columns in a file to two columns. One column with the values and other the header names. Useful when you want to do analysis (i.e. Tukey test on the data set) using python.

22: Arrange Files

This script organises files in the right folder so that they are easier to find when needed. When you wish to have a separate folder for data, scripts, scenarios, and results, this is useful. Also excellent for organising all of the contents in your download folder, which may contain photos, music, executables, and video files.

23: Outliers

To identify outliers in a data collection, use the Tukey fence test. When you merely want to know the outlier values, this is a great tool. Also useful when extreme outliers distort the display and you need to eliminate them temporarily in order to study the rest of the data.

24: Basic Statistics

Generate basic statistics. Useful when you want to generate & compare statistics of different test runs. Saves time of not filling in all the formulas in excel.

25: Generate Name

Generate list of random names. Useful when you want to have alot of random names for testing purpose. Following options are available:

1. First name Only
2. First & Last name
3. Full name
4. Full name but abbreviated Middle name

26: File Detail

Recursively list all files present in the directory (& sub directory) with File type & permission, File Size, Owner Id and Last modified date. The file detail is saved in a text file.

27: Port Scanner

Given a list of hostname/ipaddresses, scan for open ports. Useful for system admin to identify which ports are open and which are closed. Useful for performance engineers too.

28: Google Lighthouse

For a given URL, capture lighthouse performance and debug metrics. Useful for webpage performance analysis. You can extend the code to save the metrics to a time-series DB (i.e. influxdb) and use visualization tools such as Grafana to view the metrics.

29: Multi Plots

This script shows you how to create an image with multiple plots/graphs & table. Useful for generating graphs out of JMeter raw data. Can we modified to cater for other tools like Gatling.

30: Split String in Column

This script splits strings using a delimiter in a column and saves all the resultant strings to a csv. Useful when you need to get total count.

31: Remove Consumed Data

This script removes consumed data from the original data set. Useful for those tests where you have a consumable data and you need to refresh the original data file with data that is still valid and usable.

32: Word Cloud

Create a word cloud out of title, name etc.

33: Influxdb JMeter Plot

This simple utility connects to Influxdb, downloads the JMeter response time data and plots a chart. It demostrate how to accomplish the task. Script can be modified as per your need.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Note: 
1: The script was tested on Windows OS.
2: NovatecConsulting JMeter plugin was used in JMeter to send data to Influxdb. 

Link: https://github.com/NovaTecConsulting/JMeter-InfluxDB-Writer/releases

Prerequisites

What things you need to execute the script

1: Python 3.5
2: Influxdb, Panda & Matlab packages installed

Execution

1: Make sure above prerequisite are met first.
2: Update the Influxdb connection detail in the script.
3: Update the Influx query accordingly. Sample query example are provided in the script.
4: Update the script with the correct timezone. By default the script timezone is set to Australia/Melbourne.
5: Run the python script

34: OSWatcher top

There are situations when you have OSWbb data but don't have the OSWbba analyser to view it. The system metrics (CPU/memory/load/task/swap) gathered by the oswtop script are extracted using this python script. The script will extract the system metrics and save them in a CSV file after you provide it the oswtop file. The data can then be visualised using any visualisation tool or imported into a load testing tool (if the programme allows it).

35: Tornado Graph

Generate response time tornado graph. This type of graph is excellent for showing points of freezing in a system, more so than standard scatter graphs.

Contribute

Contribution is welcomed. Pull requests are welcomed too.

Author