created test for valid selector that does not increase time

[brandonscholet]

My room wappybird implement ls your library. I started pulling the updated wappalyzer libraries. They have had issues with valid json, so I started pulling the current release of, but the tally selector is malformed. I talked to the maintainer of soupsieve and they provided a function to tech for valid selectors and skip if not. This replaces the crude try/catch code

I can update your repo to pull the current technologies if you would like. Or feel free to pull from wappybird.

Also, the pip is out of date and incompatible with the updated technologies files

[tristanlatr]

Thanks @brandonscholet. Can you provide a test for an invalid selector please ?

[brandonscholet]

The current release of npm-Wappalyzer has this broken selector Broken Selector iframe[scr*='//airtable.com/'], a[href*='//airtable.com/][target='_blank']

[brandonscholet]

This will pull the latest into the technology file. They have had broken selectors for the past two releases

def update_technologies_from_latest():
	print("updating technologies")
	technologies_file = os.path.expanduser('~/.python-Wappalyzer/technologies.json')
	technologies = {}
	 
	#get release page
	latest_release = requests.get('https://api.github.com/repos/wappalyzer/wappalyzer/releases/latest').json()
	#get zip from url
	zip_url = requests.get(latest_release['zipball_url'])
	myzip = ZipFile(io.BytesIO(zip_url.content)) 

	#parse files
	for listed_file in myzip.namelist():
		#get all technology files
		if "src/technologies" in listed_file and ".json" in listed_file:
			#extract file into json
			tech_json_file=myzip.read(listed_file).decode('UTF-8')
			tech_json = json.loads(tech_json_file)
			#add to full json
			technologies = {**technologies, **tech_json}
		if "src/categories.json" in listed_file:
			#extract categories into json
			categories = json.loads(myzip.read(listed_file).decode('UTF-8'))
		#merge into one object
	combined_object = {'categories': categories, 'technologies': technologies}

	#write to file
	with open(technologies_file, 'w', encoding='utf-8') as tfile:
	    tfile.write(json.dumps(combined_object))
	    tfile.flush()
	print("done!\n")

webpage = WebPage.new_from_url("https://example.com", verify=False, timeout=60)
wappalyzer= Wappalyzer.latest(technologies_file=technologies_file)
techs = wappalyzer.analyze_with_versions_and_categories(webpage)

[brandonscholet]

looking back, the print statement should probably be removed.