some scamper tests filtered by code change, cause tarball not reporcessed

[yachang]

SELECT * FROM mlab-oti.ndt.traceroute WHERE Parseinfo.TaskFileName = "gs://archive-measurement-lab/ndt/traceroute/2019/11/02/20191102T020000.909108Z-traceroute-mlab4-lhr05-ndt.tgz" LIMIT 1000

Row	partition_date	uuid	TestTime	Parseinfo.TaskFileName	Parseinfo.ParseTime	Parseinfo.ParserVersion	Parseinfo.Filename	start_time	stop_time	scamper_version	Source.IP	Source.Port	Source.IATA	Source.Geo.continent_code	Source.Geo.country_code	Source.Geo.country_code3	Source.Geo.country_name	Source.Geo.region	Source.Geo.metro_code	Source.Geo.city	Source.Geo.area_code	Source.Geo.postal_code	Source.Geo.latitude	Source.Geo.longitude	Source.Geo.radius	Source.Network.IPPrefix	Source.Network.Systems.ASNs	Destination.IP	Destination.Port	Destination.Geo.continent_code	Destination.Geo.country_code	Destination.Geo.country_code3	Destination.Geo.country_name	Destination.Geo.region	Destination.Geo.metro_code	Destination.Geo.city	Destination.Geo.area_code	Destination.Geo.postal_code	Destination.Geo.latitude	Destination.Geo.longitude	Destination.Geo.radius	Destination.Network.IPPrefix	Destination.Network.Systems.ASNs	ProbeSize	ProbeC	Hop.Source.IP	Hop.Source.City	Hop.Source.CountryCode	Hop.Source.Hostname	Hop.Source.ASN	Hop.Linkc	Hop.Links.HopDstIP	Hop.Links.TTL	Hop.Links.Probes.Flowid	Hop.Links.Probes.Rtt	exp_version	cached_result
1	2019-11-02	ndt-qzlhp_1571972644_00000000000148CB	2019-11-02 01:45:00 UTC	gs://archive-measurement-lab/ndt/traceroute/2019/11/02/20191102T020000.909108Z-traceroute-mlab4-lhr05-ndt.tgz	2019-11-05 00:33:18.164853 UTC	https://github.com/m-lab/etl/tree/prod-v2.0.1	null	1572659100	1572659100	0.1	::ffff:212.113.31.50	0		EU	GB		United Kingdom	ENG	0	York	0	YO10	53.9573	-1.0837	1		3356	::ffff:35.225.75.192	0	NA	US		United States	VA	556		0		38.6583	-77.2481	0		15169	60	0											null	null
2	2019-11-02	ndt-qzlhp_1571972644_00000000000145CD	2019-11-02 00:10:01 UTC	gs://archive-measurement-lab/ndt/traceroute/2019/11/02/20191102T020000.909108Z-traceroute-mlab4-lhr05-ndt.tgz	2019-11-03 18:28:10.211337 UTC	https://github.com/m-lab/etl/tree/prod-v2.0.1	null	1572653281	1572653281	0.1	::ffff:212.113.31.50	0		EU	GB		United Kingdom	ENG	0	York	0	YO10	53.9573	-1.0837	1		3356	::ffff:35.225.75.192	0	NA	US		United States	VA	556		0		38.6583	-77.2481	0		15169	60	0											null	null

The reason is that ::ffff handling code change, make the majority of tests in that tarball filtered, thus failed Gardener 99% sanity check.

[yachang]

PR: #830 && m-lab/etl-gardener#232

And https://github.com/m-lab/etl/releases/tag/prod-v2.2.3

Fix this problem.

SELECT Parseinfo.ParseTime, Parseinfo.TaskFileName FROM mlab-staging.base_tables.traceroute WHERE _PARTITIONTIME > "2019-01-24" AND Parseinfo.ParseTime < "2019-12-12"

And there is 0 result. Yay :)

[yachang]

Reopen issue:

gs://archive-mlab-oti/host/traceroute/2019/11/15/20191115T034951.000655Z-traceroute-mlab1-tpe01-host.tgz

was not reprocessed since 2019/11/15 and trigger the same error again.

[yachang]

for archive-mlab-staging bucket, the host/traceroute dir has all mlab4 data, which is different from mlab-oti (with NO overlapping)

I will use mlab-testing to reproduce the reprocessing process for above tarball.

[yachang]

The reason is: back to 2019/11/155, we played with all kinds of alternatives, such as new version of paris traceroute, and for very short time, there were ill-formatted paris traceroute tests generated in

gs://archive-measurement-lab/paris-traceroute/2019/11/15/

which block the reprocessing of all 2019/11/15 data.

After we manually delete the current BQ data with PartitionTime = 20191115, this problem will be fixed automatically.