yachang

Major concerns about the design: - whether user can locate the test as easily as before when some field like UUID pushed to second level? - whether the join table...

Here are some BQ using inner join: SELECT COUNT(*) as total FROM `measurement-lab.ndt.ndt5` as ndt5 INNER JOIN `mlab-oti.base_tables.traceroute` as troute on ndt5.result.s2c.uuid = troute.uuid WHERE result.S2C IS NOT NULL AND...

SELECT COUNT(*) as total FROM `measurement-lab.ndt.ndt5` as ndt5 LEFT JOIN `mlab-oti.base_tables.traceroute` as troute on ndt5.result.s2c.ClientIP = troute.Destination.IP WHERE result.S2C IS NOT NULL AND ndt5.partition_date BETWEEN DATE("2019-09-01") AND DATE("2019-10-01") AND ndt5.result.S2C.uuid...

For .raw schema, it is defined in https://reviewable.io/reviews/m-lab/traceroute-caller/74

https://github.com/m-lab/etl/pull/886

PR: #830 && m-lab/etl-gardener#232 And https://github.com/m-lab/etl/releases/tag/prod-v2.2.3 Fix this problem. SELECT Parseinfo.ParseTime, Parseinfo.TaskFileName FROM `mlab-staging.base_tables.traceroute` WHERE _PARTITIONTIME > "2019-01-24" AND Parseinfo.ParseTime < "2019-12-12" And there is 0 result. Yay :)

Reopen issue: gs://archive-mlab-oti/host/traceroute/2019/11/15/20191115T034951.000655Z-traceroute-mlab1-tpe01-host.tgz was not reprocessed since 2019/11/15 and trigger the same error again.

for archive-mlab-staging bucket, the host/traceroute dir has all mlab4 data, which is different from mlab-oti (with NO overlapping) I will use mlab-testing to reproduce the reprocessing process for above tarball.

The reason is: back to 2019/11/155, we played with all kinds of alternatives, such as new version of paris traceroute, and for very short time, there were ill-formatted paris traceroute...

The reason is that in staging, we deploy a latest paris-traceroute binary, which generate test w/ different format compared to legacy paris traceroute output: sample of old one: ================= traceroute...