etl

etl copied to clipboard

Need row filter tags on server name or identity and protocol family At the very least the (protocol specifici) host name: ndt.iupui.mlab1v6.lga03.measurement-lab.org. Better would be to bust it out into...

mattmathis

SELECT result.StartTime, _PARTITIONTIME AS pt FROM `mlab-oti.base_tables.ndt5` WHERE DATE(result.StartTime) != DATE(_PARTITIONTIME) This shows that files created late in the day are tarred into files with next day's date, and the...

gfr10598

#685 addresses the large buffer problem, but does not address the large single row problem. We should limit the size of a single row, by reducing the number of snapshots.

gfr10598

1. Design Doc 2. Large machine config 3. File iterator 4. Load management

gfr10598

Part of #624

gfr10598

parser should use directory date instead of file date for pusher files.

1

There are also anomalies in scraper files, so should work out the proper handling there too. In the long term, we should partition by start time.

gfr10598

TEST_SERVICE_ACCOUNT_mlab_testing is currently base64 encoded. Update to use mechanism in etl-gardener.

gfr10598

TCPParser unit tests appear to depend on annotator. ``` 2019/06/19 17:56:45 task.go:138: Processed 364 files, 0 nil data, 362 rows committed, 0 failed, from testdata/20190516T013026.744845Z-tcpinfo-mlab4-arn02-ndt.tgz into ndt_test --- PASS: TestTCPParser...

stephen-soltesz

Should check a number of fields for negative, large, or zero values, both in web100 and tcpinfo. Also see https://github.com/m-lab/etl/issues/682

gfr10598

TCPInfo parser should include test filename, and dedup based on filename instead of UUID

1

Current parser/schema does not include the filename. UUID specifies a connection, but there may be more than one file for connections that are open more than 10 minutes. The dedup...

gfr10598