agdc
agdc copied to clipboard
Published
20 hours ago •
GeoscienceAustralia
GeoscienceAustralia
missing files?
Hi DataCubers, A quick question about discovering what data are available in the datacube (Landsat). First let me clarify that I am not using the Python API at the moment.
What I do is search by filename. For example if I want L8 for a given tile and a given year and month, I go to the corresponding folder and look all the filenames that match my requirements. In most cases I get a right answer but in some cases there are files missing
Example: for L8 in the tile 145 -35 and January 2015, I go to /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-035/2015/ And then search using the string _2015-01-
In that particular case I find 3 NBAR : /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-035/2015/LS8_OLI_TIRS_NBAR_145_-035_2015-01-08T00-08-18.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-035/2015/LS8_OLI_TIRS_NBAR_145_-035_2015-01-17T00-02-28.tif
- But there are no PQA files for them!! -
Another example, LS7, 2015, December, tile 145, -36 I find only two NBAR /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_NBAR_145_-036_2015-12-02T00-09-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_NBAR_145_-036_2015-12-02T00-09-58.tif
But 5 FC (which need NBAR to be done first) /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_FC_145_-036_2015-12-02T00-09-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_FC_145_-036_2015-12-02T00-09-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_FC_145_-036_2015-12-11T00-03-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_FC_145_-036_2015-12-11T00-03-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_FC_145_-036_2015-12-27T00-03-47.tif
And 8 PQA!! /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-02T00-09-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-02T00-09-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-11T00-03-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-11T00-03-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-18T00-09-51.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-18T00-10-15.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-27T00-03-47.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_PQA_145_-036_2015-12-27T00-04-11.tif
Any idea why and if I am doing anything wrong? Is there a python function that does this? Or is there any other easier way to find what’s available?
Cheers Juan
in addition to my previous post. I'm running a test and it looks like the issue I reported applies only to LS7 and LS8 and from 2013 to current (i.e. no problem with 2012 or earlier and/or LS5).
Take a look at the examples: https://github.com/GeoscienceAustralia/agdc/tree/develop/api-examples/notebooks
The docs page has a lot more information on data discovery and command line tools: http://geoscienceaustralia.github.io/agdc/api/0.1.0/html/index.html http://geoscienceaustralia.github.io/agdc/api/0.1.0/html/api_overview.html http://geoscienceaustralia.github.io/agdc/api/0.1.0/html/api_tools.html
Also you're better off using the api, as in your example you're finding files that need to be mosaiced, eg:
from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list
satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2015, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]
tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)
for tile in tiles:
print "Acquisition time: {}".format(tile.start_datetime)
nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
print "NBAR path: {}".format(nbar.path)
print "PQ path: {}".format(pq.path)
print "FC path: {}\n".format(fc.path)
Acquisition time: 2015-12-02 00:09:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-02T00-09-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-02T00-09-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-02T00-09-34.vrt
Acquisition time: 2015-12-11 00:03:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-11T00-03-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-11T00-03-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-11T00-03-34.vrt
Acquisition time: 2015-12-18 00:09:51
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-18T00-09-51.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-18T00-09-51.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-18T00-09-51.vrt
Acquisition time: 2015-12-27 00:03:47
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-27T00-03-47.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-27T00-03-47.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-27T00-03-47.vrt```
PQ data may not exist for some equivalent NBAR datasets, as the ingest requirements were relaxed so that NBAR could be ingested without an accompanying PQ or FC dataset.
Hope that helps
Thanks Josh, I didn't know about all these documentation and tools.
Juan
Josh, your example retrieves data ok. However, when I try to retrieve the same info using the API command Line Tools (which I found very useful) I don't get anything from 2015 or 2016 back.
example:
$ retrieve_dataset.py --x 145 --y -36 --satellite LS5 LS7 LS8 --acq-min 2015-01 --acq-max 2016-12 --dataset-type ARG25 --list-only --output-directory /tmp
2016-01-28 15:27:01,442 INFO
acq = 2015-01-01 to 2016-12-31
satellites = LS5 LS7 LS8
PQA mask =
WOFS mask =
2016-01-28 15:27:01,442 INFO
x = 145
y = -036
VECTOR mask =
2016-01-28 15:27:01,442 INFO
datasets to retrieve = ARG25
output directory = /tmp
over write existing = False
list only = True
output format = GEOTIFF
$
same example but starting in 2014 returns existing data for 2014 but nothing for 2015 onwards.
$ retrieve_dataset.py --x 145 --y -36 --satellite LS5 LS7 LS8 --acq-min 2014-01 --acq-max 2016-12 --dataset-type ARG25 --list-only --output-directory /tmp
2016-01-28 16:11:54,611 INFO
acq = 2014-01-01 to 2016-12-31
satellites = LS5 LS7 LS8
PQA mask =
WOFS mask =
2016-01-28 16:11:54,612 INFO
x = 145
y = -036
VECTOR mask =
2016-01-28 16:11:54,612 INFO
datasets to retrieve = ARG25
output directory = /tmp
over write existing = False
list only = True
output format = GEOTIFF
2016-01-28 16:11:55,215 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-01-05T00-09-59.vrt]
2016-01-28 16:11:55,215 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-01-05T23-58-54.048441.tif]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-01-13T00-05-09.401713.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-01-14T00-03-38.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-01-29T00-05-28.166291.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-01-30T00-03-33.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-02-06T00-09-38.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-02-06T23-59-20.672816.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-02-22T00-09-25.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-02-22T23-59-10.182069.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-03-02T00-05-23.197369.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-03T00-03-02.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-10T00-09-12.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-03-10T23-59-16.320070.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-03-18T00-05-31.689687.tif]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-19T00-02-55.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-26T00-08-56.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-04-11T00-09-07.tif]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-04-11T23-59-29.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-04-19T00-05-46.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-04-20T00-02-22.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-04-27T00-08-27.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-04-27T23-59-40.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-05-06T00-02-29.tif]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-05-13T00-08-13.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-05-13T23-59-46.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-05-29T23-59-50.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-06-06T00-06-03.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-06-15T00-00-16.tif]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-06-22T00-06-04.tif]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-07-09T00-02-18.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-07-16T00-08-31.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-07-16T23-59-59.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-07-25T00-02-20.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-08-01T00-08-36.tif]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-08-02T00-00-06.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-08-09T00-06-18.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-08-10T00-02-30.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-08-17T00-09-06.tif]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-08-18T00-00-06.tif]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-08-25T00-06-15.tif]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-08-26T00-02-32.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-02T00-08-45.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-09-03T00-00-06.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-11T00-02-36.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-18T00-08-46.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-09-19T00-00-10.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-09-26T00-06-24.tif]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-27T00-02-34.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-04T00-08-49.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-10-05T00-00-22.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-10-12T00-06-36.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-13T00-02-41.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-20T00-08-53.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-10-21T00-00-31.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-10-28T00-06-45.tif]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-29T00-02-38.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-11-05T00-08-51.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-11-06T00-00-41.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-11-13T00-06-54.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-11-14T00-02-41.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-11-21T00-08-47.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-11-22T00-00-44.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-11-29T00-06-59.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-12-07T00-08-47.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-12-15T00-07-07.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-12-16T00-02-58.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-12-23T00-08-45.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-12-24T00-01-00.tif]
2016-01-28 16:11:55,223 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-12-31T00-07-08.vrt]
$
Any ideas?
Thanks
Juan
by the way, none of the command line API tools gets anything after 2014...
last comment:
I tried to replicate what you suggested and couldn't get a proper answer either (for data in 2015). Example:
from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list
satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2015, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]
tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)
for tile in tiles:
print "Acquisition time: {}".format(tile.start_datetime)
nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
print "NBAR path: {}".format(nbar.path)
print "PQ path: {}".format(pq.path)
print "FC path: {}\n".format(fc.path)
result is empty
try this now:
from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list
satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2014, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]
tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)
for tile in tiles:
print "Acquisition time: {}".format(tile.start_datetime)
nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
print "NBAR path: {}".format(nbar.path)
print "PQ path: {}".format(pq.path)
print "FC path: {}\n".format(fc.path)
and get
Acquisition time: 2014-12-15 00:07:07
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-12-15T00-07-07.tif
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_PQA_145_-036_2014-12-15T00-07-07.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_FC_145_-036_2014-12-15T00-07-07.tif
Acquisition time: 2014-12-31 00:07:08
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-12-31T00-07-08.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_PQA_145_-036_2014-12-31T00-07-08.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_FC_145_-036_2014-12-31T00-07-08.vrt
looks like some file needs to be utdated?
Juan
Your first example works for me:
retrieve_dataset.py --x 145 --y -36 --satellite LS5 LS7 LS8 --acq-min 2015-01 --acq-max 2016-12 --dataset-type ARG25 --list-onl
y --output-directory $PWD
2016-01-29 08:54:44,470 INFO
acq = 2015-01-01 to 2016-12-31
satellites = LS5 LS7 LS8
LS7 SLC OFF = INCLUDED
LS8 PRE WRS2 = INCLUDED
PQA mask =
WOFS mask =
season =
2016-01-29 08:54:44,470 INFO
x = 145
y = -036
VECTOR mask =
2016-01-29 08:54:44,470 INFO
datasets to retrieve = ARG25
bands to retrieve = ['BLUE', 'GREEN', 'RED', 'NEAR_INFRARED', 'SHORT_WAVE_INFRARED_1', 'SHORT_WAVE_INFRARED_2', 'COASTAL_AEROSOL']
output directory = /short/v10/jps547/tmp/dcube-test
over write existing = False
list only = True
output format = GEOTIFF
2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2015/LS8_OLI_TIRS_NBAR_145_-036_2015-
01-01T00-02-53.tif]
2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-01-16T00-07-12.vrt]
2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2015/LS8_OLI_TIRS_NBAR_145_-036_2015-
01-17T00-02-28.tif]
2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-01-25T00-01-04.vrt]
2016-01-29 08:54:50,976 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_NBAR_145_-036_2015-02-01T00-0
7-19.tif]
.
.
.
.
2016-01-29 08:54:50,981 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-12-11T00-03-34.vrt]
2016-01-29 08:54:50,981 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-12-18T00-09-51.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2015/mosaic_cache/LS8_OLI_TIRS_NBAR_1
45_-036_2015-12-19T00-02-36.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-12-27T00-03-47.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2016/mosaic_cache/LS8_OLI_TIRS_NBAR_1
45_-036_2016-01-11T00-08-43.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2016/mosaic_cache/LS7_ETM_NBAR_145_-036_20
16-01-12T00-04-03.vrt]
Maybe you're using a different version???
From the u46 project modules I used the following module to produce the above output: agdc-api/0.1.0-b20160111
This might resolve your other questions regarding data not being found post 2014.
Your Python example also works for me.
from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list
satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2015, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]
tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)
for tile in tiles:
print "Acquisition time: {}".format(tile.start_datetime)
nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
print "NBAR path: {}".format(nbar.path)
print "PQ path: {}".format(pq.path)
print "FC path: {}\n".format(fc.path)
Acquisition time: 2015-12-02 00:09:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-02T00-09-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-02T00-09-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-02T00-09-34.vrt
Acquisition time: 2015-12-11 00:03:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-11T00-03-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-11T00-03-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-11T00-03-34.vrt
Acquisition time: 2015-12-18 00:09:51
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-18T00-09-51.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-18T00-09-51.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-18T00-09-51.vrt
Acquisition time: 2015-12-27 00:03:47
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-27T00-03-47.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-27T00-03-47.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-27T00-03-47.vrt
So could be the version that you're using is pointing to a different database.
I actually tried loading a newer module yesterday, but I get the following error message:
[jpg599@raijin2 ~]$ module use /projects/u46/opt/modules/modulefiles
[jpg599@raijin2 ~]$ module load agdc-api/0.1.0-b20160111
utility.c(2165):ERROR:50: Cannot open file '/projects/u46/opt/modules/modulefiles/eo-tools/dev' for 'reading'
[jpg599@raijin2 ~]$