`bm_raster` returns `NULL`

[simon-smart88]

This has been the case for a month or so now. I've updated to the dev version but the problem persists. It seems unrelated to the sf object that I pass.

library(sf)

square_coords <- matrix(c(
  -1.1, 50.5,  
  0.9,  50.5, 
  0.9,  52.5,  
  -1.1, 52.5,  
  -1.1, 50.5   
), ncol = 2, byrow = TRUE)

square_polygon <- st_polygon(list(square_coords))
square_sf <- st_sfc(square_polygon, crs = 4326)  
sf_object <- st_sf(geometry = square_sf)

blackmarbler::bm_raster(roi_sf = sf_object,
                        product_id = "VNP46A4",
                        date = "2021",
                        bearer = Sys.getenv("NASA_bearer"),
                        quiet = FALSE)

Produces:

Processing 2 nighttime light tiles
Processing: VNP46A4.A2021001.h17v03.001.2022094115448.h5
  |========================================================================================================================| 100%
Downloading: 4.1 kB     NULL

[ramarty]

@simon-smart88 Thanks for flagging the issue. Was testing now, and there seem to be some issues on the NASA LAADS archive. Will keep checking to see if the problem is with the code or with the archive that the code is trying to access — and at the very least aim to include some more helpful error messages when there seems to be issues with the LAADS archive.

[firatgundem]

I think the problem still exists. Any improvements? Below my codes:

roi_sf <- gadm(country = "GHA", level=1, path = tempdir())

Daily data: raster for February 5, 2021

r_20210205 <- bm_raster(roi_sf = roi_sf, product_id = "VNP46A2", date = "2021-02-05", bearer = bearer) r_20210205 NULL

[firatgundem]

I think no one is checking this page. Is there anyone who can help? Why does bm_raster create a null data set?

[ramarty]

@firatgundem Sorry for the delay here! Does the dev version work for you?

install.packages("devtools") devtools::install_github("worldbank/blackmarbler")

Sometimes there's issues as the NASA LAADs archive is down which will cause NULL to be returned - I'm working on having the package return a message when this is the case, but one way to see if that's the source of the error is if you can manually download a file:

https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A4/2012/001/

Another issues could be the bearer token expired, so could be worth trying to regenerate that.

[firatgundem]

Yes, I tried devtools as well but got the same thing. I have been trying to obtain this blackmarble dataset for more than 1 week and I could only get the null data. Could you please help me?

[ramarty]

Ah thanks for this!

1/ Could be worth making sure all the packages it depends on are up to date?

readr, hdf5r, dplyr, purrr, lubridate, tidyr, terra, sf, exactextractr, stringr, httr

2/ Are you on a mac or pc?

[firatgundem]

I use pc and my packages are up-to-date. This is the latest error that I got:

black_marble_data = bm_raster(roi_sf = shapefile, # this specifies the region of interest

•                            product_id = "VNP46A3", # this specifies the monthly luminosity data
  

•                            date = seq.Date(from = ymd("2021-10-01"), to = ymd("2021-12-01"), by = "month"), # this specifies the date range
  

•                            bearer = bearer, # this is the API bearer for NASA LAADS DAAC from Step 3
  

•                            variable = c("AllAngle_Composite_Snow_Free"), # this selects the layer we're interested in i.e. satellite angle and snow-free observations
  

•                            quality_flag_rm = c(255, 2)) # this drops bad quality observations and coverts them to NA values)
  

Processing 1 nighttime light tiles Processing: VNP46A3.A2021274.h33v12.001.2021321133308.h5 |===============================================| 100% Error in downloading data No encoding supplied: defaulting to UTF-8.

504 Gateway Time-out

---

nginx

Error in downloading data; bearer token likely invalid. Try regenerating the bearer token; please see this link for instructions to obtain a bearer token: https://github.com/worldbank/blackmarbler?tab=readme-ov-file#bearer-token- Processing 1 nighttime light tiles Processing: VNP46A3.A2021305.h33v12.001.2021343154743.h5 |===============================================| 100% Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1) Processing 1 nighttime light tiles Processing: VNP46A3.A2021335.h33v12.001.2022008110827.h5 |===============================================| 100% Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1)

Storing the layer names from the original raster stack

original_layer_names = names(black_marble_data)

Imputing missing values in the raster data

imputed_data = approxNA(black_marble_data, method="linear", rule=2, f=0) #addressing the NA values by linear interpolation Error: unable to find an inherited method for function ‘approxNA’ for signature ‘x = "NULL"’

It is also interesting to see these. Although it says the bearer token is likely invalid, I have just renewed it. How can it be invalid?

[marcoaurelioguerrap]

I'm having the same problem too, I'll try to debug the code now.

[marcoaurelioguerrap]

Yeah, the problem with me was the bearer key. It changed, Although I did not request the change.

[simon-smart88]

@marcoaurelioguerrap

Here's a function that will stop that being an issue. It fetches an existing token or creates a new one if none exist:

get_nasa_token <- function(username, password) {

  token_url <- "https://urs.earthdata.nasa.gov/api/users/find_or_create_token"

  req <- httr2::request(token_url)

  response <- tryCatch(
    req |>
      httr2::req_auth_basic(username, password) |>
      httr2::req_method("POST") |>
      httr2::req_perform(),
    httr2_http_401 = function(cnd){NULL}
  )

  if (httr2::resp_status(response) == 200) {
    body <- response %>% httr2::resp_body_json()
    token <- body$access_token
    return(token)
  } else {
    return()
  }

}

[ramarty]

@simon-smart88 Thanks for writing that function. Would love to add to the package + note in the documentation if OK with you.

If you're willing to do a pull request that adds the function in here that'd be great - or I'm happy to just add myself if you prefer. Then I can test as well, add that to the documentation, then send the update to CRAN.

(I may also add an option in the bm_raster/bm_extract functions for a user to either include (1) bearer or (2) username/password).

[simon-smart88]

Yes sure I will do - I think it probably needs a few tweaks to be robust to all eventualities though so will work on that and submit a PR.

[ramarty]

Amazing, thanks @simon-smart88!