'calulations'=TRUE doesn't return net and percent changes

[SteadyGiant]

When I set 'calculations'=TRUE in the payload, and then use it to call blsAPI(), the response does not contain net or percent changes. The value column contains the same values as it does when I call blsAPI() with 'calculations'=FALSE. See the following:

library(blsAPI)
library(rjson)

payload1 <- list(
  'seriesid'='LAUCN040010000000005',
  'startyear'=2010,
  'endyear'=2012,
  'catalog'=FALSE,
  'calculations'=TRUE,                                                # Here it's TRUE
  'annualaverage'=FALSE,
  'registrationKey'=Sys.getenv("BLS_REG_KEY"))
response1 <- blsAPI(payload1, 2)
json1 = fromJSON(response1)

payload2 <- list(
  'seriesid'='LAUCN040010000000005',
  'startyear'=2010,
  'endyear'=2012,
  'catalog'=FALSE,
  'calculations'=FALSE,                                               # Here it's FALSE
  'annualaverage'=FALSE,
  'registrationKey'=Sys.getenv("BLS_REG_KEY"))
response2 <- blsAPI(payload2, 2)
json2 = fromJSON(response2)

apiDF <- function(data){
  df <- data.frame(year=character(),
                   period=character(),
                   periodName=character(),
                   value=character(),
                   stringsAsFactors=FALSE)
  i <- 0
  for(d in data){
    i <- i + 1
    df[i,] <- unlist(d)
  }
  return(df)
}

unemployed.df1 <- apiDF(json1$Results$series[[1]]$data)
unemployed.df2 <- apiDF(json2$Results$series[[1]]$data)

identical(unemployed.df1, unemployed.df2)
# [1] TRUE 

Am I doing something wrong?

[dmichael555]

Hi, Mike,

I know the original post is from a while ago, but I saw the same performance as described in the original post when I used the 'calculations=TRUE' setting.

I looked into your code and noticed that the post-processing of the JSON response seems to pull out only four of the columns returned in the response, i.e., year, period, periodName, and value. For the first data element in the returned list, the code assigns NULL to remove 'footnotes' and 'latest.'

Here's what I put together as a workaround for extracting net changes and percent changes.

#Global cleanup rm(list = ls(all.names = TRUE))

library(blsAPI) library(rjson)

my_api_key<-"YOUR_API_KEY_HERE"

#Date Range Parameters

startyear<-2005 endyear<-2021

#CES65000000001, Health and Education Services payload_net<-list('seriesid'='CES6500000001','registrationKey'=my_api_key,'startyear'=startyear,'endyear'=endyear,'calculations'=TRUE,'annualaverage'=FALSE) health_and_ed_with_net<-blsAPI(payload_net,api_version=2,return_data_frame = FALSE) summary(health_and_ed_with_net) View(health_and_ed_with_net)

test<-fromJSON(health_and_ed_with_net) View(test)

New Version of loop from original function

#Iterate over the series number_of_series <- length(test$Results$series) for (i in 1:number_of_series){ #Set the default structure of the data frame df_start <- data.frame(year = character(), period = character(), periodName = character(), value = character(), net_change_1 = character(), net_change_3 = character(), net_change_6 = character(), net_change_12 = character(),
pct_change_1 = character(), pct_change_3 = character(), pct_change_6 = character(), pct_change_12 = character(),
stringsAsFactors = FALSE) #Get the data series_data <- test$Results$series[[i]]$data #Can get no data after a successful request if (length(series_data) > 0) { j <- 0 #index at 0 to start for (d in series_data) { j <- j + 1 #advance by one #Remove the footnotes from the list to stop the warnings d$footnotes <- NULL d$latest <- NULL #Add record to the data frame df_start[j, ] <- unlist(d) } #Add in the series id df_start$seriesID <- test$Results$series[[i]]$seriesID } #Create the data frame that will be returned if (!exists("df_to_return")){ #Data frame to return not defined so create it df_to_return <- df_start } else { #Append to the existing data frame df_to_return <- rbind(df_to_return, df_start) } }

View(df_to_return)

Using this approach, I was able to pull out the remaining data in the list, i.e., the net change and percentage change values.

Here's what my test case produced:

It appears to match the lists returned in the JSON response.

Let us know if you've already implemented something similar within the package.

All the best, Darren

[adhi-r]

Darren, at this point i’d recommend you use the package “blscrapeR”. it’s far better maintained and convenient. no offense intended to mike but i think he’d agree if anything.

On Fri, Dec 24, 2021 at 8:22 AM Darren Michael @.***> wrote:

Hi, Mike,

I know the original post is from a while ago, but I saw the same performance as described in the original post when I used the 'calculations=TRUE' setting.

I looked into your code and noticed that the post-processing of the JSON response seems to pull out only four of the columns returned in the response, i.e., year, period, periodName, and value. For the first data element in the returned list, the code assigns NULL to remove 'footnotes' and 'latest.'

Here's what I put together as a workaround for extracting net changes and percent changes.

#Global cleanup rm(list = ls(all.names = TRUE))

library(blsAPI) library(rjson)

my_api_key<-"YOUR_API_KEY_HERE"

#Date Range Parameters

startyear<-2005 endyear<-2021

#CES65000000001, Health and Education Services

payload_net<-list('seriesid'='CES6500000001','registrationKey'=my_api_key,'startyear'=startyear,'endyear'=endyear,'calculations'=TRUE,'annualaverage'=FALSE) health_and_ed_with_net<-blsAPI(payload_net,api_version=2,return_data_frame = FALSE) summary(health_and_ed_with_net) View(health_and_ed_with_net)

test<-fromJSON(health_and_ed_with_net) View(test) New Version of loop from original function

#Iterate over the series number_of_series <- length(test$Results$series) for (i in 1:number_of_series){ #Set the default structure of the data frame df_start <- data.frame(year = character(), period = character(), periodName = character(), value = character(), net_change_1 = character(), net_change_3 = character(), net_change_6 = character(), net_change_12 = character(), pct_change_1 = character(), pct_change_3 = character(), pct_change_6 = character(), pct_change_12 = character(), stringsAsFactors = FALSE) #Get the data series_data <- test$Results$series[[i]]$data #Can get no data after a successful request if (length(series_data) > 0) { j <- 0 #index at 0 to start for (d in series_data) { j <- j + 1 #advance by one #Remove the footnotes from the list to stop the warnings d$footnotes <- NULL d$latest <- NULL #Add record to the data frame df_start[j, ] <- unlist(d) } #Add in the series id df_start$seriesID <- test$Results$series[[i]]$seriesID } #Create the data frame that will be returned if (!exists("df_to_return")){ #Data frame to return not defined so create it df_to_return <- df_start } else { #Append to the existing data frame df_to_return <- rbind(df_to_return, df_start) } }

View(df_to_return)

Using this approach, I was able to pull out the remaining data in the list, i.e., the net change and percentage change values.

Here's what my test case produced:

[image: image] https://user-images.githubusercontent.com/45790848/147355030-8b214713-74c3-4d99-b195-44c0a74df943.png

It appears to match the lists returned in the JSON response.

[image: image] https://user-images.githubusercontent.com/45790848/147355159-3f5105e0-eb35-490a-b9ad-146fc98c1c19.png

Let us know if you've already implemented something similar within the package.

All the best, Darren

— Reply to this email directly, view it on GitHub https://github.com/mikeasilva/blsAPI/issues/26#issuecomment-1000857001, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJ4HD66U24VWX2PUUJHHN3USR627ANCNFSM4GPTVAEQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

[dmichael555]

Hi, Adhi, Thanks for the advice. I'll have at the blscrapeR package. All the best to both you and Mike. Regards,Darren

On Friday, December 24, 2021, 11:18:43 AM MST, Adhi Rajaprabhakaran ***@***.***> wrote:  

Darren, at this point i’d recommend you use the package “blscrapeR”. it’s far better maintained and convenient. no offense intended to mike but i think he’d agree if anything.

On Fri, Dec 24, 2021 at 8:22 AM Darren Michael @.***> wrote:

Hi, Mike,

I know the original post is from a while ago, but I saw the same performance as described in the original post when I used the 'calculations=TRUE' setting.

I looked into your code and noticed that the post-processing of the JSON response seems to pull out only four of the columns returned in the response, i.e., year, period, periodName, and value. For the first data element in the returned list, the code assigns NULL to remove 'footnotes' and 'latest.'

Here's what I put together as a workaround for extracting net changes and percent changes.

#Global cleanup rm(list = ls(all.names = TRUE))

library(blsAPI) library(rjson)

my_api_key<-"YOUR_API_KEY_HERE"

#Date Range Parameters

startyear<-2005 endyear<-2021

#CES65000000001, Health and Education Services

payload_net<-list('seriesid'='CES6500000001','registrationKey'=my_api_key,'startyear'=startyear,'endyear'=endyear,'calculations'=TRUE,'annualaverage'=FALSE) health_and_ed_with_net<-blsAPI(payload_net,api_version=2,return_data_frame = FALSE) summary(health_and_ed_with_net) View(health_and_ed_with_net)

test<-fromJSON(health_and_ed_with_net) View(test) New Version of loop from original function

#Iterate over the series number_of_series <- length(test$Results$series) for (i in 1:number_of_series){ #Set the default structure of the data frame df_start <- data.frame(year = character(), period = character(), periodName = character(), value = character(), net_change_1 = character(), net_change_3 = character(), net_change_6 = character(), net_change_12 = character(), pct_change_1 = character(), pct_change_3 = character(), pct_change_6 = character(), pct_change_12 = character(), stringsAsFactors = FALSE) #Get the data series_data <- test$Results$series[[i]]$data #Can get no data after a successful request if (length(series_data) > 0) { j <- 0 #index at 0 to start for (d in series_data) { j <- j + 1 #advance by one #Remove the footnotes from the list to stop the warnings d$footnotes <- NULL d$latest <- NULL #Add record to the data frame df_start[j, ] <- unlist(d) } #Add in the series id df_start$seriesID <- test$Results$series[[i]]$seriesID } #Create the data frame that will be returned if (!exists("df_to_return")){ #Data frame to return not defined so create it df_to_return <- df_start } else { #Append to the existing data frame df_to_return <- rbind(df_to_return, df_start) } }

View(df_to_return)

Using this approach, I was able to pull out the remaining data in the list, i.e., the net change and percentage change values.

Here's what my test case produced:

[image: image] https://user-images.githubusercontent.com/45790848/147355030-8b214713-74c3-4d99-b195-44c0a74df943.png

It appears to match the lists returned in the JSON response.

[image: image] https://user-images.githubusercontent.com/45790848/147355159-3f5105e0-eb35-490a-b9ad-146fc98c1c19.png

Let us know if you've already implemented something similar within the package.

All the best, Darren

— Reply to this email directly, view it on GitHub https://github.com/mikeasilva/blsAPI/issues/26#issuecomment-1000857001, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJ4HD66U24VWX2PUUJHHN3USR627ANCNFSM4GPTVAEQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: @.***>