chronon icon indicating copy to clipboard operation
chronon copied to clipboard

[FetcherCLI] Avoid throwing NPE for non-existent keys

Open vamseeyarla opened this issue 2 years ago • 0 comments

Currently run.py fetch throws a NPE error if the key is not existent. So we change this by returning an "Empty" string response if the result is null.

Test:

vamsee_y@i-0deb91c9026f8c16b:~/zipline cdh-gateway-silver-production $ python3 ~/.local/bin/run.py --mode=fetch -k '{"user_id":86820213}' -n payments/user.rya_history.v1_batch -t group-by --chronon-jar /home/vamsee_y/spark_uber-assembly-0.0.16-SNAPSHOT.jar 
Setting CHRONON_ONLINE_ARGS=-Zmussel-host=mussel-ml-dispatcher-http.synapse -Zmussel-port=6252
Setting CHRONON_ONLINE_CLASS=com.airbnb.bighead.zipline.online.MusselZiplineOnlineImpl
Setting CHRONON_LOG_TABLE=zipline.ml_infra_afp_logging_repartition_v1
Setting CHRONON_SCHEMA_TABLE=zipline.ml_infra_afp_logging_schema_v1
Setting CHRONON_MAVEN_MIRROR_PREFIX=https://artifactory.d.musta.ch/artifactory/maven-mirror
Downloading online_jar
Running command: ./scripts/fetch_online_jar.py
Downloaded jar to /tmp/airbnb_chronon_online_jar_e8fa99ac7e.jar
Running command: java -cp /home/vamsee_y/spark_uber-assembly-0.0.16-SNAPSHOT.jar ai.chronon.spark.Driver fetch --online-jar=/tmp/airbnb_chronon_online_jar_e8fa99ac7e.jar --online-class=com.airbnb.bighead.zipline.online.MusselZiplineOnlineImpl  -k {"user_id":86820213} -n payments/user.rya_history.v1_batch -t group-by -Zmussel-host=mussel-ml-dispatcher-http.synapse -Zmussel-port=6252
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
--- [START FETCHING for Map(user_id -> 86820213)] ---
Fetched group_by_serving_info from : PAYMENTS_USER_RYA_HISTORY_V1_BATCH_BATCH
Success({"groupBy":{"metaData":{"name":"payments.user.rya_history.v1_batch","online":1,"production":1,"customJson":"{\"team_override\": \"transaction_risk\", \"lag\": 0}","dependencies":["{\"name\": \"wait_for_jitney.sessionactivities_sessionactivity_v2_ds\", \"spec\": \"jitney.sessionactivities_sessionactivity_v2/ds={{ ds }}\", \"start\": \"2022-05-04\", \"end\": null}"],"tableProperties":{"abb_retention_config_json":"{\"policy\": \"delete_by_last_modified\", \"days\": 1460, \"reason\": \"Zipline default production table\"}"},"outputNamespace":"payments_ml","team":"payments"},"sources":[{"events":{"table":"jitney.sessionactivities_sessionactivity_v2","query":{"selects":{"user_id":"user_id","activity_type":"activity_type","was_rya":"1"},"wheres":["user_id is not null","activity_type='rya_started'"],"startPartition":"2022-05-04","timeColumn":"created_at","setups":[]},"isCumulative":0}}],"keyColumns":["user_id"],"aggregations":[{"inputColumn":"was_rya","operation":1,"argMap":{},"windows":[{"length":180,"timeUnit":1}]},{"inputColumn":"was_rya","operation":7,"argMap":{},"windows":[{"length":180,"timeUnit":1}]}],"accuracy":1},"selectedAvroSchema":"{\n  \"type\" : \"record\",\n  \"name\" : \"Value\",\n  \"namespace\" : \"ai.chronon.data\",\n  \"doc\" : \"\",\n  \"fields\" : [ {\n    \"name\" : \"was_rya\",\n    \"type\" : [ \"null\", \"int\" ],\n    \"doc\" : \"\"\n  } ]\n}","keyAvroSchema":"{\n  \"type\" : \"record\",\n  \"name\" : \"Key\",\n  \"namespace\" : \"ai.chronon.data\",\n  \"doc\" : \"\",\n  \"fields\" : [ {\n    \"name\" : \"user_id\",\n    \"type\" : [ \"null\", \"long\" ],\n    \"doc\" : \"\"\n  } ]\n}","batchEndDate":"2022-11-15"})
Building new stats cache for Context(group_by.fetch,null,payments_user_rya_history_v1_batch,null,true,SNAPSHOT,payments,null,null,group_by)
Building new stats cache for Context(group_by.fetch,null,payments_user_rya_history_v1_batch,null,true,SNAPSHOT,payments,null,null,null)
Batch End: 2022-11-15 00:00:00
Window Tails: 
  was_rya_max_180d -> Some(2022-05-19 00:00:00)
  was_rya_sum_180d -> Some(2022-05-19 00:00:00)
--- [FETCHED RESULT] ---
Empty
Fetched in: 599.373453 ms

vamseeyarla avatar Nov 16 '22 00:11 vamseeyarla