ArduinoJson icon indicating copy to clipboard operation
ArduinoJson copied to clipboard
verified publisher icon bblanchon

Filter Not Working

Open sebas4055 opened this issue 3 weeks ago • 2 comments

Description I'm building an NFL team stats display using an ESP32 with TFT display. The device fetches game schedule data from ESPN's API (http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/schedule). The JSON response is approximately 100KB+ and causes ArduinoJson to fail with "InvalidInput" error even with 64KB buffer allocation. JSON filters also fail to work properly - even with proper filter syntax, the parsed document contains 0 events despite the raw JSON containing multiple game events. The goal is to extract team record, next game opponent, date/time, and venue information.

Troubleshooter's report

  1. The program uses ArduinoJson 7
  2. The issue happens at run time
  3. The issue concerns serialization
  4. Output is empty
  5. JsonDocument::overflowed() returns false

Environment

  • Microcontroller: ESP32-DevKitC-32
  • Core/Framework: esp32 by Espressif Systems v3.2.1
  • IDE: Arduino IDE 2.3.6

Reproduction code

#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>

void setup() {
  Serial.begin(115200);
  WiFi.begin("SSID", "PASSWORD");
  while (WiFi.status() != WL_CONNECTED) delay(500);
}

void loop() {
  HTTPClient http;
  http.begin("http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/schedule");
  
  if (http.GET() == 200) {
    // Create filter using .add<JsonObject>() syntax from ArduinoJson Assistant
    JsonDocument filter;
    filter["team"]["recordSummary"] = true;
    
    JsonObject filter_events_0 = filter["events"].add<JsonObject>();
    filter_events_0["date"] = true;
    filter_events_0["week"]["number"] = true;
    
    JsonObject filter_events_0_competitions_0 = filter_events_0["competitions"].add<JsonObject>();
    filter_events_0_competitions_0["status"]["type"]["state"] = true;
    
    JsonObject filter_events_0_competitions_0_competitors_0 = filter_events_0_competitions_0["competitors"].add<JsonObject>();
    filter_events_0_competitions_0_competitors_0["team"]["displayName"] = true;
    
    filter["byeWeek"] = true;
    
    // Parse with filter
    Stream& stream = http.getStream();
    JsonDocument doc;
    
    DeserializationError error = deserializeJson(doc, stream, DeserializationOption::Filter(filter));
    
    Serial.print("Error: ");
    Serial.println(error.c_str());
    Serial.print("Memory used: ");
    Serial.println(doc.memoryUsage());  // Shows 0
    Serial.print("Events found: ");
    Serial.println(doc["events"].size());  // Expected: 17, Actual: 0
  }
  
  http.end();
  delay(60000);
}

Remarks The raw JSON payload can be retrieved successfully and is valid JSON Other smaller API endpoints parse correctly with the same code Increasing buffer size beyond 65KB causes memory allocation failures This appears to be a common issue with large sports API responses on ESP32 Looking for suggestions on streaming JSON parsing or alternative approaches for handling large API responses with limited memory

full code below: #include <TFT_eSPI.h> #include "mode_1.h" #include "mode_2.h"

#include <WiFi.h> #include <HTTPClient.h> #include <ArduinoJson.h>

struct GameData { // Live game data String homeTeam; String awayTeam; int homeScore; int awayScore; String quarter; String timeRemaining; String gameStatus; bool isGameLive;

// Next game data String nextOpponent; String nextGameDay; String nextGameMonth; String nextGameTime; bool isHome;

// Season stats int wins; int losses; int byeWeek; float pointsPerGame; float yardsPerGame; float passYardsPerGame; float rushYardsPerGame; float pointsAllowedPerGame; float yardsAllowedPerGame; float passYardsAllowedPerGame; float rushYardsAllowedPerGame;

bool isBroncosGame; };

GameData currentGame; unsigned long lastUpdate = 0; const unsigned long updateInterval = 30000; // 30 seconds

const char* ssid = "wifi"; const char* password = "pswd"; const char* scheduleUrl = "http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/schedule"; const char* statsUrl = "http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/statistics";

TFT_eSPI tft = TFT_eSPI();

#define BTTN_PIN 13

int displayMode = 0; int lastState = HIGH; int currentState;

void connectWiFi() {

Serial.print("Connecting to WiFi"); WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); }

Serial.println("\nWiFi Connected!"); Serial.print("IP: "); Serial.println(WiFi.localIP()); }

void setup() { Serial.begin(115200); pinMode(BTTN_PIN, INPUT_PULLUP); tft.init(); tft.setRotation(1); tft.setSwapBytes(true); tft.fillScreen(TFT_BLACK); connectWiFi(); }

void loop() { currentState = digitalRead(BTTN_PIN);

if (currentState == LOW && lastState == HIGH) { displayMode = (displayMode + 1) % 3;

if (displayMode == 1) {
  // Mode 1: General stats and next game
  tft.pushImage(0, 0, 320, 240, mode_1);
  Serial.println("Mode 1: General Stats");
  fetchGeneralData();
  //debugPrintRawJson();
  //displayGeneralMode();
} else if (displayMode == 2) {
  // Mode 2: Live game
  tft.pushImage(0, 0, 320, 240, mode_2);
  Serial.println("Mode 2: Live Game");
  //fetchLiveGameData();
  displayLiveMode();
} else {
  tft.fillScreen(TFT_BLACK);
  Serial.println("Mode 0: Off");
}

Serial.println("button");

}

lastState = currentState; } void fetchGeneralData() { if (WiFi.status() != WL_CONNECTED) { Serial.println("WiFi disconnected"); return; }

Serial.println("\n========== FETCHING DATA ==========");

HTTPClient http; http.begin(scheduleUrl); http.setTimeout(20000);

int httpCode = http.GET(); Serial.printf("HTTP Response Code: %d\n", httpCode);

if (httpCode != 200) { Serial.println("❌ HTTP error"); http.end(); return; }

// --- 1) Build FILTER document (from ArduinoJson Assistant) --- // IMPORTANT: give it a capacity (e.g. 2048), NOT JsonDocument filter; JsonDocument filter;

filter["team"]["recordSummary"] = true; JsonObject filter_events_0 = filter["events"].add<JsonObject>(); filter_events_0["name"] = true; filter_events_0["shortName"] = true; filter_events_0["date"] = true; filter_events_0["seasonType"]["abbreviation"] = true; filter_events_0["week"]["number"] = true;

JsonObject filter_events_0_competitions_0 = filter_events_0["competitions"].add<JsonObject>(); filter_events_0_competitions_0["date"] = true; filter_events_0_competitions_0["venue"]["address"]["city"] = true;

JsonObject filter_events_0_competitions_0_competitors_0 = filter_events_0_competitions_0["competitors"].add<JsonObject>(); filter_events_0_competitions_0_competitors_0["order"] = true; filter_events_0_competitions_0_competitors_0["team"]["displayName"] = true; filter_events_0_competitions_0_competitors_0["score"]["value"] = true; filter_events_0_competitions_0_competitors_0["record"][0]["displayValue"] = true;

JsonObject filter_events_0_competitions_0_status = filter_events_0_competitions_0["status"].to<JsonObject>(); filter_events_0_competitions_0_status["clock"] = true; filter_events_0_competitions_0_status["displayClock"] = true; filter_events_0_competitions_0_status["period"] = true; filter_events_0_competitions_0_status["type"]["state"] = true; filter["byeWeek"] = true;

// (Optional) debug: show what the filter looks like Serial.println("Filter doc:"); serializeJsonPretty(filter, Serial); Serial.println();

// --- 2) Deserialize with filter from the HTTP stream --- WiFiClient &input = http.getStream();

JsonDocument doc; // result after filtering (much smaller than raw JSON)

DeserializationError error = deserializeJson( doc, input, DeserializationOption::Filter(filter), DeserializationOption::NestingLimit(30)); // bump nesting limit a bit

if (error) { Serial.print("deserializeJson() failed: "); Serial.println(error.c_str()); http.end(); return; }

// Debug: check memory usage + preview Serial.print("overflowed: "); Serial.println(doc.overflowed()); Serial.print("memoryUsage: "); Serial.println(doc.memoryUsage()); Serial.println("Filtered doc preview:"); serializeJsonPretty(doc, Serial); Serial.println();

Serial.println("✓ JSON parsed successfully!");

// --- 3) team.recordSummary -> wins/losses --- const char* team_recordSummary = doc["team"]["recordSummary"]; // "9-2" if (team_recordSummary) { int w = 0, l = 0; if (sscanf(team_recordSummary, "%d-%d", &w, &l) == 2) { currentGame.wins = w; currentGame.losses = l; Serial.printf("✓ Parsed record: %d-%d\n", w, l); } else { Serial.printf("⚠ Could not parse recordSummary: %s\n", team_recordSummary); } } else { Serial.println("⚠ recordSummary missing"); }

// --- 4) Access events[] (you can later add your next-game logic here) --- JsonArray events = doc["events"].as<JsonArray>(); if (events.isNull() || events.size() == 0) { Serial.println("⚠ No events in schedule"); http.end(); return; }

Serial.printf("Events in filtered doc: %u\n", events.size());

// Example: print basic info for first event JsonObject e0 = events[0]; const char* e0_date = e0["date"] | ""; const char* e0_name = e0["name"] | ""; const char* e0_shortName = e0["shortName"] | ""; int e0_week = e0["week"]["number"] | 0;

Serial.printf("Week %d: %s (%s) on %s\n", e0_week, e0_name, e0_shortName, e0_date);

JsonObject comp0 = e0["competitions"][0]; const char* city = comp0["venue"]["address"]["city"] | ""; Serial.printf(" Venue city: %s\n", city);

JsonArray competitors = comp0["competitors"].as<JsonArray>(); for (JsonObject c : competitors) { int order = c["order"] | -1; const char* teamName = c["team"]["displayName"] | ""; int score = c["score"]["value"] | -1;

Serial.printf("  [%d] %s score=%d\n", order, teamName, score);

}

int byeWeek = doc["byeWeek"] | -1; if (byeWeek > 0) { currentGame.byeWeek = byeWeek; Serial.printf("Bye week: %d\n", byeWeek); }

http.end(); }

void displayGeneralMode() { Serial.println("\n========== DISPLAY: GENERAL MODE =========="); Serial.printf("Record: %d-%d\n", currentGame.wins, currentGame.losses); Serial.printf("Next Opponent: %s (%s)\n", currentGame.nextOpponent.c_str(), currentGame.isHome ? "HOME" : "AWAY"); Serial.printf("Points Per Game: %.1f\n", currentGame.pointsPerGame); Serial.printf("Yards Per Game: %.1f\n", currentGame.yardsPerGame); Serial.printf("Pass Yards Per Game: %.1f\n", currentGame.passYardsPerGame); Serial.printf("Rush Yards Per Game: %.1f\n", currentGame.rushYardsPerGame); Serial.println("===========================================\n"); }

void displayLiveMode() { Serial.println("\n========== DISPLAY: LIVE GAME MODE ==========");

if (currentGame.isGameLive) { Serial.printf("%s vs %s\n", currentGame.homeTeam.c_str(), currentGame.awayTeam.c_str()); Serial.printf("Score: %d - %d\n", currentGame.homeScore, currentGame.awayScore); Serial.printf("Quarter: %s\n", currentGame.quarter.c_str()); Serial.printf("Time Remaining: %s\n", currentGame.timeRemaining.c_str()); } else { Serial.println("No live game currently"); }

Serial.println("=============================================\n"); }

/* // Display functions - you'll implement these based on your UI void displayGeneralMode() { // Draw the general stats on screen tft.setTextColor(TFT_WHITE, TFT_NAVY); tft.setTextSize(2);

// Example: tft.setCursor(10, 10); tft.printf("Record: %d-%d", currentGame.wins, currentGame.losses);

tft.setCursor(10, 40); tft.printf("Next: vs %s", currentGame.nextOpponent.c_str());

tft.setCursor(10, 70); tft.printf("PPG: %.1f", currentGame.pointsPerGame);

// Add more display code here }

void displayLiveMode() { // Draw the live game info on screen if (currentGame.isGameLive) { tft.setTextColor(TFT_WHITE, TFT_NAVY); tft.setTextSize(3);

tft.setCursor(50, 80);
tft.printf("%d - %d", currentGame.homeScore, currentGame.awayScore);

tft.setTextSize(2);
tft.setCursor(80, 120);
tft.printf("Q%s %s", currentGame.quarter.c_str(), currentGame.timeRemaining.c_str());

} else { tft.setTextSize(2); tft.setCursor(60, 100); tft.print("No live game"); } }*/

sebas4055 avatar Nov 23 '25 01:11 sebas4055

Hi @sebas4055,

causes ArduinoJson to fail with "InvalidInput" error

This radically differs from the answers you gave to the Troubleshooter. Please give it another chance and try the most relevant options.

Best regards, Benoit

bblanchon avatar Nov 26 '25 10:11 bblanchon

Hi Benoit, Thanks for replying and sorry for the confusion in my initial report!

Let me try and explain the situation better:

When I deserializeJson(doc, stream) without a filter it fails with InvalidInput error because the raw JSON response is ~100KB+, which exceeds available memory

When I try it with the filter, deserializeJson(doc, stream, DeserializationOption::Filter(filter)) succeeds (no error reported) However, the resulting document is completely empty, it outputs:

doc.memoryUsage() returns 0 doc["events"].size() returns 0 doc["team"]["recordSummary"] is null All fields are missing despite being included in the filter and also doc.overflowed() returns false

I've tried multiple versions of the filter, First I used the one generated using the ArduinoJson Assistant-generated filter. This one used .add<JsonObject>() syntax for arrays Example: JsonObject filter_events_0 = filter["events"].add<JsonObject>(); When printed with serializeJsonPretty(filter, Serial), the filter structure looks correct and matches the actual JSON schema and seemed to work on the assistant website but still gave me an empty doc.

Then I tried an alternative array filter syntax. I change to: filter["events"][0].to<JsonObject>() notation (based on some examples suggesting this creates array templates) Also tried filter["events"].createNestedObject() All variations produce the same empty document

Environment:

ESP32-DevKitC (ESP8266 module) ArduinoJson 7.4.0 Also tried Stream& stream = http.getStream() and http.getString() with same results

The core issue is filtering appears to work, but produces a completely empty document even though the filter structure is correct.

Is there a known issue with filtered deserialization from HTTP streams in v7.4, or am I missing something about how filters should be structured for streaming deserialization? Happy to provide the full filter structure or raw JSON sample if that helps!

Thanks, Sebastian

sebas4055 avatar Dec 04 '25 23:12 sebas4055

it fails with InvalidInput error because the raw JSON response is ~100KB+, which exceeds available memory

I stopped reading there. InvalidInput means the input is invalid, not too big.

Can you please give the ArduinoJson Troubleshooter a try?

bblanchon avatar Dec 10 '25 08:12 bblanchon