redlib icon indicating copy to clipboard operation
redlib copied to clipboard
verified publisher icon redlib-org

🐛 Bug Report: Redlib becomes unresponsive/times out momentarily

Open wuchyi opened this issue 3 months ago • 6 comments

Describe the bug

Redlib instance (official docker container deployment) will become unresponsive after browsing for a while. No error codes logged when it happens, although a look at the log file shows many lines of ERROR redlib::utils > Error page rendered: Nothing here (see screenshot below).

Usually happens after browsing a few pages (via the Next button), or when navigating to specific subreddits, but it's been happening in all sorts of situations, including when clicking to view a specific post from the home page.

Cloudflare will sometimes show its Error 524 page.

Other docker containers still run as usual and remain contactable, so it shouldn't be a server issue.

Usually resolves itself by retrying/reloading after a few minutes.

Steps to reproduce the bug

  1. Browse as usual,
  2. click to view specific posts/next page
  3. Redlib instance becomes unresponsive/times out

Additional context / screenshot

Image
  • [ X] I checked that the instance that this was reported on is running the latest git commit, or I can reproduce it locally on the latest git commit

wuchyi avatar Sep 11 '25 11:09 wuchyi

Starting Redlib... Running Redlib v0.36.0 on [::]:8980! ERROR redlib::client > Got an invalid response from reddit expected value at line 1 column 1. Status code: 403 Forbidden ERROR redlib::utils > Error page rendered: Failed to parse page JSON data: expected value at line 1 column 1

lethallybad avatar Sep 12 '25 03:09 lethallybad

To be clear: this was happening before today's wave of rate limit issues.

wuchyi avatar Sep 12 '25 05:09 wuchyi

Starting Redlib... Running Redlib v0.36.0 on [::]:8980! ERROR redlib::client > Got an invalid response from reddit expected value at line 1 column 1. Status code: 403 Forbidden ERROR redlib::utils > Error page rendered: Failed to parse page JSON data: expected value at line 1 column 1

I'm experiencing this exact issue aswell since today. This is after the previews didn't really load for video's anymore.

It seems to go away after i redeploy the container, but after a few minutes, it appears again.

gjgradem avatar Sep 12 '25 07:09 gjgradem

I believe I've fixed it by updating the user-agents used.

diff --git a/scripts/update_oauth_resources.sh b/scripts/update_oauth_resources.sh
index 7eeb959..e244de2 100755
--- a/scripts/update_oauth_resources.sh
+++ b/scripts/update_oauth_resources.sh
@@ -5,6 +5,18 @@
 # - rg
 # - jq
 
+# User agents for APKMirror requests - AI assistants and search engines
+USER_AGENTS=(
+    "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
+    "ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])"
+    "Wayback Machine Bot/1.0"
+)
+
+# Function to get a random user agent
+get_random_user_agent() {
+    echo "${USER_AGENTS[$RANDOM % ${#USER_AGENTS[@]}]}"
+}
+
 # Fetch iOS app versions
 ios_version_list=$(curl -s "https://ipaarchive.com/app/usa/1064216828" | rg "(20\d{2}\.\d+.\d+) / (\d+)" --only-matching -r "Version \$1/Build \$2" | sort | uniq)
 
@@ -38,13 +50,11 @@ done
 # Close the array in the source file
 echo "];" >> "$filename"
 
-# Fetch Android app versions
-page_1=$(curl -s "https://apkcombo.com/reddit/com.reddit.frontpage/old-versions/" | rg "<a class=\"ver-item\" href=\"(/reddit/com\.reddit\.frontpage/download/phone-20\d{2}\.\d+\.\d+-apk)\" rel=\"nofollow\">" -r "https://apkcombo.com\$1" | sort | uniq | sed 's/      //g')
-# Append with pages
-page_2=$(curl -s "https://apkcombo.com/reddit/com.reddit.frontpage/old-versions?page=2" | rg "<a class=\"ver-item\" href=\"(/reddit/com\.reddit\.frontpage/download/phone-20\d{2}\.\d+\.\d+-apk)\" rel=\"nofollow\">" -r "https://apkcombo.com\$1" | sort | uniq | sed 's/      //g')
-page_3=$(curl -s "https://apkcombo.com/reddit/com.reddit.frontpage/old-versions?page=3" | rg "<a class=\"ver-item\" href=\"(/reddit/com\.reddit\.frontpage/download/phone-20\d{2}\.\d+\.\d+-apk)\" rel=\"nofollow\">" -r "https://apkcombo.com\$1" | sort | uniq | sed 's/      //g')
-page_4=$(curl -s "https://apkcombo.com/reddit/com.reddit.frontpage/old-versions?page=4" | rg "<a class=\"ver-item\" href=\"(/reddit/com\.reddit\.frontpage/download/phone-20\d{2}\.\d+\.\d+-apk)\" rel=\"nofollow\">" -r "https://apkcombo.com\$1" | sort | uniq | sed 's/      //g')
-page_5=$(curl -s "https://apkcombo.com/reddit/com.reddit.frontpage/old-versions?page=5" | rg "<a class=\"ver-item\" href=\"(/reddit/com\.reddit\.frontpage/download/phone-20\d{2}\.\d+\.\d+-apk)\" rel=\"nofollow\">" -r "https://apkcombo.com\$1" | sort | uniq | sed 's/      //g')
+# Fetch Android app versions from APKMirror
+page_1=$(curl -s -H "User-Agent: $(get_random_user_agent)" "https://www.apkmirror.com/uploads/?devcategory=redditinc" | rg 'href="(/apk/redditinc/reddit/reddit-[0-9]+-[0-9]+-[0-9]+-release/)"' -o -r "https://www.apkmirror.com\$1" | sort -u)
+page_2=$(curl -s -H "User-Agent: $(get_random_user_agent)" "https://www.apkmirror.com/uploads/page/2/?devcategory=redditinc" | rg 'href="(/apk/redditinc/reddit/reddit-[0-9]+-[0-9]+-[0-9]+-release/)"' -o -r "https://www.apkmirror.com\$1" | sort -u)
+page_3=$(curl -s -H "User-Agent: $(get_random_user_agent)" "https://www.apkmirror.com/uploads/page/3/?devcategory=redditinc" | rg 'href="(/apk/redditinc/reddit/reddit-[0-9]+-[0-9]+-[0-9]+-release/)"' -o -r "https://www.apkmirror.com\$1" | sort -u)
+page_4=$(curl -s -H "User-Agent: $(get_random_user_agent)" "https://www.apkmirror.com/uploads/page/4/?devcategory=redditinc" | rg 'href="(/apk/redditinc/reddit/reddit-[0-9]+-[0-9]+-[0-9]+-release/)"' -o -r "https://www.apkmirror.com\$1" | sort -u)
 
 # Concatenate all pages
 versions="${page_1}"
@@ -55,7 +65,6 @@ versions+="${page_3}"
 versions+=$'\n'
 versions+="${page_4}"
 versions+=$'\n'
-versions+="${page_5}"
 
 # Count the number of lines in the version list
 android_count=$(echo "$versions" | wc -l)
@@ -70,11 +79,14 @@ num=0
 # For each in versions, curl the page and extract the build number
 echo "$versions" | while IFS= read -r line; do
   num=$((num+1))
-  fetch_page=$(curl -s "$line")
-  build=$(echo "$fetch_page" | rg "<span class=\"vercode\">\((\d+)\)</span>" --only-matching -r "\$1" | head -n1)
-  version=$(echo "$fetch_page" | rg "<span class=\"vername\">Reddit (20\d{2}\.\d+\.\d+)</span>" --only-matching -r "\$1" | head -n1)
+
+  fetch_page=$(curl -s -H "User-Agent: $(get_random_user_agent)" "$line")
+  build=$(echo "$fetch_page" | rg 'class="colorLightBlack">([0-9]+)</span>' -o -r '$1' | head -n1)
+  # Extract version from URL instead of HTML to get the correct version for this specific release
+  version=$(echo "$line" | rg 'reddit-([0-9]+-[0-9]+-[0-9]+)-release' -o -r '$1' | sed 's/-/./g')
   echo "	\"Version $version/Build $build\"," >> "$filename"
   echo -e "[$num/$android_count] Fetched \e[32mVersion $version/Build $build\e[0m."
+  sleep 30
 done
 
 # Close the array in the source file
diff --git a/src/oauth_resources.rs b/src/oauth_resources.rs
index 01928c3..41a64cd 100644
--- a/src/oauth_resources.rs
+++ b/src/oauth_resources.rs
@@ -2,10 +2,76 @@
 // Rerun scripts/update_oauth_resources.sh to update this file
 // Please do not edit manually
 // Filled in with real app versions
-pub const _IOS_APP_VERSION_LIST: &[&str; 1] = &[""];
-pub const ANDROID_APP_VERSION_LIST: &[&str; 150] = &[
+pub const _IOS_APP_VERSION_LIST: &[&str; 1] = &[
+	"",
+];
+pub const ANDROID_APP_VERSION_LIST: &[&str;  100] = &[
+	"Version 2025.12.0/Build 2512001",
+	"Version 2025.12.1/Build 2512140",
+	"Version 2025.13.0/Build 2513001",
+	"Version 2025.14.0/Build 2514001",
+	"Version 2025.15.0/Build 2515001",
+	"Version 2025.16.0/Build 2516001",
+	"Version 2025.17.0/Build 2517030",
+	"Version 2025.18.0/Build 2518001",
+	"Version 2025.18.1/Build 2518060",
+	"Version 2025.19.0/Build 2519001",
+	"Version 2025.20.0/Build 2520001",
+	"Version 2025.21.0/Build 2521070",
+	"Version 2025.22.0/Build 2522011",
+	"Version 2025.23.0/Build 2523001",
+	"Version 2025.23.1/Build 2523110",
+	"Version 2025.24.0/Build 2524020",
+	"Version 2025.25.0/Build 2525001",
+	"Version 2025.26.0/Build 2526031",
+	"Version 2025.27.0/Build 2527001",
+	"Version 2025.28.0/Build 2528001",
+	"Version 2025.28.1/Build 2528090",
+	"Version 2025.29.0/Build 2529021",
+	"Version 2025.30.0/Build 2530051",
+	"Version 2025.31.0/Build 2531041",
+	"Version 2025.31.1/Build 2531100",
+	"Version 2025.32.0/Build 2532001",
+	"Version 2025.33.0/Build 2533071",
+	"Version 2025.34.0/Build 2534021",
+	"Version 2025.35.0/Build 2535061",
+	"Version 2025.36.0/Build 2536011",
+	"Version 2024.43.0/Build 1969283",
+	"Version 2024.44.0/Build 1980285",
+	"Version 2024.45.0/Build 1994090",
+	"Version 2024.46.0/Build 2007572",
+	"Version 2024.47.0/Build 2021239",
+	"Version 2024.47.1/Build 2035590",
+	"Version 2024.48.0/Build 2040898",
+	"Version 2024.49.0/Build 2051679",
+	"Version 2024.49.1/Build 2061535",
+	"Version 2024.50.0/Build 2065135",
+	"Version 2024.50.1/Build 2127991",
+	"Version 2024.51.0/Build 2081615",
+	"Version 2024.52.0/Build 2099715",
+	"Version 2024.53.0/Build 2110341",
+	"Version 2025.01.0/Build 2118801",
+	"Version 2025.02.0/Build 2138572",
+	"Version 2025.03.0/Build 2157131",
+	"Version 2025.03.1/Build 2181077",
+	"Version 2025.04.0/Build 2181092",
+	"Version 2025.05.0/Build 2194266",
+	"Version 2025.05.1/Build 2214840",
+	"Version 2025.06.0/Build 2221631",
+	"Version 2025.07.0/Build 2507001",
+	"Version 2025.07.1/Build 2507130",
+	"Version 2025.07.2/Build 2507170",
+	"Version 2025.08.0/Build 2508011",
+	"Version 2025.09.0/Build 2509021",
+	"Version 2025.09.1/Build 2509040",
+	"Version 2025.10.0/Build 2510001",
+	"Version 2025.11.0/Build 2511120",
+	"Version 2024.20.3/Build 1624970",
+	"Version 2024.22.0/Build 1645257",
 	"Version 2024.22.1/Build 1652272",
+	"Version 2024.23.0/Build 1660290",
 	"Version 2024.23.1/Build 1665606",
+	"Version 2024.24.0/Build 1675730",
 	"Version 2024.24.1/Build 1682520",
 	"Version 2024.25.0/Build 1693595",
 	"Version 2024.25.2/Build 1700401",
@@ -14,26 +80,24 @@ pub const ANDROID_APP_VERSION_LIST: &[&str; 150] = &[
 	"Version 2024.26.1/Build 1717435",
 	"Version 2024.28.0/Build 1737665",
 	"Version 2024.28.1/Build 1741165",
+	"Version 2024.29.0/Build 1747562",
 	"Version 2024.30.0/Build 1770787",
-	"Version 2024.31.0/Build 1786202",
-	"Version 2024.32.0/Build 1809095",
+	"Version 2024.31.0/Build 1780188",
+	"Version 2024.32.0/Build 1801227",
 	"Version 2024.32.1/Build 1813258",
-	"Version 2024.33.0/Build 1819908",
-	"Version 2024.34.0/Build 1837909",
-	"Version 2024.35.0/Build 1861437",
-	"Version 2024.36.0/Build 1875012",
-	"Version 2024.37.0/Build 1888053",
-	"Version 2024.38.0/Build 1902791",
+	"Version 2024.33.0/Build 1819905",
+	"Version 2024.34.0/Build 1831627",
+	"Version 2024.35.0/Build 1857451",
+	"Version 2024.36.0/Build 1868844",
+	"Version 2024.37.0/Build 1883040",
+	"Version 2024.38.0/Build 1898570",
 	"Version 2024.39.0/Build 1916713",
-	"Version 2024.40.0/Build 1928580",
-	"Version 2024.41.0/Build 1941199",
+	"Version 2024.40.0/Build 1924121",
+	"Version 2024.41.0/Build 1936392",
 	"Version 2024.41.1/Build 1947805",
-	"Version 2024.42.0/Build 1952440",
-	"Version 2024.43.0/Build 1972250",
-	"Version 2024.44.0/Build 1988458",
-	"Version 2024.45.0/Build 2001943",
-	"Version 2024.46.0/Build 2012731",
-	"Version 2024.47.0/Build 2029755",
+	"Version 2024.42.0/Build 1948102",
+	"Version 2023.45.0/Build 1281371",
+	"Version 2023.47.0/Build 1303604",
 	"Version 2023.48.0/Build 1319123",
 	"Version 2023.49.0/Build 1321715",
 	"Version 2023.49.1/Build 1322281",
@@ -42,117 +106,7 @@ pub const ANDROID_APP_VERSION_LIST: &[&str; 150] = &[
 	"Version 2024.02.0/Build 1368985",
 	"Version 2024.03.0/Build 1379408",
 	"Version 2024.04.0/Build 1391236",
-	"Version 2024.05.0/Build 1403584",
-	"Version 2024.06.0/Build 1418489",
-	"Version 2024.07.0/Build 1429651",
-	"Version 2024.08.0/Build 1439531",
-	"Version 2024.10.0/Build 1470045",
-	"Version 2024.10.1/Build 1478645",
-	"Version 2024.11.0/Build 1480707",
-	"Version 2024.12.0/Build 1494694",
-	"Version 2024.13.0/Build 1505187",
-	"Version 2024.14.0/Build 1520556",
-	"Version 2024.15.0/Build 1536823",
-	"Version 2024.16.0/Build 1551366",
-	"Version 2024.17.0/Build 1568106",
-	"Version 2024.18.0/Build 1577901",
-	"Version 2024.18.1/Build 1585304",
-	"Version 2024.19.0/Build 1593346",
-	"Version 2024.20.0/Build 1612800",
-	"Version 2024.20.1/Build 1615586",
-	"Version 2024.20.2/Build 1624969",
-	"Version 2024.20.3/Build 1624970",
-	"Version 2024.21.0/Build 1631686",
-	"Version 2024.22.0/Build 1645257",
-	"Version 2023.21.0/Build 956283",
-	"Version 2023.22.0/Build 968223",
-	"Version 2023.23.0/Build 983896",
-	"Version 2023.24.0/Build 998541",
-	"Version 2023.25.0/Build 1014750",
-	"Version 2023.25.1/Build 1018737",
-	"Version 2023.26.0/Build 1019073",
-	"Version 2023.27.0/Build 1031923",
-	"Version 2023.28.0/Build 1046887",
-	"Version 2023.29.0/Build 1059855",
-	"Version 2023.30.0/Build 1078734",
-	"Version 2023.31.0/Build 1091027",
-	"Version 2023.32.0/Build 1109919",
-	"Version 2023.32.1/Build 1114141",
-	"Version 2023.33.1/Build 1129741",
-	"Version 2023.34.0/Build 1144243",
-	"Version 2023.35.0/Build 1157967",
-	"Version 2023.36.0/Build 1168982",
-	"Version 2023.37.0/Build 1182743",
-	"Version 2023.38.0/Build 1198522",
-	"Version 2023.39.0/Build 1211607",
-	"Version 2023.39.1/Build 1221505",
-	"Version 2023.40.0/Build 1221521",
-	"Version 2023.41.0/Build 1233125",
-	"Version 2023.41.1/Build 1239615",
-	"Version 2023.42.0/Build 1245088",
-	"Version 2023.43.0/Build 1257426",
-	"Version 2023.44.0/Build 1268622",
-	"Version 2023.45.0/Build 1281371",
-	"Version 2023.47.0/Build 1303604",
-	"Version 2022.42.0/Build 638508",
-	"Version 2022.43.0/Build 648277",
-	"Version 2022.44.0/Build 664348",
-	"Version 2022.45.0/Build 677985",
-	"Version 2023.01.0/Build 709875",
-	"Version 2023.02.0/Build 717912",
-	"Version 2023.03.0/Build 729220",
-	"Version 2023.04.0/Build 744681",
-	"Version 2023.05.0/Build 755453",
-	"Version 2023.06.0/Build 775017",
-	"Version 2023.07.0/Build 788827",
-	"Version 2023.07.1/Build 790267",
-	"Version 2023.08.0/Build 798718",
-	"Version 2023.09.0/Build 812015",
-	"Version 2023.09.1/Build 816833",
-	"Version 2023.10.0/Build 821148",
-	"Version 2023.11.0/Build 830610",
-	"Version 2023.12.0/Build 841150",
-	"Version 2023.13.0/Build 852246",
-	"Version 2023.14.0/Build 861593",
-	"Version 2023.14.1/Build 864826",
-	"Version 2023.15.0/Build 870628",
-	"Version 2023.16.0/Build 883294",
-	"Version 2023.16.1/Build 886269",
-	"Version 2023.17.0/Build 896030",
-	"Version 2023.17.1/Build 900542",
-	"Version 2023.18.0/Build 911877",
-	"Version 2023.19.0/Build 927681",
-	"Version 2023.20.0/Build 943980",
-	"Version 2023.20.1/Build 946732",
-	"Version 2022.20.0/Build 487703",
-	"Version 2022.21.0/Build 492436",
-	"Version 2022.22.0/Build 498700",
-	"Version 2022.23.0/Build 502374",
-	"Version 2022.23.1/Build 506606",
-	"Version 2022.24.0/Build 510950",
-	"Version 2022.24.1/Build 513462",
-	"Version 2022.25.0/Build 515072",
-	"Version 2022.25.1/Build 516394",
-	"Version 2022.25.2/Build 519915",
-	"Version 2022.26.0/Build 521193",
-	"Version 2022.27.0/Build 527406",
-	"Version 2022.27.1/Build 529687",
-	"Version 2022.28.0/Build 533235",
-	"Version 2022.30.0/Build 548620",
-	"Version 2022.31.0/Build 556666",
-	"Version 2022.31.1/Build 562612",
-	"Version 2022.32.0/Build 567875",
-	"Version 2022.33.0/Build 572600",
-	"Version 2022.34.0/Build 579352",
-	"Version 2022.35.0/Build 588016",
-	"Version 2022.35.1/Build 589034",
-	"Version 2022.36.0/Build 593102",
-	"Version 2022.37.0/Build 601691",
-	"Version 2022.38.0/Build 607460",
-	"Version 2022.39.0/Build 615385",
-	"Version 2022.39.1/Build 619019",
-	"Version 2022.40.0/Build 624782",
-	"Version 2022.41.0/Build 630468",
-	"Version 2022.41.1/Build 634168",
+	];
+pub const _IOS_OS_VERSION_LIST: &[&str; 1] = &[
+	"",
 ];
-pub const _IOS_OS_VERSION_LIST: &[&str; 1] = &[""];

I think there were 153 user agents so I've reduced it to 100. But overall after 10ish minutes browsing it seems to work. I would recommend testing some more since I am unable to reproduce the original 403 error, as well as start redlib in debug mode (the documentation is very confusing).

Shagon94 avatar Sep 12 '25 07:09 Shagon94

Can you give me a small guide on how to implement this in my instance?

gjgradem avatar Sep 12 '25 08:09 gjgradem

The user-agent is not the fix, I think some cookie needs to be refreshed. I've given up on redlib as reddit wants as much data they can get their hands on, so I've decided to just not use it at all.

Shagon94 avatar Sep 13 '25 07:09 Shagon94