AutoFetcher-IG-Stories-to-GDrive
AutoFetcher-IG-Stories-to-GDrive copied to clipboard
chriskyfung
IG auto logged me out
I have two errors with autofetcher. First when I want run script I get this error: SyntaxError: Unexpected token < in JSON at position 0
I changed data in Logs section but after this I get another error Exception: Attribute provided with invalid value: Header:x-ig-www-claim
How I can fix this?
The error SyntaxError: Unexpected token < in JSON at position 0 raised because it failed to parse response from Instagram API. It may due to pass a wrong Instagram user ID.
Regarding to Exception: Attribute provided with invalid value: Header:x-ig-www-claim, please go to Issue #41
The error
SyntaxError: Unexpected token < in JSON at position 0raised because it failed to parse response from Instagram API. It may due to pass a wrong Instagram user ID.
No, id is good. I started use your project from april and since 26 July starts show this error. Before everything was good.
Regarding to
Exception: Attribute provided with invalid value: Header:x-ig-www-claim, please go to Issue #41
Yes, I read this. But I don't know how and where change this settings in your code. Can you write step by step how fix this error?
The error
SyntaxError: Unexpected token < in JSON at position 0raised because it failed to parse response from Instagram API. It may due to pass a wrong Instagram user ID.
I copy new version google sheet and copy all data in setting. Run works but when I log out from instagram this error show again and not work. After log in again in instagram x-ig-www-claim and cookie changed.
The error
SyntaxError: Unexpected token < in JSON at position 0raised because it failed to parse response from Instagram API. It may due to pass a wrong Instagram user ID.I copy new version google sheet and copy all data in setting. Run works but when I log out from instagram this error show again and not work. After log in again in instagram x-ig-www-claim and cookie changed.
You should never log out from Instagram because the script use the cookie of that log-in session.
Another error
Exception: Request failed for https://i.instagram.com returned code 400. Truncated server response: {"message":"checkpoint_required","checkpoint_url":"https://i.instagram.com/challenge/?next=/api/v1/feed/reels_media/...",... (use muteHttpExceptions option to examine full response)
I check and Instagram block my account. I try with another one, I don't log out. I hope this time it works.
Ok, I tested script and it works but after 2 days instagram block my accout, I must authorized. After authorized is error "SyntaxError: Unexpected token < in JSON at position 0" and I must copy new data to settings. Any suggestion how fix it? I want copy data once not new after 2 days.
Maybe I must change trigger? Now I have set run after every 1 hour. Maybe try set after longer interval?
Even when autorize error don't appear on Instagram I must copy new cookies date afer 2-3 days, because "SyntaxError: Unexpected token < in JSON at position 0" appear when I run fetcher. Any suggestion how fix this? I dont log out from IG account. I open Firefox - log in to IG - and close Firefox. I dont use Firefox to check sites, I using Chrome.
@fifson1 Why don't you copy your cookies from Chrome 😅
I don't want be log in on Chrome, but if this resolve error I could try And how should work testPipeline? When I run testPipeline then give me info: "Number of downloadable media files from @bbcnews: 16" but problem is I don't have @bbcnews in my sheet.
I think IG auto log out after 3 days and when I must log in again cookies is changed and new cookies must be copy to sheet because show error "SyntaxError: Unexpected token < in JSON at position 0", now I try with Chrome, maybe will be better effect
Still same problems with this ;/ IG logged me out after 3-4 days and I must log in and copy new cookies again and again. Maybe is something bad with my chrome settings? I have no idea what fix this...
Instagram enhanced their endpoint security recently. It seems the Instagram team is going to prevent robots to access their API just using cookies. They become verify user sessions and credentials more frequently. Thus, the problem won't fix very soon until rewriting entire piece of code to use username and password for authentication.
Instagram enhanced their endpoint security recently. It seems the Instagram team is going to prevent robots to access their API just using cookies. They become verify user sessions and credentials more frequently. Thus, the problem won't fix very soon until rewriting entire piece of code to use username and password for authentication.
So only way for now is wait for re-build code, because others attempts to solve problem don't work? I set trigger to get email notifs when error apear so I know when IG log me out and sheet not work and I can fast fix this, but would be better copy cookies once on month and don't fix error after few days. But most important is fact that main task in project works correct and files is saved in GDrive
New relese from 2022-09-04 can help with problems with log out? I don't know make change or not...
New relese from 2022-09-04 can help with problems with log out? I don't know make change or not...
No, it doesn't fix that issue.
I found Instagram won't require re-login and change the cookie every day if I remove the time-based trigger from the Apps Script project and only run the script manually once per day.
I found Instagram won't require re-login and change the cookie every day if I remove the time-based trigger from the Apps Script project and only run the script manually once per day.
I will test how many max. run can be before log out. Now I have 2 sheets, in both I set trigger run after 2 hours.
I found Instagram won't require re-login and change the cookie every day if I remove the time-based trigger from the Apps Script project and only run the script manually once per day.
I faced the same logout problem more than one year back, when i scheduled the script to run every 8 hours. After being banned by insta twice, and thinking that only a robot could request exactly every 8 hours, I changed the coded to run the trigger in a random time between every 8th and 9th hour:
function getRandomIntInclusive(min, max) {
min = Math.ceil(min);
max = Math.floor(max);
return Math.floor(Math.random() * (max - min + 1) + min); //The maximum is inclusive and the minimum is inclusive
}
function Triggerthis() {
const functionName = 'run';
const miliseconds = getRandomIntInclusive(60000, 360000)
ScriptApp.newTrigger(functionName)
.timeBased()
.after(miliseconds)
.create();
}
This worked nicely until recently. I think Instagram now has deployed other security modifications :(
P.S: @chriskyfung plz fix this :) This is the only auto instagram story downloader on the internet...
@Seeker220 Please update and test with the new v6 template, and let me know your result.
I added the param x-instagram-ajax to the header as in a browser, but it still have the logout problem when I used a time-based trigger a couple weeks ago. As it doesn't work as expected, so far I didn't merged the changes to the master branch.
I think there is no solution by using just the cookies in the future. It will need a headless browser, such as Selenium and Puppeteer to mimic the login process to work around. However, it will unable to build with just Apps Script. Each user may need to his/her own server or cloud function to run those third-party programs securely.
I made a single all-in-one script today... giving it:
function getRandomIntInclusive(min, max) {
min = Math.ceil(min);
max = Math.floor(max);
return Math.floor(Math.random() * (max - min + 1) + min); //The maximum is inclusive and the minimum is inclusive
}
function Trigger() {
const functionName = 'run';
const miliseconds = getRandomIntInclusive(60000, 3600000)
ScriptApp.newTrigger(functionName)
.timeBased()
.after(miliseconds)
.create();
}
// User-defined Variables
var g_username = 'username';
var g_password = 'password';
var gdrive_id = 'GOOGLE_DRIVE_FOLDER_ID';
var lastlog_id = '1XXXXXXXXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
var historylog_id = '1XXXXXXXXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
var fetchContentLog_id = '1XXXXXXXXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
var statusBadge_id = '1XXXXXXXXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
var lastTestedBadge_id = '1XXXXXXXXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
var crashReportEmail = '[email protected]';
var COOKIE = 'cookie';
var X_ASBD_ID = '123456';
var X_CSRFTOKEN = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
var X_IG_APP_ID = '123456780000000';
var X_IG_WWW_CLAIM = 'hmac.XXXXXXXXXXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
var X_INSTAGRAM_AJAX = '1000000000';
var isDebug = false;
function getQuery(ig_user_id){
return "https://i.instagram.com/api/v1/feed/reels_media/?reel_ids=" + ig_user_id;
}
function fetch_ig_stories(query){
var opt3 = {
"headers": {
"accept": "*/*",
"cache-control": "no-cache",
"pragma": "no-cache",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-site",
"x-ig-app-id": X_IG_APP_ID,
"x-ig-www-claim": X_IG_WWW_CLAIM,
"x-asbd-id": X_ASBD_ID,
"x-csrftoken": X_CSRFTOKEN,
"x-instagram-ajax": X_INSTAGRAM_AJAX,
"cookie": COOKIE
},
"referrer": "https://www.instagram.com/",
"referrerPolicy": "strict-origin-when-cross-origin",
"body": null,
"method": "GET",
"mode": "cors"
};
var data = UrlFetchApp.fetch(query, opt3).getContentText();
if (isDebug) { Logger.log(data); }
return data;
}
function parseDownloadUrl(html,isTest){
if (html == '{"reels": {}, "reels_media": [], "status": "ok"}') {
return [];
};
if (isTest) {
var data = JSON.parse(html);
} else {
var fetchContentlog = DocumentApp.openById(fetchContentLog_id);
fetchContentlog.clear();
fetchContentlog.appendParagraph(html);
var body = fetchContentlog.getBody().getText();
var data = JSON.parse(body);
}
var items = data.reels_media[0].items;
var urls = [];
for (i in items) {
var item = items[i];
if (item.video_versions) {
var videos = item.video_versions;
urls[i] = videos[0].url;
} else {
urls[i] = item.image_versions2.candidates[0].url;
}
}
if (isDebug) { Logger.log(urls); }
return urls;
}
function __main__(targetIgUser,gdrive_id) {
// Access the body of the log google doc file
var d = new Date();
var lastlogfile = DocumentApp.openById(lastlog_id);
var oldfile = lastlogfile.getBody().copy(); //backup the previous log
lastlogfile.clear(); // clear the log file for logging current process
lastlogfile.getBody().appendParagraph(targetIgUser.name + "\t" + d); // append date and time to the log file
// Fetch URL
var query_url = getQuery(targetIgUser.id);
var html = fetch_ig_stories(query_url);
var urls = parseDownloadUrl(html,false);
var msg = '';
// For each article element found in the HTML
if (urls != null && urls.length > 0) {
for (i in urls) {
var url = urls[i];
// log the url found to the Doc file
lastlogfile.getBody().appendParagraph('');
lastlogfile.getBody().appendParagraph(url);
// check DL history
var u0 = url.split("?")[0];
var hasDL = oldfile.findText(u0);
if (!hasDL) { // download the file to Google Drive
msg = uploadToDrive(url, gdrive_id , '');
}
else {
msg = 'File has already been fetched once.'
}
lastlogfile.getBody().appendParagraph('\n\t' + msg); // log the upload msg to the Doc file
Logger.log(msg);
}
}
var logtxt = lastlogfile.getBody().getText();
var historyfile = DocumentApp.openById(historylog_id);
historyfile.getBody().appendParagraph(logtxt);
return logtxt;
}
function doGet(e,gfolderid) {
var usr = '';
var pwd = '';
var target = '';
// parse query
try {
usr = e.parameter.usr.trim();
pwd = e.parameter.pwd.trim();
target = (typeof(e.parameter.target) === 'string') ? JSON.parse(e.parameter.target) : e.parameter.target;
}
catch(err) {
Logger.log(err);
}
var msg = '';
// execute if correct username and password in the query
if (usr == g_username && pwd == g_password && target != '') {
msg = __main__(target,gfolderid);
msg = __main__(target,gfolderid);
msg = ContentService.createTextOutput(msg);
}
else { msg = ContentService.createTextOutput('Invalid username and password or targetID!');
Logger.log(msg);
};
return msg;
}
function story_get(targetusername,userid,savelocation) {
var e = {
"parameter": {
"usr": g_username,
"pwd": g_password,
"target" : { "name": targetusername, "id": userid}
//"target" : { "name": "medium", "id":"1112881921"}
}
};
doGet(e,savelocation);
}
function u1() {
try{
story_get("username1","id1","folder1");
}
catch(err) {
Logger.log(err);
}
}
function u2() {
try{
story_get("username2","id2","folder2");
}
catch(err) {
Logger.log(err);
}
}
function u3() {
try{
story_get("username3","id3","folder3");
}
catch(err) {
Logger.log(err);
}
}
function run() {
u1();
u2();
u3();
}
and set trigger each 12 hours to the function Trigger()
@chriskyfung v6 gives the following error:
Please name cell B8 to X_INSTAGRAM_AJAX on the Settings page as below.
I made a single all-in-one script today... giving it:
function getRandomIntInclusive(min, max) { ⋮ ⋮and set trigger each 12 hours to the function
Trigger()
@Seeker220 DO NOT post any sensitive personal data, please remember to remove them from the code next time.
I made a single all-in-one script today... giving it:
function getRandomIntInclusive(min, max) { ⋮ ⋮and set trigger each 12 hours to the function
Trigger()@Seeker220 DO NOT post any sensitive personal data, please remember to remove them from the code next time.
Actually I changed the values a little bit :) just gave them for format...
@chriskyfung I opened a new insta account for testing, and set v6 trigger 12 hours. I noticed that in both my accounts, "X_ASBD_ID" and "X_IG_APP_ID" has the same value, although opened one in Chrome and one in Firefox...
@chriskyfung I opened a new insta account for testing, and set v6 trigger 12 hours. I noticed that in both my accounts, "X_ASBD_ID" and "X_IG_APP_ID" has the same value, although opened one in Chrome and one in Firefox...
@Seeker220 any details after testing? IG still log out?
@chriskyfung I opened a new insta account for testing, and set v6 trigger 12 hours. I noticed that in both my accounts, "X_ASBD_ID" and "X_IG_APP_ID" has the same value, although opened one in Chrome and one in Firefox...
@Seeker220 any details after testing? IG still log out?
@fifson1 no, IG did not log out as far. :)