audnexus
audnexus copied to clipboard
laxamentumtech
Feature Request: Original Publish Date
One thing which I have not been able to find a consistent source for is an original publish date for audiobooks. I'm not talking about the date an audiobook was released, I'm talking specifically about the date the first edition of a book was released. In terms of organizing an Audiobook collection (in my case, in Plex), it is generally far more convenient to allow things to be sorted by when the books were published. This is especially true for books in a series which may have multiple releases for their audiobook versions.
I know GoodReads has this info, but I also know they closed their API access. However, it's surely still possible to scrape the info from their website with something like cheerio. I know Readarr has GoodReads scraping integration but I haven't had a chance to look through their code for how they do it yet.
There is also the Google Books API, specifically the volume search which is a convenient way to search the Google books library with a JSON response, but they don't include the original publish date in that response. Which is weird because they do offer that info on their book pages. It could be realistic to use their API to find a book match and then use the books ID to scrape the Google Books page for that book.
Anyway, I'm not sure if attempting to scrape more sources like this is in the scope for this project, I've just been thinking that the original publish date for a book is one of the only things I haven't been able to get from Audible that is actually useful to have. I'm curious if you have any thoughts on the topic!
This isn’t of real interest, as, in my opinion, the date the physical book was published is not relevant data to audiobooks. They are very different mediums, and should not be globbed together.
I will concede date of first publish of an audiobook could be interesting, but that’s something which requires a release group format of metadata. This is something which AudiobookDB will be able to handle.
Scraping implementations are bound to fail, it’s always just a matter of time. This is why Audnexus is an API first, scraping second approach, with massive safeguards when scraping.
Just wanted to post an update on my opinion on this. The main reason I feel as though the original publish date of a book is useful is that, in any real world use of this tool, sorting by the original publish date is the only way to guarantee books are sorted in the correct order, especially in a series. Audible frequently removes original versions of audiobooks in favor of "movie editions", which often aren't any different from an original version except for the cover. If you try to use the release date of the movie version to sort a series of books in the order you should read them, you'll often end up with the movie edition of the first book in a series last, or towards the end.
For a practical example, I'd use the Plex metadata agent that is based on this tool. I'd argue that sorting audiobooks on Plex simply by the date that an audiobook version of a book is released, is almost entirely useless compared to when the book was originally published.
And as far as how to get that info on a book, I discovered that Readarr (which I linked in my original post) actually includes their API key for the Goodreads API in the source code, albeit in an obfuscated way. I tried implementing that in my own project for tagging audiobook files with proper metadata and it works like a charm. I'm not necessarily encouraging using theirs, but if you could get your hands on a Goodreads API key, they definitely provide that information with an API first approach like you said.
Hmm, you make a good point there. I'm not opposed to including original publication date where available. I would probably just not make it the default sort.
My concern with even integrating Goodreads is they've made it clear they don't intend to support the public API moving forward, so the rug could be pulled from under us at any moment. I also wouldn't want to put Readarr at risk of having their key revoked, as I'm not sure what if any TOS they agreed to in getting the key.
I'll still take a look at their implementation to see if I can glean any knowledge.
I understand not making it the default, my main point was just that its an important field to have in terms of overall audiobook metadata.
And definitely understandable not wanting to yoink their API key, I've only used it in a project that isn't public and I'm only doing so because they're confident enough in using the same API key for each self hosted instance of their app that is out there.
And if you're at all curious what the results from their API looks like, here is an example request: https://gist.github.com/csandman/f86dabe760a90477504c1a15fcada874
Unfortunately the response is in XML, but I was able to use the xml2js to parse it to a usable format in TS.
As far as a them closing their API, I definitely understand not wanting to rely on an API that is planned to be removed. I've been keeping my eyes out for any alternatives that offer the same feature but haven't yet found one. If I do though, I'll definitely post any updates.
XML shouldn't be too big of a problem. I forgot they're distributing the API key, so I wouldn't be in breach of TOS since it's on my system already (I think).
I'll see if I can get some preliminary testing going this week.
As for other services, AudiobookDB is almost polished enough for public usage (frontend still needs to be written), so keep an eye out for that when it's opened to public 😏 It has Audnexus integration as well for import logic, so it makes sense to add Goodreads here first and then over there.
Sounds good! So what exactly is AudiobookDB? Is there a public repo for it yet? Is it just supposed to be a collection of data acquired from audnexus essentially?
Also, here's an example of how I'm parsing the Goodreads api in TypeScript in case it could offer you any ideas: https://gist.github.com/csandman/dba05dc48f29592d0db535282c00a2af
I made it a little hastily but its mostly effective.
@csandman
And as far as how to get that info on a book, I discovered that Readarr (which I linked in my original post) actually includes their API key for the Goodreads API in the source code, albeit in an obfuscated way. I tried implementing that in my own project for tagging audiobook files with proper metadata and it works like a charm. I'm not necessarily encouraging using theirs, but if you could get your hands on a Goodreads API key, they definitely provide that information with an API first approach like you said.
With some knowledge, you can obtain an access token by make a request to https://www.goodreads.com/oauth/grant_access_token.xml. Goodreads use the https://api.amazon.com/auth/register endpoint to register a new goodreads device. These way you got a private Amazon access token which can be used to authenticate your requests to the Goodreads API and the x-amz-access-token header instead using the key in the url query.
@mkb79 any chance you could give more details on what you're describing? do you still need a goodreads API key in the first place to make that work?
@csandman
@mkb79 any chance you could give more details on what you're describing? do you still need a goodreads API key in the first place to make that work?
You only need your Goodreads username/password to register a new Goodreads device and obtain your access and refresh token!
I am definitely curious about the exact process, because I can't seem to find any up to date resources on it. Overall it could be helpful for this issue as well so you could post it here, but otherwise you could post more details on one of the gists above if you'd like.
@csandman
Here are a proof-of-concept how registration and deregistration works:
import base64
import gzip
import hashlib
import hmac
import json
import secrets
import uuid
from datetime import datetime
from io import BytesIO
from functools import partialmethod
from typing import Tuple, Union
import httpx
from pyaes import AESModeOfOperationCBC, Encrypter, Decrypter
USER_AGENT = "AmazonWebView/GoodreadsForIOS App/4.0.1/iOS/15.4.1/iPhone"
FRC_SIG_SALT: bytes = b"HmacSHA256"
FRC_AES_SALT: bytes = b"AES/CBC/PKCS7Padding"
class FrcCookieHelper:
def __init__(self, password: str) -> None:
self.password = password.encode()
def _get_key(self, salt: bytes) -> bytes:
return hashlib.pbkdf2_hmac("sha1", self.password, salt, 1000, 16)
get_signature_key = partialmethod(_get_key, FRC_SIG_SALT)
get_aes_key = partialmethod(_get_key, FRC_AES_SALT)
@staticmethod
def unpack(frc: str) -> Tuple[bytes, bytes, bytes]:
pad = (4 - len(frc) % 4) * "="
frc = BytesIO(base64.b64decode(frc+pad))
frc.seek(1) # the first byte is always 0, skip them
return frc.read(8), frc.read(16), frc.read() # sig, iv, data
@staticmethod
def pack(sig: bytes, iv: bytes, data: bytes) -> str:
frc = b"\x00" + sig[:8] + iv[:16] + data
frc = base64.b64encode(frc).strip(b"=")
return frc.decode()
def verify_signature(self, frc: str) -> bool:
key = self.get_signature_key()
sig, iv, data = self.unpack(frc)
new_signature = hmac.new(key, iv + data, hashlib.sha256).digest()
return sig == new_signature[:len(sig)]
def decrypt(self, frc: str, verify_signature: bool = True) -> bytes:
if verify_signature:
self.verify_signature(frc)
key = self.get_aes_key()
sig, iv, data = self.unpack(frc)
decrypter = Decrypter(AESModeOfOperationCBC(key, iv))
decrypted = decrypter.feed(data) + decrypter.feed()
decompressed = gzip.decompress(decrypted)
return decompressed
def encrypt(self, data: Union[str, dict]) -> str:
if isinstance(data, dict):
data = json.dumps(data, indent=2, separators=(",", " : ")).replace("/", "\\/").encode()
compressed = BytesIO()
with gzip.GzipFile(fileobj=compressed, mode="wb", mtime=False) as f:
f.write(data)
compressed.seek(8)
compressed.write(b"\x00\x13")
compressed = compressed.getvalue()
key = self.get_aes_key()
iv = secrets.token_bytes(16)
encrypter = Encrypter(AESModeOfOperationCBC(key, iv))
encrypted = encrypter.feed(compressed) + encrypter.feed()
key = self.get_signature_key()
signature = hmac.new(key, iv + encrypted, hashlib.sha256).digest()
packed = self.pack(signature, iv, encrypted)
return packed + len(packed) % 4 * "="
def register(username, password):
url = "https://api.amazon.com/auth/register"
device_serial = secrets.token_hex(16).upper()
frc = {
"ApplicationVersion": "4.1",
"DeviceOSVersion": "iOS/15.5",
"ScreenWidthPixels": "428",
"TimeZone": "+02:00",
"ScreenHeightPixels": "926",
"ApplicationName": "Goodreads",
"DeviceJailbroken": False,
"DeviceLanguage": "en-DE",
"DeviceFingerprintTimestamp": round(datetime.utcnow().timestamp()) * 1000,
"ThirdPartyDeviceId": str(uuid.uuid4()).upper(),
"DeviceName": "iPhone",
"Carrier": "Vodafone.de"
}
frc = FrcCookieHelper(device_serial).encrypt(frc)
headers = {
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": USER_AGENT,
"Accept-Encoding": "gzip",
"Accept": "application/json",
"Accept-Language": "en-DE",
"Accept-Charset": "utf-8"
}
json_body = {
"requested_extensions": [
"device_info",
"customer_info"
],
"cookies": {
"website_cookies": [],
"domain": ".goodreads.com"
},
"registration_data": {
"domain": "Device",
"app_version": "4.1",
"device_type": "A3NWHXTQ4EBCZS",
"os_version": "15.5",
"device_serial": device_serial,
"device_model": "iPhone",
"app_name": "GoodreadsForIOS App",
"software_version": "1"
},
"auth_data": {
"user_id_password": {
"user_id": username,
"password": password
}
},
"user_context_map": {
"frc": frc
},
"requested_token_type": [
"bearer",
"mac_dms",
"website_cookies"
]
}
r = httpx.post(url, headers=headers, json=json_body)
return r
def deregister(access_token):
json_body = {"deregister_all_existing_accounts": True}
headers = {"Authorization": f"Bearer {access_token}"}
r = httpx.post(
"https://api.amazon.com/auth/deregister",
json=json_body,
headers=headers
)
return r
def refresh_access_token(refresh_token):
pass
def exchange_cookies(refresh_token):
pass
You are need httpx and pyaes installed from PyPI. Refreshing the access token and obtaining additional cookies must be implemented.
Then you can request the Goodreads API using x-amz-access-token ACCESS_TOKEN and User-Agent Goodreads/4.0.1 (iPhone; iOS 15.4.1; Scale/3.00) in your request headers.
Thanks for the detailed example! Now time to see if I can translate this to node...
Thanks for the detailed example! Now time to see if I can translate this to node...
@csandman Have you tried this code out. If yes, could you successfully register/unregister a device? I've tested this only on my machine.
haven't tried unregister yet but it does appear to be working! The token you're talking about I assume is the response.success.tokens.bearer.access_token right?
I'm also close to finishing a node version, but idk if I did it right. Translating all this buffer manipulation stuff is always a pain haha.
Yes, this is the access token. The token is valid for 60 minutes after registration. Before the token is invalid, you have to deregister or refresh the token with the refresh token. Refreshing token is easy with the correct request headers, params and body.
I'm having quite the time writing this in TS, since I've never worked with python's byte type and TS doesn't have the same type. @csandman let me know if you get a working POC on this. In the meantime I'll integrate using the API key directly.
Will do, I feel like I'm close but for some reason it's still not working. I've done similar conversion of python code to TS code before, and I've found using the Nodejs Buffer instead of Python's byte type is generally the way to go. If I can't figure out what's wrong just from tinkering then I'll probably end up breaking it down into pieces to figure out which part went wrong. It's just tough to check for comparisons when multiple values are randomly generated each time.
Here's what I have so far if you want to check it out:
EDIT: I ended up figuring it out! Man that was rough, had to learn way more about how the crypto module from Node works than I thought I ever would haha. But in the end, I was able to get it all working with only built in node modules (besides node-fetch but you can replace that part with whatever you want). I decided to replace this code to keep this post from getting ridiculously long.
Let me know if you have any trouble getting it working!
// types/goodreads.ts
export interface GoodreadsFrc {
ApplicationVersion: string;
DeviceOSVersion: string;
ScreenWidthPixels: string;
TimeZone: string;
ScreenHeightPixels: string;
ApplicationName: string;
DeviceJailbroken: boolean;
DeviceLanguage: string;
DeviceFingerprintTimestamp: number;
ThirdPartyDeviceId: string;
DeviceName: string;
Carrier: string;
}
export interface GoodreadsRegisterRequest {
requested_extensions: string[];
cookies: {
website_cookies: string[];
domain: string;
};
registration_data: {
domain: string;
app_version: string;
device_type: string;
os_version: string;
device_serial: string;
device_model: string;
app_name: string;
software_version: string;
};
auth_data: {
user_id_password: {
user_id: string;
password: string;
};
};
user_context_map: {
frc: string;
};
requested_token_type: string[];
}
export interface GoodreadsRegisterFailureResponse {
response: {
challenge: {
challenge_reason: string;
uri: string;
required_authentication_method: string;
};
};
request_id: string;
}
export interface GoodreadsRegisterSuccessResponse {
response: {
success: {
extensions: {
device_info: {
device_name: string;
device_serial_number: string;
device_type: string;
};
customer_info: {
account_pool: string;
preferred_marketplace: string;
country_of_residence: string;
user_id: string;
home_region: string;
name: string;
given_name: string;
source_of_country_of_residence: string;
};
};
tokens: {
mac_dms: {
device_private_key: string;
adp_token: string;
};
bearer: {
access_token: string;
refresh_token: string;
expires_in: string;
};
};
customer_id: string;
};
};
request_id: string;
}
export interface GoodreadsDeregisterFailureResponse {
response: {
error: {
code: string;
message: string;
};
};
request_id: string;
}
export interface GoodreadsDeregisterSuccessResponse {
response: {
success: Record<string, never>;
};
request_id: string;
}
export interface GoodreadsRefreshFailureResponse {
error_index: string;
error_description: string;
error: string;
}
export interface GoodreadsRefreshSuccessResponse {
access_token: string;
token_type: string;
expires_in: number;
}
export interface GoodreadsCookieFailureResponse {
response: {
error: {
code: string;
detail: string;
message: string;
};
};
request_id: string;
}
export interface GoodreadsCookie {
Path: string;
Secure: boolean;
Value: string;
Expires: string;
HttpOnly: boolean;
Name: string;
}
export interface GoodreadsCookieSuccessResponse {
response: {
tokens: {
cookies: {
".goodreads.com": GoodreadsCookie[];
};
};
};
request_id: string;
}
import {
createCipheriv,
createDecipheriv,
createHmac,
pbkdf2Sync,
randomBytes,
randomFill,
randomUUID,
} from "crypto";
import fetch from "node-fetch";
import type {
GoodreadsCookieFailureResponse,
GoodreadsCookieSuccessResponse,
GoodreadsDeregisterFailureResponse,
GoodreadsDeregisterSuccessResponse,
GoodreadsFrc,
GoodreadsRefreshFailureResponse,
GoodreadsRefreshSuccessResponse,
GoodreadsRegisterFailureResponse,
GoodreadsRegisterRequest,
GoodreadsRegisterSuccessResponse,
} from "types/goodreads";
import { promisify } from "util";
import { gunzipSync, gzipSync } from "zlib";
const REGISTER_URL = "https://api.amazon.com/auth/register";
const DEREGISTER_URL = "https://api.amazon.com/auth/deregister";
const REFRESH_URL = "https://api.amazon.com/auth/token";
const COOKIES_URL = "https://api.amazon.com/ap/exchangetoken/cookies";
const USER_AGENT = "AmazonWebView/GoodreadsForIOS App/4.0.1/iOS/15.4.1/iPhone";
class FrcCookieHelper {
static FRC_SIG_SALT = Buffer.from("HmacSHA256");
static FRC_AES_SALT = Buffer.from("AES/CBC/PKCS7Padding");
static CIPHER_ALGORITHM = "aes-128-cbc";
password: string;
constructor(password: string) {
this.password = password;
}
getKey(salt: Buffer) {
return pbkdf2Sync(this.password, salt, 1000, 16, "sha1");
}
getSignatureKey() {
return this.getKey(FrcCookieHelper.FRC_SIG_SALT);
}
getAesKey() {
return this.getKey(FrcCookieHelper.FRC_AES_SALT);
}
static getRandomIv() {
return new Promise<Uint8Array>((resolve, reject) => {
randomFill(new Uint8Array(16), (err, arr) => {
if (err) {
reject(err);
}
resolve(arr);
});
});
}
static unpack(frc: string): [Buffer, Buffer, Buffer] {
const pad = "=".repeat(4 - (frc.length % 4));
const newFrc = Buffer.from(frc + pad, "base64");
const sig = newFrc.slice(1, 9);
const iv = newFrc.slice(9, 25);
const data = newFrc.slice(26);
return [sig, iv, data];
}
static pack(sig: Buffer, iv: Uint8Array, data: Buffer): string {
let frc = Buffer.concat([Buffer.from([0x00]), sig.slice(0, 8), iv, data]);
const rem = Buffer.from("=");
while (frc.indexOf(rem) === 0) {
frc = frc.slice(1);
}
while (frc.lastIndexOf(rem) === frc.length - 1) {
frc = frc.slice(0, frc.length - 1);
}
return frc.toString("base64");
}
verifySignature(frc: string): boolean {
const key = this.getSignatureKey();
const [sig, iv, data] = FrcCookieHelper.unpack(frc);
const hmac = createHmac("sha256", key);
hmac.write(Buffer.concat([iv, data]));
const newSignature = hmac.digest();
return sig === newSignature.slice(0, sig.length);
}
decrypt(frc: string, verifySignature = true): Buffer {
if (verifySignature) {
this.verifySignature(frc);
}
const key = this.getAesKey();
const [, iv, data] = FrcCookieHelper.unpack(frc);
const decipher = createDecipheriv(
FrcCookieHelper.CIPHER_ALGORITHM,
key,
iv
);
const decrypted = decipher.update(data);
const decryptedFinal = decipher.final();
const decryptedData = Buffer.concat([decrypted, decryptedFinal]);
return gunzipSync(decryptedData);
}
async encrypt(data: string | GoodreadsFrc): Promise<string> {
let dataStr: string;
if (typeof data === "object") {
dataStr = JSON.stringify(data, null, 2);
} else {
dataStr = data;
}
const zip = gzipSync(dataStr);
const gzippedData = Buffer.concat([
zip.slice(0, 8),
Buffer.from([0x00, 0x13]),
zip.slice(10),
]);
const aesKey = this.getAesKey();
const iv = await FrcCookieHelper.getRandomIv();
const cipher = createCipheriv(FrcCookieHelper.CIPHER_ALGORITHM, aesKey, iv);
const encrypted = cipher.update(gzippedData);
const encryptedFinal = cipher.final();
const encryptedData = Buffer.concat([encrypted, encryptedFinal]);
const sigKey = this.getSignatureKey();
const hmac = createHmac("sha256", sigKey);
hmac.write(Buffer.concat([iv, encryptedData]));
const signature = hmac.digest();
const packed = FrcCookieHelper.pack(signature, iv, encryptedData);
return packed + "=".repeat(packed.length % 4);
}
}
const randomBytesAsync = promisify(randomBytes);
export const register = async (username: string, password: string) => {
const deviceSerial = await randomBytesAsync(16).then((buf) =>
buf.toString("hex").toUpperCase()
);
const frcBase: GoodreadsFrc = {
ApplicationVersion: "4.1",
DeviceOSVersion: "iOS/15.5",
ScreenWidthPixels: "428",
TimeZone: "+02:00",
ScreenHeightPixels: "926",
ApplicationName: "Goodreads",
DeviceJailbroken: false,
DeviceLanguage: "en-DE",
DeviceFingerprintTimestamp: new Date().getTime(),
ThirdPartyDeviceId: randomUUID().toUpperCase(),
DeviceName: "iPhone",
Carrier: "Vodafone.de",
};
const frcHelper = new FrcCookieHelper(deviceSerial);
const frc = await frcHelper.encrypt(frcBase);
const headers = {
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": USER_AGENT,
"Accept-Encoding": "gzip",
Accept: "application/json",
"Accept-Language": "en-DE",
"Accept-Charset": "utf-8",
};
const body: GoodreadsRegisterRequest = {
requested_extensions: ["device_info", "customer_info"],
cookies: {
website_cookies: [],
domain: ".goodreads.com",
},
registration_data: {
domain: "Device",
app_version: "4.1",
device_type: "A3NWHXTQ4EBCZS",
os_version: "15.5",
device_serial: deviceSerial,
device_model: "iPhone",
app_name: "GoodreadsForIOS App",
software_version: "1",
},
auth_data: {
user_id_password: {
user_id: username,
password,
},
},
user_context_map: {
frc,
},
requested_token_type: ["bearer", "mac_dms", "website_cookies"],
};
const res = await fetch(REGISTER_URL, {
method: "POST",
headers,
body: JSON.stringify(body),
});
const resData = (await res.json()) as
| GoodreadsRegisterSuccessResponse
| GoodreadsRegisterFailureResponse;
console.log("GOODREADS AUTH RESPONSE");
console.dir(resData, { depth: null });
if ("challenge" in resData.response) {
throw new Error(resData.response.challenge.challenge_reason);
}
return resData as GoodreadsRegisterSuccessResponse;
};
export const deregister = async (
accessToken: string
): Promise<GoodreadsDeregisterSuccessResponse> => {
const res = await fetch(DEREGISTER_URL, {
method: "POST",
headers: {
Authorization: `Bearer ${accessToken}`,
},
body: JSON.stringify({
deregister_all_existing_accounts: true,
}),
});
const resData = (await res.json()) as
| GoodreadsDeregisterSuccessResponse
| GoodreadsDeregisterFailureResponse;
console.log("GOODREADS DEREGISTER RESPONSE");
console.dir(resData, { depth: null });
if ("error" in resData.response) {
throw new Error(resData.response.error.message);
}
return resData as GoodreadsDeregisterSuccessResponse;
};
export const refreshAccessToken = async (
refreshToken: string
): Promise<GoodreadsRefreshSuccessResponse> => {
const res = await fetch(REFRESH_URL, {
method: "POST",
headers: {
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": USER_AGENT,
"Accept-Encoding": "gzip",
"Content-Type": "application/x-www-form-urlencoded",
},
body: new URLSearchParams({
app_name: "GoodreadsForIOS App",
app_version: "4.0.1",
"di.sdk.version": "6.12.1",
source_token: refreshToken,
package_name: "com.goodreads.Goodreads",
"di.hw.version": "iPhone",
platform: "iOS",
requested_token_type: "access_token",
source_token_type: "refresh_token",
"di.os.name": "iOS",
"di.os.version": "15.4.1",
current_version: "6.12.1",
}),
});
const resData = (await res.json()) as
| GoodreadsRefreshSuccessResponse
| GoodreadsRefreshFailureResponse;
console.log("GOODREADS REFRESH TOKEN RESPONSE");
console.dir(resData, { depth: null });
if ("error_description" in resData) {
throw new Error(resData.error_description);
}
return resData as GoodreadsRefreshSuccessResponse;
};
export const exchangeCookies = async (
refreshToken: string
): Promise<GoodreadsCookieSuccessResponse> => {
const res = await fetch(COOKIES_URL, {
method: "POST",
headers: {
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": USER_AGENT,
"Accept-Encoding": "gzip",
"Content-Type": "application/x-www-form-urlencoded",
},
body: new URLSearchParams({
"openid.assoc_handle": "amzn_goodreads_web_na",
app_name: "GoodreadsForIOS App",
app_version: "4.0.1",
"di.sdk.version": "6.12.1",
domain: ".goodreads.com",
source_token: refreshToken,
"di.hw.version": "iPhone",
cookies: "eyJjb29raWVzIjp7Ii5nb29kcmVhZHMuY29tIjpbXX19",
requested_token_type: "auth_cookies",
source_token_type: "refresh_token",
"di.os.name": "iOS",
"di.os.version": "15.4.1",
}),
});
const resData = (await res.json()) as
| GoodreadsCookieSuccessResponse
| GoodreadsCookieFailureResponse;
console.log("GOODREADS COOKIES RESPONSE");
console.dir(resData, { depth: null });
if ("error" in resData.response) {
throw new Error(resData.response.error.message);
}
return resData as GoodreadsCookieSuccessResponse;
};
I ended up figuring it out, updated my previous comment with a working example.
Oh, I also finally tested actually using the bearer token to pull from the API, and it works! Probably should have made sure of that before I went through the whole process of converting the file, but glad it works anyway haha.
Here are the function to refresh the access token:
def refresh_access_token(refresh_token):
url = "https://api.amazon.com/auth/token"
headers = {
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": USER_AGENT,
"Accept-Encoding": "gzip"
}
body = {
"app_name": "GoodreadsForIOS App",
"app_version": "4.0.1",
"di.sdk.version": "6.12.1",
"source_token": refresh_token,
"package_name": "com.goodreads.Goodreads",
"di.hw.version": "iPhone",
"platform": "iOS",
"requested_token_type": "access_token",
"source_token_type": "refresh_token",
"di.os.name": "iOS",
"di.os.version": "15.4.1",
"current_version": "6.12.1"
}
r = httpx.post(url, data=body, headers=headers)
return r
Finally here comes the exchange token for cookies part:
def exchange_cookies(refresh_token):
url = "https://api.amazon.com/ap/exchangetoken/cookies"
headers = {
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": USER_AGENT,
"Accept-Encoding": "gzip"
}
body = {
"openid.assoc_handle": "amzn_goodreads_web_na",
"app_name": "GoodreadsForIOS App",
"app_version": "4.0.1",
"di.sdk.version": "6.12.1",
"domain": ".goodreads.com",
"source_token": refresh_token,
"di.hw.version": "iPhone",
"cookies": "eyJjb29raWVzIjp7Ii5nb29kcmVhZHMuY29tIjpbXX19",
"requested_token_type": "auth_cookies",
"source_token_type": "refresh_token",
"di.os.name": "iOS",
"di.os.version": "15.4.1"
}
r = httpx.post(url, data=body, headers=headers)
return r
So, it’s on yours for further progress ;)! If you need any help, feel free to contact me.
Here are some undocumented API endpoints I could find:
https://www.goodreads.com/api/current_user_data?_nc=true¤tly_reading=true&format=xml&id={USER_ID}&include_social_shelving_info=true&user_shelves=true&v=2
https://www.goodreads.com/api/v3/updates/newsfeed?_nc=true&format=xml&max_updates=10
https://www.goodreads.com/api/book/basic_book_data/182506?_extras%5Bbook_covers_large%5D=true&_nc=true&format=xml
https://www.goodreads.com/api/book/book_reviews/182506?_extras%5Bbook_covers_large%5D=true&_nc=true&format=xml
https://www.goodreads.com/api/book/related_books/182506?_extras%5Bbook_covers_large%5D=true&_nc=true&format=xml
https://www.goodreads.com/api/gca_metadata?_nc=true&format=xml&locale=de-DE
https://www.goodreads.com/api/current_user_shelves?_nc=true&format=xml
@mkb79 thanks for all the extra info! I was just thinking about asking if you had a hint on the other requests. Also thanks a ton for this whole thing, I can definitely think of a few applications for it!
So I was able to get the refresh function working, but I did have to add "Content-Type": "application/json" to the request headers. However, for some reason the exchange cookies function isn't working. I'm getting the following error when I try:
{
response: {
error: {
code: 'MissingValue',
detail: 'Missing parameter: app_name',
message: 'One or more required values are missing'
}
},
request_id: 'F8Q090DH5W85QP7REAGK'
}
Which is odd because I copied your body exactly, and the app_name is in there. Any ideas?
I was also curious, what's the exchange_cookies function even for? I'm having trouble seeing a case where one might need cookies in this whole process, considering the whole point of this was to get a bearer token for the header.
So I was able to get the refresh function working, but I did have to add "Content-Type": "application/json" to the request headers.
The right Content-Type for https://api.amazon.com/auth/token and https://api.amazon.com/ap/exchangetoken/cookies is application/x-www-form-urlencoded. This is set by httpx automatically when post the request body as data instead of json (like the register and deregister function). So this header was missing in my code. Sorry for that.
Which is odd because I copied your body exactly, and the
app_nameis in there. Any ideas?
Maybe the solution is sending the data in urlencoded format? Or can you post your code implementation?
I was also curious, what's the exchange_cookies function even for? I'm having trouble seeing a care where one might need cookies in this whole process, considering the whole point of this was to get a bearer token for the header.
Some requests using these cookies in addition to the access token. But I had no issues sending the request without the cookies. So I post the code here for completeness. Maybe this cookies can be used to make authenticated requests to Goodreads.com?!
Edit: I mean authenticated requests to the Goodreads.com webpages and not using the API.
The right Content-Type for https://api.amazon.com/auth/token and https://api.amazon.com/ap/exchangetoken/cookies is application/x-www-form-urlencoded.
Interesting that you say that because I saw the same thing in some different AWS docs, but it has been working so far for all of the other requests to use the application/json type. I ended up trying out using form data instead and am still running into the same issue. This is my code so far if you're curious:
import FormData from "form-data";
import fetch from "node-fetch";
const COOKIES_URL = "https://api.amazon.com/ap/exchangetoken/cookies";
const USER_AGENT = "AmazonWebView/GoodreadsForIOS App/4.0.1/iOS/15.4.1/iPhone";
export const exchangeCookies = async (refreshToken: string) => {
const headers = {
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": USER_AGENT,
"Accept-Encoding": "gzip",
"Content-Type": "application/x-www-form-urlencoded",
};
const body = {
"openid.assoc_handle": "amzn_goodreads_web_na",
app_name: "GoodreadsForIOS App",
app_version: "4.0.1",
"di.sdk.version": "6.12.1",
domain: ".goodreads.com",
source_token: refreshToken,
"di.hw.version": "iPhone",
cookies: "eyJjb29raWVzIjp7Ii5nb29kcmVhZHMuY29tIjpbXX19",
requested_token_type: "auth_cookies",
source_token_type: "refresh_token",
"di.os.name": "iOS",
"di.os.version": "15.4.1",
};
const formBody = new FormData();
Object.entries(body).forEach(([key, value]) => {
formBody.append(key, value);
});
const res = await fetch(COOKIES_URL, {
method: "POST",
headers,
body: formBody,
});
const resData = await res.json();
console.log("COOKIES RESPONSE");
console.dir(resData, {
depth: null,
});
return resData;
};
I'm not overly concerned about making this function work though, I don't think I'd really need it for my use case. But like you said, completeness is nice.
@csandman
This works for me
const fetch = require('node-fetch')
const res = fetch('https://api.amazon.com/ap/exchangetoken/cookies', {
method: 'POST',
headers:{
'Content-Type': 'application/x-www-form-urlencoded',
"x-amzn-identity-auth-domain": "goodreads.com",
"User-Agent": 'AmazonWebView/GoodreadsForIOS App/4.0.1/iOS/15.4.1/iPhone',
"Accept-Encoding": "gzip"
},
body: new URLSearchParams({
"openid.assoc_handle": "amzn_goodreads_web_na",
app_name: "GoodreadsForIOS App",
app_version: "4.0.1",
"di.sdk.version": "6.12.1",
domain: ".goodreads.com",
source_token: "YOUR_REFRESH_TOKEN",
"di.hw.version": "iPhone",
cookies: "eyJjb29raWVzIjp7Ii5nb29kcmVhZHMuY29tIjpbXX19",
requested_token_type: "auth_cookies",
source_token_type: "refresh_token",
"di.os.name": "iOS",
"di.os.version": "15.4.1",
})
})
.then(res => res.json())
.then(json => {
console.dir(json, {
depth: null,
});
})
Edit: I'm using Node 12 and node-fetch@2 because I'm coding on my iOS device.
Interesting that you say that because I saw the same thing in some different AWS docs, but it has been working so far for all of the other requests to use the application/json type.
Some post requests made by the Goodreads/Audible/Kindle iOS Apps are json encoded and some url encoded. The refresh token and cookie exchange requests are url encoded. Maybe json will work too, but I doesn’t try this out yet.