Annotation download fails for specific audiobooks

[devvythelopper]

@mkb79 has discovered the following information. Has any work been done on this? Otherwise I would try implementing it, maybe in a new branch?

I've checked now what the Audible App for iOS do.

The app calls https://api.audible.de/1.0/annotations/lastpositions?asins=B0D186SQWV&response_groups=always-returned and receives

{
  "asin_last_position_heard_annots": [{
    "asin": "B0D186SQWV",
    "last_position_heard": {
      "last_updated": "2025-06-16 20:23:18.818",
      "position_ms": 4542015,
      "status": "Exists"
    }
  }],
  "response_groups": ["always-returned"]
}

This are not the annotations we are want because clips are missing. So I've created myself a new clip. The client makes a POST request to https://cde-ta-g7g.amazon.com/FionaCDEServiceEngine/sidecar?countryCode=DE&device_lto=120&type=AUDI with content

<annotations version="1.0" timestamp="2025-08-28T13:25:26+0200">
    <book key="B0D186SQWV" type="AUDI" version="63671221" guid="CR!G8ATWB6RT10TF1W9A79J6M9D9E2R:63671221" format="M4A_XHE"><last_heard action="modify" begin="4571103" timestamp="2025-08-28T13:25:26+0200" /><clip action="create" begin="4540857" end="4568601" timestamp="2025-08-28T13:25:24+0200"><metadata><![CDATA[{"title":"","c_version":"63671221"}]]></metadata></clip></book>
</annotations>

and then make a GET request to https://cde-ta-g7g.amazon.com/FionaCDEServiceEngine/sidecar?format=M4A_XHE&guid=CR!6HM3EF2YRS3G7576NTBYNEH0DGMT:5304471&key=3837144763&type=AUDI. That’s new. In previous versions of Audible it uses only the asin and type=AUDI`. Now a format and guid is provided in the request as well. That’s not a problem. These information can be taken from the voucher. The guid can be found as the acr key and the format as the key content_format.

Originally posted by @mkb79 in #235

[mkb79]

@devvythelopper

Below I'll give you the code to build the sidecar url dynamically from a voucher file or a license response. You can request this url with an authenticated client e.g. audible.AsyncClient.

from __future__ import annotations

import json
from dataclasses import dataclass
from typing import Any, Mapping
from urllib.parse import urlencode, urlunparse, quote


@dataclass(frozen=True)
class ContentReference:
    """Typed view on 'content_license.content_metadata.content_reference'.

    Only the fields required to build the Sidecar URL are enforced:
    - acr
    - version
    - asin
    - content_format

    All other keys may or may not be present and are ignored.
    """

    acr: str
    version: str
    asin: str
    content_format: str

    @classmethod
    def from_mapping(cls, data: Mapping[str, Any]) -> ContentReference:
        """Create a ``ContentReference`` from a mapping.

        Args:
            data (Mapping[str, Any]): Mapping of the ``content_reference`` node.

        Returns:
            ContentReference: A validated ``ContentReference`` instance.

        Raises:
            KeyError: If one or more required keys are missing or empty.
        """
        required = ("acr", "version", "asin", "content_format")
        missing = [k for k in required if not data.get(k)]
        if missing:
            raise KeyError(f"Missing required content_reference keys: {', '.join(missing)}")

        return cls(
            acr=str(data["acr"]),
            version=str(data["version"]),
            asin=str(data["asin"]),
            content_format=str(data["content_format"]),
        )

    @property
    def guid(self) -> str:
        """Assemble the GUID value as ``acr:version``.

        Returns:
            str: GUID string built from ``acr`` and ``version``.
        """
        return f"{self.acr}:{self.version}"


def _urlencode_with_safe(params: Mapping[str, str]) -> str:
    """Encode query parameters while keeping '!' and ':' unescaped.

    This matches observed Sidecar URLs where these characters are not
    percent-encoded.

    Args:
        params (Mapping[str, str]): Query parameters.

    Returns:
        str: Encoded query string.
    """

    def _quote(v: str, safe: str = "!:") -> str:
        return quote(v, safe=safe)

    return urlencode(params, quote_via=_quote)


def build_sidecar_url_from_full_response(
    full_response: Mapping[str, Any] | str,
    *,
    base_host: str = "cde-ta-g7g.amazon.com",
    service_path: str = "/FionaCDEServiceEngine/sidecar",
    fixed_type: str = "AUDI",
) -> str:
    """Build the Sidecar URL from a complete Audible response body.

    The function expects the following structure:

        response["content_license"]["content_metadata"]["content_reference"]

    Required keys inside ``content_reference`` are:
    - acr
    - version
    - asin
    - content_format

    The resulting URL has the shape::

        https://{base_host}{service_path}?format={content_format}&guid={acr}:{version}&key={asin}&type=AUDI

    Args:
        full_response (Mapping[str, Any] | str): Full response as mapping or
            JSON string.
        base_host (str, optional): Service host. Defaults to
            ``cde-ta-g7g.amazon.com``.
        service_path (str, optional): Service path component. Defaults to
            ``/FionaCDEServiceEngine/sidecar``.
        fixed_type (str, optional): Fixed ``type`` parameter. Defaults to
            ``AUDI``.

    Returns:
        str: Fully-qualified Sidecar URL.

    Raises:
        ValueError: If JSON parsing fails.
        TypeError: If structures are of unexpected types.
        KeyError: If required nodes or fields are missing.
    """
    if isinstance(full_response, str):
        try:
            full_response = json.loads(full_response)
        except json.JSONDecodeError as exc:
            raise ValueError("full_response is not valid JSON") from exc

    if not isinstance(full_response, Mapping):
        raise TypeError("full_response must be a Mapping or JSON string")

    cl = full_response.get("content_license")
    if not isinstance(cl, Mapping):
        raise KeyError("content_license is missing or not a mapping")

    cm = cl.get("content_metadata")
    if not isinstance(cm, Mapping):
        raise KeyError("content_license.content_metadata is missing or not a mapping")

    cref_raw = cm.get("content_reference")
    if not isinstance(cref_raw, Mapping):
        raise KeyError(
            "content_license.content_metadata.content_reference is missing or not a mapping"
        )

    cref = ContentReference.from_mapping(cref_raw)

    query = _urlencode_with_safe(
        {
            "format": cref.content_format,
            "guid": cref.guid,  # always computed as acr:version
            "key": cref.asin,
            "type": fixed_type,
        }
    )
    return urlunparse(("https", base_host, service_path, "", query, ""))


def example() -> None:
    """Demonstrate URL building from a sample response structure."""
    sample  = json.load(VOUCHER_FILE) or directly license response as Python dict
    print(build_sidecar_url_from_full_response(sample))

[devvythelopper]

@mkb79 thank you very much.

So I basically just pasted your code into models.py and used the url within get_annotations, and committed and pushed the code into refactor/download-command ( e5551a9195f26f37c36317330f3bf51176d083b6).

It did not solve the problem, i.e. the annotations for the same audiobooks as before fail to download, and it still works for those for which it worked before.

But I've found a book that is included freely within my (german) audible subscription, with which you maybe could test yourself: Beauty by Roger Scruton, ASIN 1977345921, url: https://www.audible.de/pd/Beauty-Hoerbuch/1977345921 to see which parameters the iOS app uses to download the annotations. The book is part of the subscription until the 23rd this month.

[devvythelopper]

@mkb79 I've stumbled upon another problem when downloading the forever free podcast episodes 22 and 21 from the podcast Aquinas 101 (https://www.audible.de/podcast/Aquinas-101-Course-2-Introduction-to-Thomistic-Philosophy/B08K5MVNLL).

The relevant loglines are:

debug: Adding annotations download job for B095X3KTSM (Episode 22 – Thinking Like Angels - the Spark of Reason & the Fire of Intellectuality | Fr. Raymund Snyder, O.P.)
error: 'Missing required content_reference keys: version'

It is possible that the error occurs for all episodes, since after the second error the download queue just hangs despite --ignore-errors being set.

[devvythelopper]

Since I have not created a toolset to investigate what the audible app on either iOS or android is doing, I cannot proceed myself.

[mkb79]

@devvythelopper Do you have an iPhone? The I can tell you an app which can decrypt ssl traffic and bypass ssl pinning (in most cases)!

[devvythelopper]

@mkb79 sadly no. And my time in the future is quite limited. I won't have time to dive so deeply into this project, sorry.