Link header-based paging

[kevinchalet]

Hey,

To make our lives more painful than they are, some SendGrid APIs - like global suppressions - use Link response headers instead of their usual _metadata node to expose paging-related information. For instance, here's what the suppression/unsubscribes endpoint returns:

Link: <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=50>; rel="next"; title="2", <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=0>; rel="prev"; title="1", <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=400>; rel="last"; title="9", <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=0>; rel="first"; title="1"

Is there currently any way to access this response header using the typed client?

Off-topic: have you considered joining the GitHub Sponsors program? I enjoyed using StrongGrid in a project I worked on recently and I'd love to contribute back.

[Jericho]

To make our lives more painful than they are, some SendGrid APIs ...

Your comment made me laugh and cry at the same time because it's so true. The lack of consistency in SendGrid's API is really astounding! There are so many gotchas and one thing that makes my head explode is that, no matter how many times you report these problems, SendGrid doesn't seem interested in fixing any of them. I reported issues 4-5 years ago and they still haven't been fixed. Even issues related to incorrect or incomplete documentation sometimes don't get fixed. I do my best to ensure that StrongGrid shields people from these inconsistencies and work around some of the problems and I honestly don't understand how people can tolerate interacting with this API using SendGrid's own client which forces every developer to deal with these issues.

Anyway, enough ranting. Let's move on to addressing your question.

Is there currently any way to access this response header using the typed client?

This is the first time I hear about this!?!?!?! I quickly searched the documentation and I can't find any mention of this link header so I tested with a GET request to the suppression/unsubscribes endpoint and I indeed notice a "Link" header in the response, exactly as you described. Their documentation for this specific endpoint leads you to believe that the response simply contains an array of records without any kind of paging information.

I'm happy to add functionality to parse this Link header. I will publish it to my MyGet feed as soon as I have a draft and welcome your feedback.

I wonder if there are other endpoints in SendGrid's API that have this header?

have you considered joining the GitHub Sponsors program?

It's very kind of you to offer sponsoring me, I appreciate it but at this time I am not looking to earn any revenue from the work I put into this library. My reward is when I realize that people benefit from my efforts. I get a lot more satisfaction from interacting with developers and hearing that my library helped them. Maybe some day the support burden will be too great and my opinion on sponsorship will change.

If you want to help me in the short term, I am looking for assistance to spread the word about StrongGrid. Maybe you could write a blog post (if you have a blog) or write a testimonial in the discussions section? Maybe you could interact with other developers and introduce them to the library? I do this sometimes on SendGrid's repo but I am always afraid to be perceived negatively since I am promoting a project that I created.

[Jericho]

Here's my first draft: https://www.myget.org/feed/jericho/package/nuget/StrongGrid/0.76.0-paginated-respon0056

Let me konw what you think.

[jsgoupil]

@Jericho Hey, I work with @kevinchalet on this. Your API seems to work but the Link header is technically an RFC: https://tools.ietf.org/html/rfc2068#section-19.6.2.4

In the RFC, link-param has more things it can have. I would hate that it fails because they change this part (if they are even following this RFC...) It does seem the order cannot change, but maybe you want to double check that in your code or at least double check if the sub-header is present. Maybe a TryParse on that title as well.

[Jericho]

ok, but you're assuming that SendGrid is implementing the RFC that you mentioned. I'm willing to invest the time and effort to implement a more robust Link parser that conforms to the RFC but we need to confirm with SendGrid that they indeed intend to respect the RFC.

[jsgoupil]

@Jericho as I mentioned, they might not follow it. I just thought it would be better it doesn't crash in the core of your library if something were to differ slightly. You mentioned earlier that they simply don't update their documentation or help with support. I am 200% with you on this. I believe they will not change their API (v3) any further (or doc for that matter); their project seems "dead" to me. Their support takes about 2-4 weeks to get an answer and they do not fix any bugs. (ranting on my part!)

[Jericho]

I asked a question about this Link header in SendGrid's documentation repo but to be honest I don't really expect a response from them. I have asked other questions in this documentation repo that have been sitting unanswered for several months.

In the mean time I published a new version to my MyGet feed with a more robust parser that should conform to the RFC: https://www.myget.org/feed/jericho/package/nuget/StrongGrid/0.76.0-paginated-respon0062

[Jericho]

One think I should have asked earlier: what is it that you want to accomplish with the link information? How is this information helpful to you?

[jsgoupil]

Pagination. I need a count. Since this is a paginated API, I don't have a "Next" only; but even without the total count, I would have to have a Next even if there is nothing after.

[Jericho]

How do you intend to get the count? The link information does not include this data. For instance, the following example tells us that there are more than 400 records and less than 450 but it doesn't tell you precisely how many records there are.

Link: 
    <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=50>; rel="next"; title="2",
    <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=0>; rel="prev"; title="1",
    <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=400>; rel="last"; title="9",
    <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=50&offset=0>; rel="first"; title="1"

The only way to get a total count that I can think of would be to get 1 record per page in which case the link information would contain something similar to:

    <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=1&offset=411>; rel="last"; title="412",

This would allow you to assume there are 412 records.

[jsgoupil]

With your example, I have the limit of 50 and the last page is 9, so I know there are at most 450 records. This is sufficient for me at the moment. I can present a paging control that makes sense to my user.

[kevinchalet]

Hey @Jericho,

Thanks a lot ❤️

Your comment made me laugh and cry at the same time because it's so true. The lack of consistency in SendGrid's API is really astounding! There are some many gotchas and one thing that makes my head explode is that, no matter how many times you report these problems, SendGrid doesn't seem interested in fixing any of them. I reported issues 4-5 years ago and they still haven't been fixed. Even issues related to incorrect or incomplete documentation sometimes don't get fixed.

Yeah 😭 Last year, I reported an interoperability issue with their OpenAPI doc that basically makes it useless as you can't generate a typed client from it. One of their employees replied with a "putting it in the backlog" message but they haven't fixed it yet. I suspect there's a very high turnover at Twilio, which may explain why things are so slow to move and so inconsistent. What a shame for a company that makes $1.2B/year in revenue 😠

Maybe you could interact with other developers and introduce them to the library? I do this sometimes on SendGrid's repo but I am always afraid to be perceived negatively since I am promoting a project that I created.

You can count on me to spread the word! 👍🏻

[Jericho]

"putting it in the backlog"

It's so frustrating when they say that because it basically means "stop bothering us". They pretend to have some some of internal issues list but my experience is that nothing ever gets resolved.

Twilio always had a reputation of being developer friendly but SendGrid, not so much. I was hoping the situation would improve after the acquisition but nothing has changed.

You can count on me to spread the word! 👍🏻

Excellent, thank you so much.