Search Service

Open jonquandt opened this issue 6 years ago • 11 comments

Here's an upcoming service we're thinking about. We'd love to have feedback on any additional features within this service.

Story

As a govinfo api user, I want to be able to get a limited subset of packages or granules based on a query.

Acceptance Criteria

search by keyword or specific search operators (similar to existing govinfo ui)
allow user to specify fields returned (based on availability within index)
field names should follow summary json naming convention
specify whether response should be at package, granule level, or both
default fields returned
- packageId
- granuleId
- lastModified

Examples

FR documents that have "bicycle" in the title, including xml and pdf links
Congressional Record entries for a particular speaking Member

May 01 '18 18:05 jonquandt

Hi we are using the API at the Library of Congress and would like to see the ability:

to filter/search by collection and congress along with lastModified.
to filter the CDOC collection by document type , i.e. filter out just treaties for example.

Jul 20 '18 16:07 relarson1

Keyword search capability would be a terrific feature and would significantly enhance the value of the API.

Jan 30 '19 14:01 eheitfield

@eheitfield -thanks for the input. We're looking to enable similar/equivalent search functionality as is available via the UI. Requires some additional planning and design for consistency's sake.

From an api user standpoint, would it be preferable to keep the syntax for searches to be similar to the UI -- allowing field operators, filters, etc.?

Jan 30 '19 15:01 jonquandt

A search service as proposed by jonquandt would be a big plus. A syntax similar to the existing UI is fine.

Dec 19 '19 14:12 rvschlichtegroll

Yes, please keep the syntax similar to the UI because it'll make research easier.

It would be great to be able to search 1. what committee(s) a BILL is referred too 2. who sponsored/co-sponsored a BILL

Feb 28 '20 06:02 chrisbabayans

The keyword search would be great. Is it available now?

Jul 13 '20 18:07 yanbinghan

@yanbinghan - not yet. We're looking at prioritizing this into a future release. In the meantime, you may be interested in looking at our recent feature allowing for CSV/JSON exports of search results on the UI.

any additional feedback or thoughts on what you would want to see in the search service would be very valuable.

Jul 13 '20 18:07 jonquandt

@jonquandt Thank you for your quick response. Could you please refer me to the link on how to use the recent feature allowing for CSV/JSON exports of search results on the UI?

Jul 13 '20 19:07 yanbinghan

@yanbinghan -- You just need to perform a search. Then in the top right of the search results there is a series of icons that include social media, email, and a link icon. To the right is a download button. Click on that and then choose csv or json

Here's an image that demonstrates the feature: https://www.govinfo.gov/media/dl_search_results.png

Jul 20 '20 11:07 jonquandt

Congressional Record entries for a particular speaking Member

I'd really like to be able to get an API call for this. Am I correct in that it's not available yet, but it's something you're looking to add? Something like:

All Bills introduced by a particular House of Representative member in the 113th Congress
All Bills co-sponsored by a particular House of Rep member in the 116th Congress

Etc.

Dec 18 '20 01:12 krisrobinson

@krisrobinson - yes, that's correct. The search service is still something that's on our roadmap, and any additional examples will be helpful as we develop it.

Dec 18 '20 03:12 jonquandt

Additional example from a user - they would like to be able to pull specific USCOURTS packages by the Nature of Suit metadata.

I am doing some research on the US Court Opinions leverage the API framework. I am writing to see if there is a way to extract the "nature of the suit" metadata in the API responses. If not how can I structure my GET statement to filter by that field. I see the capability and the metadata on the advanced search and response pages but not in the API response.

Mar 21 '23 19:03 jonquandt

We have a goal to deliver some additional information for public feedback over the summer, including:

sample request syntax, including available fields
sample response
response schema

Our initial goal is to provide capabilities that allows the use of a large number of fields currently available in the user interface, with a default set of fields returned for responsive items in the search query.

As we receive feedback, we may make tweaks to the set of fields or other items to address community questions. We are currently targeting Fall 2023 for an initial launch of search functionality in the API.

May 15 '23 17:05 jonquandt

I'm building a tool that will regularly look for any record that fulfills a given Boolean string. Is the planned search function on the API going to enable this to be done? I will need retrieve all the data of records that come up for the search. A manual version of the type of search is below:

https://www.govinfo.gov/app/search/%7B%22query%22%3A%22(PVOH%20OR%20%5C%22polyvinyl%20alcohol%5C%22%20OR%20(PVAL%20AND%20(chemical*%20OR%20compound*%20OR%20substance%20OR%20molecule*))%20OR%20polyethenol%20OR%20%5C%22alcoholysis%20resin%5C%22%20OR%20%5C%22alcohol-soluble%20resin%5C%22%20OR%20%5C%22ethanol%20homopolymer%5C%22%20OR%20elvanol%20OR%20vinol%20OR%20gelvatol)%20%22%2C%22offset%22%3A0%2C%22facetCustomSorts%22%3A%7B%22publishdatehier%22%3Atrue%7D%7D

Jun 27 '23 15:06 RogueSergeant

@RogueSergeant - I am adding this to our list of test cases, but it appears that we will be able to support that kind of request, yes.

Jun 27 '23 16:06 jonquandt

Hi @jonquandt thanks for that, do you have an update on how development is coming along and when we can expect a launch?

Aug 09 '23 14:08 RogueSergeant

@RogueSergeant - We are in the midst of internal testing now. A Public preview version of the search will be available next week for initial feedback. We will post more information on GitHub and GovInfo when it's available.

Aug 09 '23 15:08 jonquandt

@RogueSergeant - Quick sample search and response from a limited dataset test environment. I will provide additional examples when we go live, along with some additional documentation as part of an overview article.

To get the next page of results, you would send the same post request, but pull the offsetMark value from the response and replace * in the request body.

curl --location 'https://api.govinfo.gov/search' \
--header 'Content-Type: application/json' \
--header 'x-api-key: DEMO_KEY' \
--data '{
    "query":"(PVOH OR '\''polyvinyl alcohol'\'' OR (PVAL AND (chemical* OR compound* OR substance OR molecule*)) OR polyethenol OR '\''alcoholysis resin'\'' OR '\''alcohol-soluble resin'\'' OR '\''ethanol homopolymer'\'' OR elvanol OR vinol OR gelvatol)",
    "resultLevel":"default",
    "pageSize":"10",
    "offsetMark":"*",
    "sorts":[
        {
            "field":"score",
            "sortOrder":"DESC"
        }
    ]
}'

Response:

{
    "results": [
        {
            "title": "Polyvinyl alcohol film.",
            "packageId": "CFR-2018-title21-vol3",
            "granuleId": "CFR-2018-title21-vol3-sec177-1670",
            "lastModified": "2023-08-07T05:32:59Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2018-04-01",
            "collectionCode": "CFR",
            "resultLink": "https://api.govinfo.gov/packages/CFR-2018-title21-vol3/granules/CFR-2018-title21-vol3-sec177-1670/summary",
            "dateIngested": "2023-02-09",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/CFR-2018-title21-vol3/premis",
                "xmlLink": "https://api.govinfo.gov/packages/CFR-2018-title21-vol3/granules/CFR-2018-title21-vol3-sec177-1670/xml",
                "zipLink": "https://api.govinfo.gov/packages/CFR-2018-title21-vol3/zip",
                "modsLink": "https://api.govinfo.gov/packages/CFR-2018-title21-vol3/granules/CFR-2018-title21-vol3-sec177-1670/mods",
                "pdfLink": "https://api.govinfo.gov/packages/CFR-2018-title21-vol3/granules/CFR-2018-title21-vol3-sec177-1670/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol From Taiwan: Preliminary Results of Antidumping Duty Administrative Review; 2010-2012",
            "packageId": "FR-2013-04-08",
            "granuleId": "2013-08110",
            "lastModified": "2023-08-06T14:24:07Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2013-04-08",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2013-04-08/granules/2013-08110/summary",
            "dateIngested": "2023-02-07",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2013-04-08/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2013-04-08/granules/2013-08110/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2013-04-08/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2013-04-08/granules/2013-08110/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2013-04-08/granules/2013-08110/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol From China and Japan; Institution of Five-Year Reviews",
            "packageId": "FR-2020-04-01",
            "granuleId": "2020-06718",
            "lastModified": "2023-08-04T21:15:49Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2020-04-01",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2020-04-01/granules/2020-06718/summary",
            "dateIngested": "2023-02-07",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2020-04-01/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2020-04-01/granules/2020-06718/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2020-04-01/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2020-04-01/granules/2020-06718/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2020-04-01/granules/2020-06718/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol From China, Japan, and Korea; Revised Schedule for Full Five-Year Reviews",
            "packageId": "FR-2015-03-12",
            "granuleId": "2015-05599",
            "lastModified": "2023-08-06T10:50:34Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2015-03-12",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2015-03-12/granules/2015-05599/summary",
            "dateIngested": "2023-02-07",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2015-03-12/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2015-03-12/granules/2015-05599/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2015-03-12/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2015-03-12/granules/2015-05599/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2015-03-12/granules/2015-05599/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol From China, Japan, and Korea: Revised Schedule for Full Five-Year Reviews",
            "packageId": "FR-2015-02-05",
            "granuleId": "2015-02286",
            "lastModified": "2023-08-06T11:35:46Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2015-02-05",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2015-02-05/granules/2015-02286/summary",
            "dateIngested": "2023-02-08",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2015-02-05/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2015-02-05/granules/2015-02286/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2015-02-05/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2015-02-05/granules/2015-02286/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2015-02-05/granules/2015-02286/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol from Taiwan: Investigation No. 731-Ta-1088 (Final)",
            "packageId": "GOVPUB-ITC1-PURL-gpo15352",
            "granuleId": null,
            "lastModified": "2023-06-07T12:48:18Z",
            "governmentAuthor": [
                "International Trade Commission"
            ],
            "dateIssued": "2011-01-01",
            "collectionCode": "GOVPUB",
            "resultLink": "https://api.govinfo.gov/packages/GOVPUB-ITC1-PURL-gpo15352/summary",
            "dateIngested": "2023-06-07",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/GOVPUB-ITC1-PURL-gpo15352/premis",
                "zipLink": "https://api.govinfo.gov/packages/GOVPUB-ITC1-PURL-gpo15352/zip",
                "modsLink": "https://api.govinfo.gov/packages/GOVPUB-ITC1-PURL-gpo15352/mods",
                "pdfLink": "https://api.govinfo.gov/packages/GOVPUB-ITC1-PURL-gpo15352/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol (PVA); TSCA Section 21 Petition for Rulemaking; Reasons for Agency Response; Denial of Requested Rulemaking",
            "packageId": "FR-2023-04-27",
            "granuleId": "2023-08864",
            "lastModified": "2023-08-09T15:47:19Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2023-04-27",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2023-04-27/granules/2023-08864/summary",
            "dateIngested": "2023-08-09",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2023-04-27/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2023-04-27/granules/2023-08864/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2023-04-27/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2023-04-27/granules/2023-08864/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2023-04-27/granules/2023-08864/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol From the People's Republic of China: Rescission of Antidumping Duty Administrative Review; 2013-2014",
            "packageId": "FR-2015-04-01",
            "granuleId": "2015-07471",
            "lastModified": "2023-08-05T08:37:16Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2015-04-01",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2015-04-01/granules/2015-07471/summary",
            "dateIngested": "2023-02-07",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2015-04-01/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2015-04-01/granules/2015-07471/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2015-04-01/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2015-04-01/granules/2015-07471/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2015-04-01/granules/2015-07471/pdf"
            }
        },
        {
            "title": "Polyvinyl Alcohol From the People's Republic of China and Japan: Continuation of Antidumping Duty Orders",
            "packageId": "FR-2021-04-09",
            "granuleId": "2021-07303",
            "lastModified": "2023-08-05T18:29:52Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2021-04-09",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2021-04-09/granules/2021-07303/summary",
            "dateIngested": "2023-02-07",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2021-04-09/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2021-04-09/granules/2021-07303/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2021-04-09/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2021-04-09/granules/2021-07303/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2021-04-09/granules/2021-07303/pdf"
            }
        },
        {
            "title": "Notice of Initiation of Antidumping Duty Investigations: Polyvinyl Alcohol From Germany, Japan, the People's Republic of China, the Republic of Korea, and Singapore",
            "packageId": "FR-2002-10-01",
            "granuleId": "02-24928",
            "lastModified": "2023-08-06T19:01:54Z",
            "governmentAuthor": [
                "National Archives and Records Administration",
                "Office of the Federal Register"
            ],
            "dateIssued": "2002-10-01",
            "collectionCode": "FR",
            "resultLink": "https://api.govinfo.gov/packages/FR-2002-10-01/granules/02-24928/summary",
            "dateIngested": "2023-02-08",
            "download": {
                "premisLink": "https://api.govinfo.gov/packages/FR-2002-10-01/premis",
                "txtLink": "https://api.govinfo.gov/packages/FR-2002-10-01/granules/02-24928/htm",
                "zipLink": "https://api.govinfo.gov/packages/FR-2002-10-01/zip",
                "modsLink": "https://api.govinfo.gov/packages/FR-2002-10-01/granules/02-24928/mods",
                "pdfLink": "https://api.govinfo.gov/packages/FR-2002-10-01/granules/02-24928/pdf"
            }
        }
    ],
    "offsetMark": "AoIIRFp5lDZGUi0yMDAyLTEwLTAxLTAyLTI0OTI4",
    "count": 568
}

Aug 10 '23 14:08 jonquandt

Good morning @RogueSergeant @relarson1 @chrisbabayans @yanbinghan @krisrobinson @rvschlichtegroll @eheitfield - the new Search Service is now available as a Public Preview! This means that while it is feature complete as it stands, there is the potential for significant changes based on your feedback.

Read more here: https://www.govinfo.gov/features/search-service-overview

I also updated the readme and added a new Postman collection with example requests and responses.

Hopefully, this initial release will meet most of your needs. If you have feedback on the search service as implemented, please open a new issue so we can triage. As issues and enhancements are triaged, we will add them to milestones and update as changes occur with notice ahead of deployment windows.

At some point in the coming months, we will make the call when the search service is fully production-ready; at that point, we will announce the change in status, and that will signal that the search service request and response formats are essentially stabilized, with only minor additive enhancements anticipated in the future.

Aug 15 '23 15:08 jonquandt

Hi @jonquandt, just been playing around with the API - looks incredibly promising, however, I have started getting a 401 status error. Any idea what that's from?

I switched to the DEMO key and was still getting the same error, so it's not authorization. It could be that I've reached my limit but wouldn't that give me a different error code and shouldn't it then work when I switch to the DEMO key?

Aug 16 '23 13:08 RogueSergeant

@RogueSergeant - let me look to see if I can identify the issue from logs. I'm not able to reproduce using any of the examples that I've tried.

Can you provide an example curl or similar of what you're sending? Are you supplying the DEMO_KEY key via header like this:

curl --location 'https://api.govinfo.gov/search' \
--header 'Content-Type: application/json' \
--header 'x-api-key: DEMO_KEY' \
--data '{
    "query":"(PVOH OR '\''polyvinyl alcohol'\'' OR (PVAL AND (chemical* OR compound* OR substance OR molecule*)) OR polyethenol OR '\''alcoholysis resin'\'' OR '\''alcohol-soluble resin'\'' OR '\''ethanol homopolymer'\'' OR elvanol OR vinol OR gelvatol)",
    "resultLevel":"default",
    "pageSize":"1000",
    "offsetMark":"*",
    "sorts":[
        {
            "field":"score",
            "sortOrder":"DESC"
        }
    ]
}'

or as a query parameter?

curl --location 'https://api.govinfo.gov/search?api_key=DEMO_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "query":"(PVOH OR '\''polyvinyl alcohol'\'' OR (PVAL AND (chemical* OR compound* OR substance OR molecule*)) OR polyethenol OR '\''alcoholysis resin'\'' OR '\''alcohol-soluble resin'\'' OR '\''ethanol homopolymer'\'' OR elvanol OR vinol OR gelvatol)",
    "resultLevel":"default",
    "pageSize":"1000",
    "offsetMark":"*",
    "sorts":[
        {
            "field":"score",
            "sortOrder":"DESC"
        }
    ]
}'

Aug 16 '23 14:08 jonquandt

@jonquandt - thanks!

I'm using a python script which WAS working earlier, and so I'm not sure why it's not working now.

I also tried visiting the URL search direct with the following and got a similar response: https://api.govinfo.gov/search?API_KEY=DEMO_KEY&query=%22test%22

{"message":"Oops, Something went wrong, Please contact govinfo team for further assistance"}

Here's the script:

import requests
import json
import re

url = "https://api.govinfo.gov/search"
api_key = "DEMO_KEY"
query = input("Paste your Boolean here: ")

# this is not the most efficient way to do this, but works until I can get around to condensing this section:
query = query.replace(' "',r" '\''")
query = query.replace('" ',r"''\' ")
query = query.replace('")',r"''\')")
query = query.replace(r'("',r"('\''")
query = re.sub(r'^"|"$', r"'\''", query)

print(query)

headers = {
    "Content-Type": "application/json",
    "Accept": "application/json",
    "apikey": api_key
}

data = {
    "query": f"{query}",
    "pageSize": "10",
    "offsetMark": "*",
    "sorts": [
        {
            "field": "publishdate",
            "sortOrder": "DESC"
        }
    ]
}

response = requests.post(url, data=json.dumps(data), headers=headers)

if response.status_code == 200:
    print("Success!")
    results = json.loads(response.content)['results']
    count = json.loads(response.content)['count']
    remainingCalls = json.loads(response.content)['X-RateLimit-Remaining']
    for result in results:
        print(f"\nTitle: {result['title']}")
        print(f"Package ID: {result['packageId']}")
        print(f"Granule ID: {result['granuleId']}")
        print(f"Last Modified: {result['lastModified']}")
        print(f"Government Author: {result['governmentAuthor']}")
        print(f"Date Issued: {result['dateIssued']}")
        print(f"Collection Code: {result['collectionCode']}")
        print(f"Result Link: {result['resultLink']}")
        print(f"Downloads:")
        for link_type, link in result['download'].items():
            print(f"  {link_type}: {link}?API_KEY={api_key}")
    print(f"Total results: {count}")
    print(f"Remaining calls: {remainingCalls}")
else:
    print(f"Failed! Status code: {response.status_code}")

Aug 16 '23 14:08 RogueSergeant

@RogueSergeant - you need to set the api key header with x-api-key, not apikey:

headers = {
    "Content-Type": "application/json",
    "Accept": "application/json",
    "x-api-key": api_key
}

You could also have the failure return the response body as part of your else block, which would return more details: print(f"Here's the response message: {response.content}")

which will return: Here's the response message: b'{\n "error": {\n "code": "API_KEY_MISSING",\n "message": "No api_key was supplied. Get one at https://api.data.gov/signup"\n }\n}'

Also, It looks like the remainingCalls variable will throw an error because the X-RateLImit-Remaining is part of the response headers and not the content.

Aug 16 '23 14:08 jonquandt

I also tried visiting the URL search direct with the following and got a similar response: https://api.govinfo.gov/search?API_KEY=DEMO_KEY&query=%22test%22 The search service doesn't support GET requests - there was a design change during development.

If that's something that's desirable, please create a new issue and we can triage if there's enough community support for it.

Aug 16 '23 14:08 jonquandt

@jonquandt Got it working! Not sure how I managed to change the api parameter between runs...!

As for the GET requests - not necessary at all, just wanted to check. Thanks for the help

Aug 16 '23 14:08 RogueSergeant

I am closing this issue now, as the service is now available. Please create new issues if you see any problems or potential enhancements to the search service.

Aug 18 '23 14:08 jonquandt

api api copied to clipboard

Search Service

Story

Acceptance Criteria

Examples

api
api copied to clipboard