Large Searches
While the normal shipment and company endpoints are suitable for result sets up to 5000 records, there are use-cases which require large sets of record to be returned.
In these situations, using the provided async endpoints will allow for the retrieval of these large result sets in an efficient manner.
The async endpoints operate by receiving a particular search, or a set of searches, and compiles the results in the background.
When ready, the entire set of results is downloaded as a single file.
The async endpoints are not always available by default, please reach out to your Panjiva representative to request access.
Types of Large Searches
| Type | Limit | Description |
|---|---|---|
| Bulk | 1,000,000 Records | Receives a single search request and compiles the full set of records that match the provided request |
| Batch | 50,000 Requests | Receives multiple search requests and compiles the results with the same limit for each request as the normal endpoint |
Even though the purpose of async searches is to produce result sets beyond the normal limits, there are still restrictions to the amount of data that can be processed.
Please reach out to your Panjiva representative if additional data is needed.
Format of the response
The compiled results of the search are composed in either a JSON-lines format, or a CSV format.
If the results are to be processed programmatically, specifying the jsonl format is best.
Otherwise, using a csv format allows for integration with typical spreadsheet tools.
When is the search complete?
Each async endpoint accepts a notify_email parameter, which will be emailed with the download URL when the search results have been finalized.
Alternatively, the status and download URL of the file is also available programmatically through the status endpoint.
Each async search has a unique job_id which is returned in the response to the initial request.
Use this job_id to specify the exact search in the status endpoint.