Paging vs Streaming

The b.well FHIR server allows two methods to get large amounts of data efficiently:

Paging
Streaming

How to use Paging

Paging with the b.well FHIR server can be done in two ways:

Using Page Numbers: You can specify the desired page number in your request to retrieve a specific set of results.

For example, to get the second page of patients:
```
GET /Patient?_page=2&_count=10
```
Using the next link: The FHIR Bundle returned by the server will include a next link. You can use this link to fetch the subsequent page of results, providing a convenient way to navigate through large datasets.

After an initial request, the server might return a bundle with a next link:
```
{
  "resourceType": "Bundle",
  "link": [
    {
      "relation": "next",
      "url": "https://example.com/fhir/Patient?_page=2&_count=10"
    }
  ],
  "entry": [...]
}
```
You would then use the URL provided in the next link for the subsequent request.

How to use Streaming

The b.well FHIR Server also supports streaming data, which offers several benefits for efficient data retrieval.

Benefits for Clients:

Clients can receive streamed data using a single connection.
Utilizes HTTP Keep-alive, reducing network overhead per call.
Clients begin receiving data quickly, allowing for processing while more data arrives.

Benefits for Server:

Reduced memory usage as the server doesn't need to load the entire result set into memory before sending.
The cursor is maintained for the entire request, preventing MongoDB from re-executing searches for subsequent chunks.

HTTP Chunk Transfer: FHIR Server streaming employs HTTP Chunk Transfer to send data efficiently.

Client Implementation:

Open Connection: The client opens a connection and sends a request, similar to a non-streaming scenario.
Process Chunks: The server sends data in chunks, which the client processes as they are received, waiting for the next one.

Clients can utilize the chunk transfer mechanism available in their HTTP client libraries. For example, using Python's requests library (async):

async with ClientSession(timeout=ClientTimeout(total=0)) as http:
    async with http.request("GET", fhir_server_url, headers=headers, data=payload, ssl=False) as response:
        async for line in response.content:
            print(f"{line }", end='''\r''')

For a complete example, refer to: https://github.com/icanbwell/fhir-server-performance/blob/main/simple_with_progress.py

NDJSON Format: Streaming is supported for both standard FHIR JSON and NDJSON formats. NDJSON (Newline Delimited JSON) is an optimized format for streaming, as each resource resides on a separate line. It is also a supported format for FHIR Bulk API.

To request the NDJSON format, include the following Accept header in your client request:

Accept: application/fhir+ndjson

MongoDB Query Timeout Retry Logic: The server incorporates a retry mechanism for MongoDB query timeouts, managed by two environment variables:

MONGO_TIMEOUT: Time in milliseconds for non-streaming requests.
MONGO_STREAMING_TIMEOUT: Time in milliseconds for streaming requests.

An initial query timeout is subjected to MONGO_TIMEOUT. If the first batch exceeds this time, a 500 error is returned. If the initial query completes within 2 minutes and streaming is identified, the timeout is updated to MONGO_STREAMING_TIMEOUT to allow the streaming process to continue for an extended period.