Getting FailedToFetch error for DynamoDBClient QueryCommand

[renchix]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Fetching data from DynamoDB through QueryCommand ocasionally results in FailedToFetch error. Error persists temporarily and comes and goes periodically. First cases were observed in Oct 31, 2024 and Nov 1, 2024. And again today.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Browser

Details of the browser/Node.js/ReactNative version

Microsoft Edge

Reproduction Steps

        const command = new QueryCommand(input);
        response = await client.send(command);

Observed Behavior

Code has been working for at least a year, no changes have been made recently.
First idea was that it some how related to CORS, but I am not sure if this can somehow be configured for DynamoDBClient.

Expected Behavior

DyanmoDb results should be fetched without raising error.

Possible Solution

No response

Additional Information/Context

No response

[renchix]

It is CORS issue, but the question is how comes it hasn't been before and how can it be resolved:

Access to fetch at 'https://dynamodb.eu-north-1.amazonaws.com/' from origin 'https://XXXX.XXXX.XXX has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

[zshzbh]

Hey @renchix ,

There's some potential root causes:

1. Network condition : Intermittent network issues or latency spikes could cause requests to timeout or fail, which might trigger CORS errors in some browsers.
2. AWS Service Availability: While rare, there could be intermittent issues with AWS services in your region if the DDB service or its associated auth services have brief outages or performance issues.
3. Browser Caching: Browsers can cache CORS preflight responses. If a successful request is cached, subsequent requests might work, while new requests or requests after the cache expires might fail. ref: https://repost.aws/questions/QUt1qUofacSDKawoShV1fgKA/api-gateway-method-cors-error-cloudformation. And there's a stack overflow topic regarding this issue.
   To Implement Retry Logic: Add a retry mechanism for failed requests. This can help mitigate intermittent issues.
4. DNS Resolution
5. Load Balancing: If AWS is using multiple servers or load balancers for the DynamoDB service, some might be configured correctly for CORS while others are not.

To identify the patterns in the failures, you can implement detailed logging to include the timestamps and full request details and response headers.

To help potentially resolve this issue, you can apply retry logic:

const MAX_RETRIES = 3;
const RETRY_DELAY = 1000; 

This can help mitigate intermittent issues.

Thanks!
Maggie

[renchix]

Thanks for the detailed response.

To my understanding default in-built retry mechanism () already makes 3 attempts after failed request, so it seems that this doesn't solve the issue as it should already be working.

Caching errors seems to be a possible cause, however, as we have seen cases where even after trying to fully refresh page for several times error still persists, then this also doesn't feel as the likely cause.

I could hope that this ir really some temporary issue - the fact that 99.9% of time everything works correctly also kind of supports this logic. However, I am still puzzled about the fact that in the last week there have been multiple such cases, without anything similar before.

Also, in those cases where DynamoDBClient keeps failing Lambda Client works without issues (I understand that there the logic is a bit different - but still it goes under aws-sdk which then makes causes #1 and #4 as less likely) - this also make me feel that this is some issue specific to DynamoDBClient.

[zshzbh]

Thanks for the updates! I agree, and also it's hard to troubleshoot to find the root cause as it works well in 99.9% of time. I have asked Dynamo DB team and see if there's any known issue happened last week in your region.

[renchix]

Still getting the same error every day or two:

monitor:1 Access to fetch at 'https://dynamodb.eu-north-1.amazonaws.com/' from origin 'https://www.xxxxxxx.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
dynamodb.eu-north-1.amazonaws.com/:1

    Failed to load resource: net::ERR_FAILED

main.535af7c7.js:2

    Uncaught (in promise) TypeError: Failed to fetch
at Vv.handle (main.535af7c7.js:2:1408941)
at async main.535af7c7.js:2:1317124
at async clientCommand.middlewareStack.addRelativeTo.name (main.535af7c7.js:2:1427841)
at async main.535af7c7.js:2:1318306
at async main.535af7c7.js:2:1362663
at async main.535af7c7.js:2:1314299
at async R (main.535af7c7.js:2:3258575)

Is there some way to force those headers for DynamoDBClient request in our React code?

[renchix]

To me it seems that only solution then is to do all requests through API gateway (not directly to DynamoDB as it is now)
That obviously would mean that there is one extra intermediary step involved as well as additional costs for API gateway use, and a lot of code rewriting. This doesn't sound good.

[renchix]

To add maybe a little bit more context for DynamoDB team - even if QueryItems fails and then problem persists for some time - at the same time GetItem requests are fulfilled.

[renchix]

After some additional investigation I found that preflight request is made and received successfully, however the real request in those failed cases is missing 'Remote address' in the Headers.
This is how failed one looks:

and this is successful request:

Preflight requests for both of them seem to be identical.

@zshzbh any suggestions?

[zshzbh]

Hey @renchix ,

I can't reproduce this error. Could you please provide any pre-condition and minimal code reproduction so we can better troubleshoot this issue?

[renchix]

There is not much of code to provide.
This basically is all that leads to the error:

const client = new DynamoDBClient({region: region, credentials: credentials, maxRetries: 5 });
const command = new QueryCommand(input);
const response = await DynamoDBDocumentClient.from(client).send(command)

As error usually comes by surprise, I cannot reliably reproduce it as well.
When it has happened I have inspected the http requests and missing Remote address field in the Headers, mentioned in my previous post was only thing that seemed out of normal in all of these cases.