Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limiting long graphql query times #3

Open
liamphmurphy opened this issue Feb 11, 2022 · 1 comment
Open

Limiting long graphql query times #3

liamphmurphy opened this issue Feb 11, 2022 · 1 comment

Comments

@liamphmurphy
Copy link

liamphmurphy commented Feb 11, 2022

At the moment, the GraphQL API can hang and cause a long query time under two known conditions:

  1. A query is requested with a long delta between the startTime and endTime, so the processing of the data from S3 can take a few minutes (more?). This could be confusing to the user and from our point of view, could have performance repercussions / extra cost by pulling down a lot of S3 data.
    Note: Based on this pricing table, there may not be a cost associated per GB of data retrieved, but we could at least stop an unneeded request to S3 before it happens.

2.If the API is unable to connect to S3, the API can hang. May be worth returning an error back to the user saying that a connection to S3 couldn't be made.

Action Items (corresponding to the above list):

  1. Look into ways to predict when a query may cause an overly long processing time. Having a limit on the start / end time is obvious, but I'd want to make sure this wouldn't reject any "legit" queries. If we want to get fancier, could setup a timeout?
  2. Assuming the behavior with opentransit-state-api -> S3 is actually happening, return an error message when an S3 connection error occurs and terminate the query.
@youngj
Copy link
Collaborator

youngj commented Feb 11, 2022

For issue 2 if there are problems with the connection to S3, I think the timeout would be determined by the httpOptions.connectTimeout and httpOptions.timeout parameters to the AWS.S3 constructor documented at https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#constructor-property . Maybe it would be better to set those timeouts to ~10 seconds so that the API returns an error message in a reasonable amount of time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants