Upload Takes More Than 30 Seconds Aws

API Gateway supports a reasonable payload size limit of 10MB. One way to work within this limit, simply still offering a ways of importing large datasets to your backend, is to permit uploads through S3. This commodity shows how to use AWS Lambda to betrayal an S3 signed URL in response to an API Gateway asking. Effectively, this allows you to betrayal a machinery assuasive users to deeply upload data directly to S3, triggered by the API Gateway.

The basic flow of the import process is as follows: the user makes an API, which is served past API Gateway and backed by a Lambda function. The Lambda role computes a signed URL granting upload access to an S3 bucket and returns that to API Gateway, and API Gateway forrard the signed URL back to the user. At this indicate, the user tin use the existing S3 API to upload files larger than 10MB.

Upload through S3 signed URL

In practice, implementing this idea requires several interconnected parts: an S3 bucket, a Lambda function, and the API Gateway. Let's walk through how to create a working upload system using these components.

First, we need an S3 saucepan for storing our data. All objects in S3 are private by default and only the object owner has permission to admission these objects. All the same, the object owner tin can optionally share objects with others past creating a pre-signed URL, using their own security credentials, to grant time-limited permission to upload or download the objects. We are going to take reward of this feature to allow users to upload objects to an otherwise private S3 bucket.

Second, we need a Lambda part that generates pre-signed URLs in response to user API requests. In this example, we will employ the Python AWS SDK to create our Lambda part.

Third, nosotros need to expose our Lambda function through API Gateway. This requires creating a basic API that proxies requests to and from Lambda. We will define this API using Swagger and import it to API Gateway to start serving requests.

The S3 Bucket

The S3 bucket can be created via the AWS user interface, the AWS control line utility, or through CloudFormation. The only requirement is that the bucket be set to allow read/write permission only for the AWS user that created the bucket. This is the default gear up of permissions for any new bucket.

For example, we can hands create a new S3 saucepan using AWS CLI by running the following command:

1	$ aws s3 mb s3://<my-unique-bucket>

The Lambda function

Given your bucket name, you can test you have everything setup correctly by generating a pre-signed URL. In the post-obit case, we use the Python AWS SDK (boto) to generate the signed URL and so use cURL to upload data using a PUT request.

i ii 3 4 5 half dozen seven 8 9 10 xi 12 13 14 fifteen 16

import os import boto3 # Get the service client. s3 = boto3.client('s3') temp_url = s3.generate_presigned_url( ClientMethod= 'put_object', Params={ 'Saucepan': '<my-unique-bucket>', 'Cardinal': '<my-key>' } ) # Upload a file named 'examination.txt' using curlicue os.arrangement('whorl --request PUT --upload-file ./test.txt "' + temp_url + '"')

API Gateway expects responses to be returned as JSON, which corresponds to a Python dictionary. We tin convert this simple test program into a Lambda function usable by API Gateway past returning the signed url as a JSON payload.

1 ii 3 4 v half dozen vii 8 9 10 xi 12 13 xiv 15 16 17 18 19 xx 21 22 23 24 25

import uuid import boto3 def lambda_handler(issue, context): # Get the service client. s3 = boto3.customer('s3') # Generate a random S3 key proper name upload_key = uuid.uuid4().hex # Generate the presigned URL for put requests presigned_url = s3.generate_presigned_url( ClientMethod= 'put_object', Params={ 'Bucket': '<my-unique-bucket>', 'Primal': upload_key } ) # Return the presigned URL render { "upload_url": presigned_url }

Finally, nosotros zip this function and upload it to AWS as a new Lambda role. Assuming your function is named url_signer.py,

            $ zip -r UrlSigner.zip url_signer.py                      

Now, we can create the lambda office. Caution must be taken here: the IAM role assigned to the Lambda part must have read/write access to S3 so that information technology can create the signed URLs, and it must exist assumable past Lambda.

This implies the role must have the post-obit policy granting admission to S3.

ane 2 3 iv v half dozen 7 8 9 10

{ "Version": "2012-10-17", "Statement": [ { "Result": "Permit", "Action": "s3:*", "Resources": "*" } ] }

And the following trust human relationship, which makes the role assumable by Lambda functions.

            {   "Version": "2012-ten-17",   "Statement": [     {       "Effect": "Permit",       "Master": {         "Service": "lambda.amazonaws.com"       },       "Activity": "sts:AssumeRole"     }   ] }                      

With this in place, you can create your Lambda function:

            aws lambda create-function \     --region us-east-1 \     --function-name urlsigner \     --aught-file fileb:///<path-to-cipher>/UrlSigner.zip \     --handler url_signer.lambda_handler \     --runtime python2.7 \     --role <your-arn>                      

API Gateway

At this point, nosotros take an S3 bucket, and a Lambda function that creates signed URLs for uploading to that bucket. The terminal pace is creating the API Gateway frontend that calls the Lambda function. For API Gateway to invoke a Lambda office, y'all must attach a role assumable by API Gateway that has permission to call Lambda'south InvokeFunction action.

This ways you must have a role capable of being assumed by API Gateway with the following trust relationship:

ane two 3 four v 6 seven 8 ix x 11 12 13

{ "Version": "2012-ten-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "apigateway.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

And this function must exist able to invoke Lambda functions using the following policy:

i 2 iii 4 5 6 7 8 9 10

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "lambda:InvokeFunction", "Resource": "*" } ] }

Now we tin define the final API. It is fairly straightforward: we define a single API endpoint that integrates with the Lambda function we previously created, and that has the correct office for executing the Lambda function. This API returns the signed URL for uploading directly to S3.

1 2 3 iv 5 6 vii 8 9 10 11 12 13 xiv 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

--- swagger: '2.0' info: title: Signed URL Import Service clarification: | An import service for uploading data to an S3 saucepan through a signed URL. version: v1 consumes: - awarding/json produces: - application/json schemes: - https definitions: Chore: type: object properties: upload_url: description: A cocky-expiring signed URL for uploading files. type: cord paths: /import/: post: summary: Create a new import chore. responses: 200: description: Created schema: "$ref": "#/definitions/Job" x-amazon-apigateway-integration: type: "aws" uri: "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:united states of america-e-1:820873945423:office:urlsigner/invocations" httpMethod: "POST" credentials: "arn:aws:iam::820873945423:part/ApiGatewayRole" responses: "default": statusCode: "200" responseTemplates: application/json: |- { "upload_url": $input.json('$.upload_url') }

Yous can apply the AWS user interface to create a new API using this definition, and deploying it as an API stage.

The Terminal Outcome

Now you should exist able to make a Mail service request to your import endpoint and be returned a signed URL:

1 2 3 four 5 6 vii viii 9 10

$ http POST https://<...>.execute-api.the states-east-1.amazonaws.com/v1/import HTTP/ane.1 200 OK Connection: keep-alive Content-Length: 609 Content-Type: application/json Date: Tue, 23 May 2017 twenty:37:28 GMT { "upload_url": "https://s3.amazonaws.com/<your-bucket-id>/..." }

You lot tin utilize the signed URL to upload information directly to S3.

Come across also

• The Fake Dichotomy of Pattern-First and Lawmaking-Showtime API Development
• The Cathedral, The Bazaar, and the API Marketplace
• Marrying RESTful HTTP with Asynchronous and Outcome-Driven Services
• Overambitious API gateways
• Comparing Swagger with Austerity or gRPC

campanellicanalountes.blogspot.com

Source: https://sookocheff.com/post/api/uploading-large-payloads-through-api-gateway/