I have been performing some data transformations using Lambda and S3 Events and for certain S3 keys I noticed that I was getting an Error 403 in my code:
[ERROR] ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden Traceback (most recent call last): File "/var/task/index.py", line 159, in handler if s3_obj.content_length < (400 * MB): File "/var/runtime/boto3/resources/factory.py", line 339, in property_loader self.load() File "/var/runtime/boto3/resources/factory.py", line 505, in do_action response = action(self, *args, **kwargs) File "/var/runtime/boto3/resources/action.py", line 83, in __call__ response = getattr(parent.meta.client, operation_name)(**params) File "/var/runtime/botocore/client.py", line 320, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 623, in _make_api_call raise error_class(parsed_response, operation_name)
The interesting thing that came from this was that the error seemed to specifically happen on files that contained special (i.e. non-alphanumeric) characters. Research indicated that the characters were actually valid - see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html - so it isn’t an issue with the key name itself.
It turns out that the issue is with the fact that when the S3 Event returns the queue it is URL encoded but when you attempt to use the key in boto3 then it fails. The solution is to do the following:
from urllib import parse import boto3 S3_RESOURCE = boto3.resource('s3') def handler(event, context): # pylint: disable=unused-argument """ Lambda entry point. """ for record in event["Records"]: # Lambda struggles to process a file over 400Mb, so am not going to process files larger # than that. bucket = record["s3"]["bucket"]["name"] key = parse.unquote(record["s3"]["object"]["key"]) s3_obj = S3_RESOURCE.Object(bucket, key)
s3_obj above will then start working.