Nikki's Lunch

 I had the issue where Lambda function launched in private network times out when trying to get object from S3 bucket. Typically, there are two solutions: Use S3 VPC endpoint (either gateway or interface) since it resolves s3 endpoint to private IP. Attach public IP. This is done using NAT Gateway with Elastic IP (EIP). The problem is, in my case, the S3 bucket is in different region, different account, than the Lambda function while the first solution, even though S3 is a global service, the VPC endpoint can't resolve to S3 in different region. In short, the first solution only works when S3 bucket and Lambda function are in the same region. That left us with solution 2 which is more expensive but works. Also I need to make sure that the S3 bucket policy allows cross account access.

So, I have to launch Opensearch in ECS. And I need to add persistent storage. The container ran fine but it threw AccessDeniedException. And even though the container ran, my application was unable to connect to it.  After few tries, I found out that it is due to the permission of the directory where the data are supposed to reside. The container runs in ECS on EC2. The path, in this case, I use /usr/share/opensearch/data on EC2 is owned by root, but the container runs as ec2-user. So, I had to update the user data field on the launch template (since I used ASG) to include the following commands: mkdir -p /usr/share/opensearch/data sudo chown 1000:1000 /usr/share/opensearch/data That fixed the exception and the reachability issue.