# Streamlining Data Flow: Integrating SFTP User Access with S3, SQS, and VM Directory Mounting

<details data-node-type="hn-details-summary"><summary>Introduction :</summary><div data-type="detailsContent">In the blog post, we can explore how to set up a seamless data flow by integrating various components such as SFTP user access, S3 storage, SQS queues, and VM directory mounting. We'll discuss the steps involved in configuring SFTP users with limited access to specific directories, mounting S3 buckets to a virtual machine (VM) directory, and utilizing SQS for event notification.</div></details>

### Pre-requisite

* Create s3 bucket.
    
* Create sqs queue.
    
* create ec2 instance (ubuntu)
    
* create IAM User for accessing s3 bucket and add s3 permission to it.
    

### Creating user for sftp

Install s3fs

```bash
sudo apt update
sudo apt install s3fs
```

Setup creds for s3fs

Create file

```bash
sudo vim /etc/passwd-s3fs

# write below content with its actual value for iam user created
ACCESS_KEY_ID:SECRET_KEY_ID
```

create user for sftp

```bash
#!/bin/bash

# Usage: sudo ./create-sftp-user.sh username

USERNAME=$1
S3_BUCKET_NAME="s3-bucket-name"
S3_BUCKET_PATH="$S3_BUCKET_NAME:/incoming/$USERNAME"
MOUNT_DIR="/mnt/sftp-s3/$USERNAME"

sudo useradd -m -d "$MOUNT_DIR" -s /bin/bash -G sftponly "$USERNAME"
# Set password for the user
sudo passwd "$USERNAME"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
sudo mkdir -p "$MOUNT_DIR/data"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR/data" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
echo "User $USERNAME added and configured."
```

`sudo vim /etc/ssh/sshd_config`

at last write

```bash
Match User userName
    ChrootDirectory /mnt/sftp-s3/$USERNAME
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
    PasswordAuthentication yes
```

now try to access over sftp .

### Update sqs queue access policy :

```bash
{
  "Version": "2012-10-17",
  "Id": "Policy1679925546977",
  "Statement": [
    {
      "Sid": "Stmt1679925532180",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:*",
      "Resource": "arn:aws:sqs:region:1111111111:sqs_name",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::s3-buket-name"
        }
      }
    }
  ]
}
```

Create s3 event notification.

Thanks

For more such content follow me on :

Twitter : `https://x.com/akash202k_`