If you are moving to AWS Identity Center (formerly SSO) and want to keep your Google Workspace users and groups in sync, ssosync is the go-to tool. However, setting up the permissions can be tricky.

If you’ve seen the error unauthorized_client: Client is unauthorized to retrieve access tokens, you are likely missing a specific scope or delegation step.

Here is the easy way to set it up from scratch.

Step 1: Create your Google Cloud Credentials

1. Go to the Google Cloud Console.

2. Create a New Project (e.g., "AWS-SSO-Sync").

3. Enable the Admin SDK API: Search for it in the Library and click Enable.

4. Create a Service Account:

   • Go to IAM & Admin > Service Accounts.

   • Create an account, name it ssosync-service-account.

   • Once created, click on the account, go to the Keys tab, and Add Key > Create new key (JSON).

   • Save this file! You will need to base64 encode this for your .env file.

   •     # Replace file name `ssosync-486114-bb1ce7f2ebb9.json` with your file name.
         cat ssosync-486114-bb1ce7f2ebb9.json | base64 | tr -d "\n" | pbcopy
     
         # And result of this we need to store in GOOGLE_CREDENTIALS_JSON variable in .env file
     

5. Copy the Unique ID: Back on the Service Account details page, copy the Unique ID (a long string of numbers). You'll need this for the next step.

Step 2: Grant "Domain-Wide Delegation"

This is the part most people miss. You have to tell Google Workspace that this Service Account is allowed to "read" your organization's data.

1. Open the Google Admin Console.

2. Go to Security > Access and data control > API controls.

3. Click Manage Domain Wide Delegation.

4. Click Add new and enter:

   • Client ID: Paste the Unique ID you copied in Step 1.

   • OAuth Scopes: Copy and paste this exact list: https://www.googleapis.com/auth/admin.directory.group.readonly, https://www.googleapis.com/auth/admin.directory.group.member.readonly, https://www.googleapis.com/auth/admin.directory.user.readonly

5. Click Authorize.

Step 3: Get your AWS SCIM Details

1. Open the AWS Identity Center console.

2. Go to Settings > Provisioning.

3. Enable Automatic Provisioning.

4. Copy the SCIM endpoint and the Access token. (Keep the token safe; it only shows once!)

Step 4: Configure your Environment

Create a .env file. To get your GOOGLE_CREDENTIALS_JSON, run cat your-key.json | base64 in your terminal and paste the result.

Bash

# .env example

# 1. Base64 encoded service account JSON key
GOOGLE_CREDENTIALS_JSON=your_base64_string_here

# 2. A real Admin email address from your Google Workspace
GOOGLE_USER=admin@yourdomain.com

# 3. AWS SCIM Details
SCIM_ENDPOINT=https://scim.us-east-1.amazonaws.com/xxx/scim/v2/
SCIM_TOKEN=your_secret_token

# 4. AWS Identity Store & Region
AWS_IDENTITY_STORE_ID=d-123456789
AWS_DEFAULT_REGION=us-east-1

# 5. AWS Access Keys (to talk to the AWS API)
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...

Step 5: Run the Sync

Now you can run the sync command. Bash

ssosync \
  -e $SCIM_ENDPOINT \
  -t $SCIM_TOKEN \
  -u $GOOGLE_USER \
  -i $AWS_IDENTITY_STORE_ID \
  -r $AWS_DEFAULT_REGION \
  --debug

Note : You can refer official github repo ssosync to know about more sync commands

Groups in Google Workspace

Groups In AWS Identity Center

Admin Group Members in Google Workspace

Admin Group Members in AWS Identity Center

Dev Group Members in Google Workspace

Dev Group Members in AWS Identity Center

Why this works:

By adding the admin.directory.group.readonly and admin.directory.user.readonly scopes in the Google Admin panel, you are explicitly giving ssosync the permission to "look" at your users. Without this, Google rejects the connection (401 Unauthorized), even if your password/key is correct!

In this tutorial, I will guide you through establishing a SAML connection between Google Workspace and AWS Identity Center. Later, you will synchronize users from Google Workspace using SCIM. To verify that everything is configured correctly, you will sign in as a Google Workspace user and verify access to AWS resources. This tutorial is based on a small Google Workspace directory test environment and does not include directory structures such as groups and organisational units. After completing this tutorial, your users will be able to access the AWS access portal using their Google Workspace credentials by altering your Identity Center to an external identity provider.

Understanding IAM Identity Center

IAM Identity Center is the recommended AWS service for managing human user access to AWS resources. It provides a single place to manage users, groups, and consistent access to multiple AWS accounts and applications. The best part is that IAM Identity Center is offered at no additional cost.

Why Integrate Google Workspace with IAM Identity Center?

Integrating Google Workspace with AWS SSO streamlines user management and enhances security. It allows for seamless single sign-on, reducing the need for multiple passwords and improving user convenience. Additionally, it centralizes access control, making it easier to enforce security policies and manage permissions across both platforms.

How It Works

User information from Google Workspace is synchronized into IAM Identity Center using the System for Cross-domain Identity Management (SCIM) v2.0 protocol. You configure this connection in Google Workspace using your SCIM endpoint for IAM Identity Center and an IAM Identity Center bearer token. This involves setting up Google Workspace as an IAM identity provider and an IAM Identity Center identity provider.

Prerequisites

1. Access to the Google Workspace admin portal to configure the SAML app.

2. Access to IAM Identity Center.

Considerations

1. Before configuring SCIM provisioning, review the considerations listed here.

2. Currently SCIM automatic synchronisation from Google Workspace is limited to user provisioning. Automatic group provisioning is not supported at this time. Group creation and user management will be covered in the next tutorial.

Steps to Configure the SAML Application

Step 1: Configure the SAML Application in Google Workspace

• Sign in to your Google Admin Console using an account with administrator permissions.

• Navigate to Apps → Web and Mobile Applications.

• From the Add App dropdown, search for “Amazon Web Services” and select the Amazon Web Services (SAML) app from the list.

• Download the IdP metadata.

• Leave this page open and move to the IAM Identity Center console.

Step 2: Enable IAM Identity Center

• Go to the IAM Identity Center console page and select the Enable button if it is not enabled.

  

• Select your organization to allow access to multiple AWS accounts using IAM Identity Center.

• After a few minutes, the IAM Identity Center will be ready to use.

• In the left navigation pane, choose Settings.

• On the Settings page, choose Actions and then Change Identity Source.

  

• On the Choose Identity Source page, select External Identity Provider, and then choose Next.

• 

• On the Configure External Identity Provider page, complete the following:

• Upload the Google SAML metadata as the IdP SAML metadata in the IAM Identity Center console.

  

• Confirm the change and click Next, then ACCEPT the Change request.

  

• After providing the Google metadata, copy the AWS access portal sign-in URL, IAM Identity Assertion Consumer Service (ACS) URL, and IAM Identity Center issuer URL. Provide these URLs in the Google Admin console.

  

• On the next page, you will get the SAML Authentication metadata.

• Map the AWS URLs in Google Workspace.

  

• On the Service Provider Details page, complete the fields under Name ID:

• For Name ID format, select EMAIL.

• For Name ID, select Basic Information > Department

• Choose Finish.

• On the Attribute Mapping page, choose ADD MAPPING and configure these fields under Google Directory attribute.

• Choose Finish.

Step 3: Google Workspace: Enable the App

• Return to the Google Admin Console and locate the AWS IAM Identity Center application under Apps → Web and Mobile Apps.

• In the User Access panel, expand User Access to display the Service Status panel.

• In the Service Status panel, choose ON for everyone, and then choose SAVE.

Step 4: Set Up IAM Identity Center Automatic Provisioning

• Return to the IAM Identity Center console.

• On the Settings page, enable Automatic provisioning.

• Copy the SCIM endpoint and Access token information displayed.

• Close the dialog box.

Step 5: Configure Auto Provisioning on Google Workspace

• Return to the Google Admin Console, Select Apps → Web and Mobile Apps → Amazon Web Services (SAML)

• In the Auto Provisioning section, choose Configure Auto Provisioning.

• Paste the Access token and SCIM endpoint values copied earlier.

• Verify that all mandatory IAM Identity Center attributes are mapped to Google Cloud Directory attributes.

• In the Provisioning Scope section, optionally choose a group to provide access to the Amazon Web Services app.

• In the Deprovisioning section, specify how to respond to different events that remove access from a user.

• Choose Finish and turn on the auto-provisioning toggle switch.

• To verify user synchronization, return to the IAM Identity Center console → Users.

Access AWS Management Console using Google Workspace

• Go to IAM Identity Center → Settings → Identity Source

  

• Click on the AWS access portal URL

• This will open a window and then you can select your Google Workspace ID.

Conclusion

You have successfully set up a SAML connection between Google Workspace and AWS and verified that automatic provisioning is working.

Applications that require coordinated multi-pod execution:

• Distributed ML Training: Multi-GPU model training (PyTorch DDP, TensorFlow Distributed)

• High-Performance Computing: Weather simulation, molecular dynamics

• Parallel Data Processing: Large-scale ETL with coordinated workers

• Multi-node Databases: Distributed database initialization

The Pain Point: Traditional Kubernetes schedules pods individually. For a 4-GPU training job:

• Pod 1 starts → claims 1 GPU

• Pods 2-4 wait indefinitely for resources

• Result: $8/hour burning while GPUs sit idle

Our Demo: Simulates distributed TensorFlow training requiring 2 GPUs across 2 nodes.

Repository: nvidia-gpu-volcano-k8s ⭐

The Problem

Traditional Kubernetes scheduling with expensive GPU workloads:

Job needs: 2 GPUs total
Available: 2 GPUs across 2 nodes

❌ Standard Scheduler:
Pod 1: Scheduled immediately (claims 1 GPU)
Pod 2: Waits for "sufficient resources" 
Result: 1 GPU idle, job fails

✅ Volcano Gang Scheduler: 
Both pods: Wait until 2 GPUs available
Then: Start simultaneously
Result: Efficient resource utilization

Implementation

1. GPU Node Setup

# GPU-enabled AMI with pre-installed NVIDIA drivers
eksctl create nodegroup \
  --node-type g4dn.xlarge \
  --nodes 2 --spot \
  --node-ami AL2_x86_64_GPU

2. GPU Support

# Install NVIDIA device plugin
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.1/nvidia-device-plugin.yml

# Setup GPU node taints
chmod +x gpu_node_setup.sh
./gpu_node_setup.sh

From gpu_node_setup.sh:

#!/bin/bash
echo "Setting up GPU nodes for Volcano demo..."

# Get GPU node names
GPU_NODES=$(kubectl get nodes --no-headers | grep g4dn | awk '{print $1}')

# Apply standard NVIDIA GPU taints
echo "Applying NVIDIA GPU taints..."
for node in $GPU_NODES; do
    kubectl taint nodes $node nvidia.com/gpu=present:NoSchedule --overwrite
    echo "Tainted node: $node"
done

3. Install Volcano

helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace

4. Gang Scheduled Job

From manifests/tensorflow-job.yaml:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: gang-tf-job
spec:
  minAvailable: 2          # Critical: Both pods or none
  schedulerName: volcano   # Use Volcano instead of default
  queue: ml-queue
  tasks:
    - replicas: 2
      name: trainer
      template:
        spec:
          tolerations:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: "node.kubernetes.io/instance-type"
                    operator: In
                    values: ["g4dn.xlarge", "g4dn.2xlarge", "g5.xlarge"]
          containers:
            - name: trainer
              image: akash202k/tf-volcano-demo:v1
              resources:
                requests:
                  cpu: "2"
                  memory: "4Gi"
                  nvidia.com/gpu: 1
                limits:
                  cpu: "2"
                  memory: "4Gi"
                  nvidia.com/gpu: 1

Demo: Pain Point Simulation

Normal Gang Scheduling

kubectl apply -f manifests/volcano-queue.yaml
kubectl apply -f manifests/tensorflow-job.yaml
kubectl get pods -l volcano.sh/job-name=gang-tf-job -w

Result: Both pods start simultaneously ✅

Resource Contention (The Pain Point)

From manifests/gpu-blocker-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: blocker
spec:
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: stress
      image: nvidia/cuda:11.0-base
      command: ["/bin/sh"]
      args: ["-c", "sleep 60"]
      resources:
        requests:
          nvidia.com/gpu: 1

# Simulate real-world contention - block 1 GPU
kubectl apply -f manifests/gpu-blocker-pod.yaml

# Deploy gang job (needs 2 GPUs, only 1 available)
kubectl apply -f manifests/tensorflow-job.yaml

Traditional scheduler would: Start 1 pod, waste resources
Volcano gang scheduler: Keeps both pods Pending until sufficient resources ✅

Resource Release

kubectl delete pod blocker

Result: Both pods start together immediately ✅

Results Achieved

Gang Scheduling Success

NAME                    READY   STATUS      RESTARTS   AGE
gang-tf-job-trainer-0   0/1     Completed   0          3m46s
gang-tf-job-trainer-1   0/1     Completed   0          3m46s

GPU Training Confirmed

From our actual training logs in logs/volcano_training_success_20250726_045527_d4ead352.json:

Pod 1 (ap-southeast-1c):

{
  "aws_metadata": {
    "instance_id": "i-0bd485d90b62fcc01",
    "instance_type": "g4dn.xlarge",
    "availability_zone": "ap-southeast-1c"
  },
  "gpu_info": {
    "gpu_details": [{
      "details": {
        "compute_capability": [7, 5],
        "device_name": "Tesla T4"
      }
    }],
    "nvidia_smi": [{
      "name": "Tesla T4",
      "memory_total_mb": "15360",
      "memory_used_mb": "103",
      "utilization_percent": "0",
      "temperature_c": "26"
    }]
  },
  "training_results": {
    "device_used": "/GPU:0",
    "final_loss": 0.08402693271636963,
    "status": "success"
  }
}

Pod 2 (ap-southeast-1b) from logs/volcano_training_success_20250726_045528_65078d2b.json:

{
  "aws_metadata": {
    "instance_id": "i-0edff736d5e7d3a59",
    "instance_type": "g4dn.xlarge", 
    "availability_zone": "ap-southeast-1b"
  },
  "training_results": {
    "device_used": "/GPU:0",
    "final_loss": 0.08474162966012955,
    "status": "success"
  }
}

Perfect coordination: Both pods trained on separate Tesla T4 GPUs simultaneously across different AZs.

Pain Point Solved

Why Distributed Training Needs Gang Scheduling

Single Machine Limitations:

• Large models can't fit in single GPU memory (e.g., LLaMA, GPT models)

• Training time becomes prohibitive (weeks vs days)

• Memory constraints limit batch sizes and model complexity

Distributed Training Requirements:

• All workers must start together for synchronized gradient updates

• Coordinated parameter sharing across nodes

• Consistent training state - partial deployments corrupt the training process

Before Gang Scheduling

• Resource waste: Partial deployments burn GPU time while waiting for missing workers

• Training failures: Incomplete worker allocation breaks distributed algorithms

• Cost inefficiency: $2-3/hour per idle GPU waiting for coordination

• Model corruption: Partial worker sets produce invalid gradients

After Gang Scheduling

• Resource efficiency: 100% GPU utilization across all workers simultaneously

• Training reliability: All workers start together, ensuring proper distributed training

• Cost control: No GPU time wasted on incomplete worker deployments

• Model integrity: Consistent distributed training with complete worker sets

Demo Economics: $0.16 for complete coordinated training vs. potential hours of idle GPU costs waiting for worker coordination.

Conclusion

For distributed GPU workloads requiring multiple nodes, gang scheduling isn't optional—it's mandatory for functional training.

Single machine can't handle: Modern ML workloads requiring distributed computation
Volcano delivers: Coordinated multi-node scheduling that ensures training actually works

Repository: nvidia-gpu-volcano-k8s ⭐

If you're in a hurry to solve deployment time issues, simply add the WEBSITE_RUN_FROM_PACKAGE environment variable to your Azure Web App settings and set its value to 1. This change can significantly reduce your deployment times.

Why slow deployment in azure webapp ?

When deploying applications to Azure Web Apps without the WEBSITE_RUN_FROM_PACKAGE setting, the process typically involves:

1. Uploading Files: Your deployment pipeline uploads the application files to the Azure Web App.

2. Extracting Files: If the application is uploaded as a compressed package (like a ZIP file), the web app service extracts the contents to the file system.

3. Restarting the Web App: The web app might need to restart to pick up the new files.

4. File Synchronisation: The Azure Web App synchronises the new files with the existing file system.

Problems with Traditional Deployment

• Long Deployment Times: Extracting and synchronising files can be slow, especially for large applications or many files.

• File Corruption: There is a risk of file corruption during extraction or synchronisation.

• Downtime: The web app might be unavailable during the file extraction and restart process, causing service disruption.

What WEBSITE_RUN_FROM_PACKAGE Does ?

The WEBSITE_RUN_FROM_PACKAGE setting simplifies the deployment process by running the application directly from a package file, such as a ZIP file. This eliminates the need to extract the files, reducing deployment times and potential issues.

Possible Values for WEBSITE_RUN_FROM_PACKAGE

• 1: This value tells Azure to run the web app directly from a package file that has been deployed. The app reads the package and executes it without extracting the contents.

• URL: You can also set WEBSITE_RUN_FROM_PACKAGE to a URL pointing to a package file stored in Azure Blob Storage. This can be useful for scenarios where you manage your application packages in Blob Storage and want the web app to fetch the package directly from there.

How to Implement WEBSITE_RUN_FROM_PACKAGE ?

Here’s a step-by-step guide to implementing WEBSITE_RUN_FROM_PACKAGE in your deployment process:

1. Prepare Your Deployment Package

   • Compress your application into a ZIP file. Ensure that the package contains all the necessary files for your application to run.

2. Upload the Package

   • Use your CI/CD pipeline to upload the ZIP file to Azure. This can be done using Azure DevOps, GitHub Actions, or any other CI/CD tool you prefer.

3. Set the WEBSITE_RUN_FROM_PACKAGE Setting

   •   az webapp config appsettings set --resource-group <group-name> --name <app-name> --settings WEBSITE_RUN_FROM_PACKAGE=1
     

     Replace <group-name> and <app-name> with your resource group name and web app name.

4. Either do above step or simply add env in webapp manually from azure console

Example GitHub Actions Workflow

Here’s an example of how you can set up a GitHub Actions workflow to deploy your Node.js application using WEBSITE_RUN_FROM_PACKAGE:

# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
# More GitHub Actions for Azure: https://github.com/Azure/actions

name: Build and deploy Node.js app to Azure Web App - my-webapp

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Node.js version
        uses: actions/setup-node@v3
        with:
          node-version: "20.x"

      - name: npm install, build, and test
        run: |
          npm install
          npm run build --if-present


      - name: Zip artifact for deployment
        run: zip release.zip ./* -r

      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v3
        with:
          name: node-app
          path: release.zip

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: "Production"
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
    permissions:
      id-token: write #This is required for requesting the JWT

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v3
        with:
          name: node-app

      - name: Unzip artifact for deployment
        run: unzip release.zip

      - name: Login to Azure
        uses: azure/login@v1
        with:
          client-id: ${{ secrets.AZUREAPPSERVICE_CLIENTID }}
          tenant-id: ${{ secrets.AZUREAPPSERVICE_TENANTID }}
          subscription-id: ${{ secrets.AZUREAPPSERVICE_SUBSCRIPTIONID }}

      - name: "Deploy to Azure Web App"
        id: deploy-to-webapp
        uses: azure/webapps-deploy@v2
        with:
          app-name: "my-webapp"
          slot-name: "Production"
          package: .

Follow me on

LinkedIn : https://www.linkedin.com/in/akash202k/

Twitter : https://x.com/akash202k_

Thank You

Happy Learning : )

The Indian Premier League (IPL) presents a colossal challenge for streaming platforms like JioCinema due to the surge in viewership during key moments. This technical analysis delves into JioCinema's infrastructure and strategies for handling the unprecedented traffic spikes during IPL matches, focusing on the utilization of cutting-edge technologies to ensure seamless streaming experiences.

• Delve into the sheer magnitude of IPL viewership and its impact on streaming platforms.

• Introduction to JioCinema's pivotal role as a primary streaming provider for IPL matches.

• In-depth exploration of JioCinema's technology stack, comprising:

  • Content Delivery Network (CDN) architecture for efficient content distribution.

  • Load Balancers leveraging algorithms like round-robin or least connections for optimal traffic distribution.

  • Auto-scaling mechanisms facilitated by container orchestration platforms like Kubernetes for dynamic resource allocation.

  • Advanced content caching strategies utilizing technologies such as Redis or Varnish to alleviate server load during peak periods.

• Detailed elucidation of JioCinema's proactive approach to managing traffic surges during IPL matches.

• Utilization of predictive analytics powered by machine learning algorithms to forecast viewer demand and preemptively scale resources.

• Implementation of intelligent caching policies, including edge caching and micro-caching, to serve popular content with minimal latency.

• Seamless integration with cloud providers such as AWS or Google Cloud for elastic scaling of infrastructure resources based on demand.

• Granular examination of the traffic influx during a significant IPL moment, such as Virat Kohli's century in a particular match.

• Real-time analysis of JioCinema's infrastructure response to the sudden surge in viewership.

• Evaluation of key performance metrics, including latency, throughput, and server response times, during the peak period.

• Analogous analysis spotlighting the traffic surge during Dhoni's pivotal innings in the IPL final.

• Comparative assessment of infrastructure performance between regular traffic and high-demand scenarios.

• Thorough discussion on JioCinema's measures to uphold service reliability and ensure high availability during critical IPL moments.

• Deployment of redundant systems and fault-tolerant architectures, such as active-active clusters and multi-region setups, to mitigate potential failures.

• Continuous monitoring and observability through tools like Prometheus and Grafana for proactive issue detection and remediation.

• Recapitulation of the robust technological foundation and proactive strategies employed by JioCinema to navigate the challenges posed by IPL traffic spikes.

• Reflection on the significance of scalability, reliability, and performance optimization in the realm of streaming platforms.

• Insights into how JioCinema's tech-centric approach serves as a benchmark for addressing similar scalability challenges in the streaming industry.

• Citations and resources documenting the technologies, methodologies, and case studies referenced throughout the analysis.

Pre-requisite

• Create s3 bucket.

• Create sqs queue.

• create ec2 instance (ubuntu)

• create IAM User for accessing s3 bucket and add s3 permission to it.

Creating user for sftp

Install s3fs

sudo apt update
sudo apt install s3fs

Setup creds for s3fs

Create file

sudo vim /etc/passwd-s3fs

# write below content with its actual value for iam user created
ACCESS_KEY_ID:SECRET_KEY_ID

create user for sftp

#!/bin/bash

# Usage: sudo ./create-sftp-user.sh username

USERNAME=$1
S3_BUCKET_NAME="s3-bucket-name"
S3_BUCKET_PATH="$S3_BUCKET_NAME:/incoming/$USERNAME"
MOUNT_DIR="/mnt/sftp-s3/$USERNAME"

sudo useradd -m -d "$MOUNT_DIR" -s /bin/bash -G sftponly "$USERNAME"
# Set password for the user
sudo passwd "$USERNAME"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
sudo mkdir -p "$MOUNT_DIR/data"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR/data" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
echo "User $USERNAME added and configured."

sudo vim /etc/ssh/sshd_config

at last write

Match User userName
    ChrootDirectory /mnt/sftp-s3/$USERNAME
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
    PasswordAuthentication yes

now try to access over sftp .

Update sqs queue access policy :

{
  "Version": "2012-10-17",
  "Id": "Policy1679925546977",
  "Statement": [
    {
      "Sid": "Stmt1679925532180",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:*",
      "Resource": "arn:aws:sqs:region:1111111111:sqs_name",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::s3-buket-name"
        }
      }
    }
  ]
}

Create s3 event notification.

Thanks

For more such content follow me on :

Twitter : https://x.com/akash202k_

import json
import boto3
import string
import random

dynamodb = boto3.resource('dynamodb')
table_name = 'url-shortener-table'
table = dynamodb.Table(table_name)

def lambda_handler(event, context):
    print(event)

    http_method = event['requestContext']['http']['method']

    if http_method == 'POST':
        body = json.loads(event['body'])
        long_url = body['long_url']
        short_url = generate_short_url()
        table.put_item(Item={'short_id': short_url, 'long_url': long_url})

        response = {
            'statusCode': 200,
            'body': json.dumps({'short_url': short_url})
        }
    elif http_method == 'GET':
        short_url = event['rawPath'][1:]
        response = table.get_item(Key={'short_id': short_url})
        if 'Item' in response:
            long_url = response['Item']['long_url']
            response = {
                'statusCode': 301,
                'headers': {
                    'Location': long_url
                }
            }
        else:
            response = {
                'statusCode': 404,
                'body': json.dumps({'error': 'Short URL not found'})
            }
    return response

def generate_short_url():
    characters = string.ascii_letters + string.digits
    short_url = ''.join(random.choice(characters) for _ in range(3))
    return short_url

NOTE : Replace table_name with your dynamoDb table name once we create it later in this steps . and partitionKey should be short_id

Step 3: Create an API Gateway:

Lets create api first and later we can do configuration

Open the API Gateway console.

• Click on "Create API" and select "HTTP API". Review and create api

  

• Once we created API Define routes for POST and GET requests.

  

  Select create and keep route empty and method POST then select create

  ref following image

  

  once created select POST like this

  

  click on attach integration and go ahead with option

  create and attach integration

• Next select integration type as lambda function and then select lambda function created earlier for this task

• Make sure this option is enabled and go ahead with create option

  

  Cool we just created our first route , lets go and create second route which is GET

  for redirecting shorturl we got in previous response

• Lets create second route , now go back to Routes and select create

  

  

  Make sure its /{short_id}

• Now select GET Method attach integration and click on attach just right side option

• 

  From this option you can get your APIGW url to make request

• 

  These are the curls to test Replace url with actual url

Create Short Url : POST

curl --location 'https://4tr7c6bru76e.execute-api.us-east-1.amazonaws.com' \
--header 'Content-Type: application/json' \
--data '{"long_url": "https://cloudcdk.com"}'

Access Short Url : GET

curl --location 'https://4tr7c6bru76e.execute-api.us-east-1.amazonaws.com/{Replace_with_id_u_get_in_response_for_above_post_request}'

Follow me on Twitter for more : Twitter

1. LoadBalancer Service Type:

• Purpose: Creates an external Elastic Load Balancer (ELB) that distributes traffic to the service's pods.

• Advantages:

  • Provides a stable public IP or DNS for the service.

  • Handles load balancing automatically among pods.

• Limitations:

  • Additional cost for the ELB.

  • Limited flexibility for advanced routing configurations.

1. NodePort Service Type:

   

• Purpose: Allocates a static port on each cluster node, forwarding external traffic to the service's pods.

• Advantages:

  • No additional cost for load balancing.

  • Simple setup without external load balancers.

• Limitations:

  • Not suitable for large-scale production deployments due to potential port conflicts.

  • The exposed port range might be limited based on the cluster configuration.

1. Ingress Resource with Application Load Balancer (ALB) { Recommended }:

   

• Purpose: Uses an AWS Application Load Balancer (ALB) as the Ingress controller, enabling advanced HTTP routing and SSL termination.

• Advantages:

  • Advanced HTTP-based routing, host-based routing, and SSL termination.

  • Better integration with AWS services, like AWS Certificate Manager (ACM).

• Limitations:

  • Ingress controllers require additional resources, which might impact cluster performance.

  • Setting up ALB Ingress might involve some initial configuration.

1. ClusterIP Service Type:

   

• Purpose: Exposes the service internally within the cluster for communication between services.

• Advantages:

  • Ideal for internal communication between services within the cluster.

  • Provides a stable internal IP address for service discovery.

• Limitations:

  • Not accessible from outside the cluster, limiting external access.

1. ExternalName Service Type:

   

• Purpose: Maps a service to an external DNS name, enabling access to resources outside the cluster.

• Advantages:

  • Allows services in the cluster to access external resources easily.

  • Simplifies the transition from external services to internal Kubernetes services.

• Limitations:

  • Only supports mapping to external DNS names, not IP addresses.

  • Limited to read-only access to the external resource.

1. External Load Balancer (Manually Provisioned):

• Purpose: Manually provisions an external load balancer outside of EKS and points it to the service's pods.

• Advantages:

  • Full control over load balancer configuration and capabilities.

  • Flexibility to choose a load balancer from any provider.

• Limitations:

  • Requires manual setup and maintenance, which can be time-consuming.

  • May involve additional costs and complexities depending on the external load balancer.

Conclusion:

Exposing services in an Amazon EKS cluster can be achieved through multiple straightforward methods, each with its advantages and limitations. Consider the specific requirements of your application, such as scalability, security, and ease of management, to choose the method that best aligns with your needs.