Q16 — AWS SAA-C03 Ch.17

Question 16 of 89 | ← Chapter 17

Q1316. A company receives data from several sources and stores the data in an Amazon S3 data lake. The company transforms the data and ingests it into multiple downstream data stores, including databases and data warehouses.A solutions architect wants to design an extract, transform, and load(ETL) pipeline to orchestrate a process to prepare and load the data for analytics activities. The processed data must also be available for stakeholders through dashboards and visualizations.Which solution will meet these requirements with the LEAST development effort?

Correct Answer: B. Use AWS Glue to create ETL pipelines. Use AWS Step Functions to orchestrate the workflow. Use Amazon QuickSight to generate dashboards and visualizations.

Explanation

The most efficient solution with the least development effort for designing an ETL pipeline that processes data from Amazon S3, loads it into downstream stores, and enables dashboards and visualizations is:B. Use AWS Glue to create ETL pipelines. Use AWS Step Functions to orchestrate the workflow. Use Amazon QuickSight to generate dashboards and visualizations.Why Option B is Correct:AWS Glue for ETL Pipelines (Least Development Effort)AWS Glue is a fully managed serverless ETL service that automatically generates Python/Scala scripts for data transformation.It discovers schema metadata (via the AWS Glue Data Catalog) and handles scaling, fault tolerance, and monitoring with minimal coding.Supports batch and streaming ETL jobs, making it ideal for data lake integrations.AWS Step Functions for Workflow OrchestrationAWS Step Functions provides serverless workflow orchestration, allowing you to chain together multiple AWS services (e.g., Glue, Lambda, S3, DynamoDB) into a state machine. It eliminates the need for custom scripting (unlike Lambda-based orchestration in Option A/C) and provides visual debugging and retry mechanisms.Amazon QuickSight for Dashboards & VisualizationsAmazon QuickSight is a fully managed BI service that integrates seamlessly with S3, Redshift, RDS, and Glue Data Catalog.It requires no infrastructure setup and supports interactive dashboards, ML-powered insights, and embedded analytics.Why Other Options Are Incorrect:A. Amazon EMR + Custom Scala Script + Lambda + OpenSearch Service High development effort: Requires manual scripting (Scala for EMR, Lambda for orchestration). OpenSearch is overkill: Better suited for log analytics/search use cases, not business dashboards. Operational overhead: Managing EMR clusters increases cost and complexity.C. AWS Lambda for ETL + Step Functions + QuickSightLambda limitations: Not ideal for large-scale ETL (15-minute max runtime, memory constraints). Glue is better for ETL: Lambda-based ETL requires more manual coding than Glue's auto-generated scripts.D. Custom EC2 Scripts + AWS Glue Orchestration + Glue DataBrew for Dashboards High operational overhead: Managing EC2 instances for ETL is inefficient compared to serverless Glue. Glue DataBrew is not for dashboards: It's a data preparation tool, not a visualization service (QuickSight is needed instead).Implementation Steps for Option B:Set Up AWS Glue ETL JobsCreate a Glue Crawler to discover and catalog metadata from S3 (e.g., CSV/JSON files). Develop a Glue ETL Job (using the GUI or auto-generated script) to transform data (e.g., cleaning, aggregations).Configure the job to output processed data to S3 or a downstream database (e.g., Redshift, Aurora).Orchestrate with AWS Step FunctionsDefine a state machine in Step Functions that:Triggers the Glue ETL job.Waits for completion (with error handling/retries).Optionally invokes another service (e.g., Lambda for post-processing).Use AWS SDK integrations to interact with Glue, S3, etc.Build Dashboards with Amazon QuickSightConnect QuickSight to S3 (for processed data) or Redshift/RDS (for structured data). Create interactive visualizations (e.g., line charts, pivot tables) and publish dashboards. Share dashboards with stakeholders via email/SMS alerts or embedded URLs.Key Considerations:Cost Optimization:Use Glue Spot Instances for cost savings on ETL jobs.Set Step Functions standard workflows (cheaper than express for long-running jobs).Security:Encrypt data at rest (S3 SSE-S3/KMS) and in transit (SSL/TLS). Use IAM roles with least-privilege permissions for Glue, Step Functions, and QuickSight.Scalability:Glue automatically scales compute resources based on job demand. Step Functions can handle thousands of concurrent executions.Conclusion:Option B is the best choice because it leverages fully managed AWS services (Glue, Step Functions, QuickSight) to minimize development effort while providing scalable ETL, reliable orchestration, and powerful visualizations. Other options introduce unnecessary complexity or use mismatched tools (e.g., OpenSearch for dashboards, EC2 for ETL).