AWS SAA-C03 Study

← D3 · High-Performing 🖥️ Compute 💾 Storage 🌐 Networking & Delivery 📨 Messaging & Serverless 🏗️ Infrastructure 🗄️ Databases 📊 Analytics & Streaming

📊Analytics & Streaming↑ Top

D3 · High-Performing

Redshift

Amazon Redshift

"Data warehouse — OLAP, columnar, petabyte SQL analytics"

🎯 Sebab Apa Wujud

Redshift wujud sebab database transaksi (RDS) lemah bila kau nak aggregate berbilion baris untuk laporan/BI — query SUM/GROUP BY atas data besar jadi sangat lambat sebab data simpan ikut row. Redshift simpan data COLUMNAR + compress + agih kerja ke banyak node (MPP), jadi analytics atas data warehouse berskala petabyte jadi laju — tanpa membebankan database OLTP production.

Apa Dia

Petabyte-scale data warehouse untuk OLAP (analytics), BUKAN OLTP (transaksi → guna RDS). Simpan data secara COLUMNAR + compress → query aggregate (SUM, GROUP BY) atas berbilion baris jadi laju. Massively Parallel Processing (MPP): leader node agih query ke banyak compute node. Spectrum boleh query S3 terus tanpa load.

Contoh Guna

BI/reporting atas data berkumpul (sales, finance), join besar merentas berjuta baris, dashboard berulang yang sama tiap hari

Redshift Anatomy (MPP)

Leader Node → terima query, buat plan, agih ke compute node, kumpul hasil

Compute Nodes → simpan data + jalankan query selari (MPP)

RA3 + Managed Storage → compute & storage berasingan, bayar ikut guna; scale tanpa pindah data

Redshift Spectrum → query data dalam S3 TERUS tanpa load ke cluster

Concurrency Scaling → auto-tambah cluster sementara bila ramai user query serentak

AQUA (Advanced Query Accelerator) → lapisan cache + compute DEKAT storage; tolak (push-down) filter/aggregate dekat data → kurang data lalu network + kurang beban CPU compute node. Auto-managed (RA3, no extra charge)

Redshift MPP + Spectrum flow

SQL Client / BI

JDBC/ODBC

Leader Node

plan + agih

Compute 1

Compute 2

Compute 3

Managed Storage

columnar

Spectrum → S3

query data lake

Leader node pecahkan query, compute nodes proses selari (MPP) atas data columnar. Spectrum extend query terus ke S3 tanpa load. Inilah sebab Redshift laju untuk aggregate besar — tapi LEMAH untuk single-row insert/update (itu kerja OLTP → RDS).

Anatomy komponen Redshift (cluster → slices → storage)

Rendering diagram…

Hierarki: satu Cluster = 1 Leader Node (otak: plan & agih query) + banyak Compute Node (pekerja, ada CPU/RAM/disk sendiri). Tiap compute node dipecah jadi Slices — slice inilah unit MPP yang proses data selari. RA3 simpan data dalam Managed Storage (RMS) columnar atas S3, jadi compute & storage boleh scale berasingan. Spectrum extend query terus ke S3.

Pilih servis query/analytics (decision tree)

Rendering diagram…

OLTP (transaksi) → RDS/Aurora. Analytics berulang/kompleks → Redshift. Ad-hoc SQL atas S3 tanpa setup → Athena. Custom Spark/Hadoop → EMR. Ni trap paling kerap di exam: jangan pilih Redshift untuk "ad-hoc, jarang query" (itu Athena) atau untuk OLTP (itu RDS).

Redshift dah ada, tapi LAMBAT — pilih feature mana? (decision tree)

Rendering diagram…

INGAT exam: baca PUNCA yang soalan sebut. "network bandwidth + CPU limits" → AQUA (BUKAN Spectrum — Spectrum extend ke S3, tak selesai bottleneck dalam cluster, malah boleh tambah trafik). "ramai user serentak" → Concurrency Scaling. "query data lake S3" → Spectrum. AQUA kini auto-managed = pilihan minimum operational overhead.

Redshift — komponen dalaman (anatomy)

Komponen	Peranan	Nota exam
Cluster	Keseluruhan: 1 Leader Node + 1 atau lebih Compute Node	Unit yang kau provision (atau guna Serverless)
Leader Node	Terima query SQL, buat execution plan, agih kerja ke compute node, kumpul & pulang hasil	TAK dikira caj dalam cluster multi-node; single-node = leader & compute jadi satu
Compute Node	Simpan data + jalankan query; ada CPU, memory & disk sendiri	Tambah node = lebih storage + lebih laju (scale horizontal)
Node Slices	Tiap compute node dipecah jadi beberapa slice; tiap slice proses sebahagian data SELARI	Inilah unit MPP sebenar — lebih slice = lebih parallelism
Managed Storage (RMS)	RA3: data disimpan dalam Redshift Managed Storage atas S3, auto-scale	Asingkan compute & storage → scale satu tanpa tambah satu lagi
Columnar + Compression	Simpan data ikut COLUMN (bukan row) + mampat	Sebab aggregate (SUM/GROUP BY) laju; lemah untuk single-row update
Zone Maps	Simpan min/max tiap blok data dalam memory → langkau blok tak relevan	Kurangkan jumlah data di-scan = query lagi laju
Distribution Style (DISTKEY)	Cara baris diagih antara slices: KEY / EVEN / ALL	KEY co-locate baris untuk join; ALL replicate table kecil ke semua node
Sort Key (SORTKEY)	Cara data disusun fizikal atas disk	Range scan & merge join laju bila filter ikut sort key
Redshift Spectrum	Query data dalam S3 TERUS guna external table, tanpa load	Warehouse + data lake S3 dalam satu query
Concurrency Scaling	Auto-tambah cluster sementara bila ramai query serentak	Spike concurrent users → tak beratur

Ingat: Aliran: Cluster → Leader Node (otak: plan & agih) → Compute Nodes (pekerja) → Node Slices (unit selari MPP) → Managed Storage (columnar atas S3). DISTKEY & SORTKEY tuning prestasi; Spectrum extend ke S3; Concurrency Scaling handle ramai user serentak.

Redshift vs Athena vs EMR vs RDS

Aspect	Redshift	Athena	EMR	RDS/Aurora
Jenis	Data warehouse (OLAP)	Serverless SQL on S3	Big-data cluster	Relational (OLTP)
Data	Structured, loaded/Spectrum	Data dalam S3	Apa saja dalam S3/HDFS	Structured transaksi
Server	Cluster (atau Serverless)	🟢 Serverless, no infra	Kau urus cluster	Managed instance
Kos model	Per node/jam atau RPU	Per TB di-scan	Per instance (Spot jimat)	Per instance/jam
Best bila	Recurring complex analytics, BI	Ad-hoc, jarang, no setup	Custom Spark/Hadoop/ML	App transaksi (CRUD)

Ingat: Recurring + complex + structured + perlu laju konsisten → Redshift. Sekali-sekala / jarang / tak nak urus apa-apa → Athena. Custom big-data processing → EMR. Transaksi → RDS. Jangan campur OLAP (Redshift) dengan OLTP (RDS).

Redshift — features penting exam

Feature	Apa dia	Trigger
RA3 + Managed Storage	Compute & storage berasingan, scale sendiri	"Scale compute tanpa bayar storage lebih"
Redshift Spectrum	Query S3 TERUS tanpa load ke cluster	"Query data warehouse + data lake S3 sekali"
Redshift Serverless	Auto-scale RPU, no cluster management	"Analytics tapi tak nak urus cluster / beban tak tetap"
Concurrency Scaling	Auto-tambah cluster bila ramai query serentak	"Banyak user query serentak, jangan slow / queue"
Zero-ETL (Aurora→Redshift)	Data Aurora auto-flow ke Redshift, tak payah pipeline ETL	"Analytics near-real-time tanpa bina ETL"
AQUA (Advanced Query Accelerator)	Cache+compute dekat storage; push-down filter/aggregate → kurang data lalu network + beban CPU. Auto-managed, RA3, no extra charge	🟢 "performance issues due to NETWORK BANDWIDTH + CPU limits" → AQUA

Ingat: Spectrum = warehouse + S3 dalam satu query. Serverless = tak nak urus cluster. Concurrency Scaling = ramai user serentak. Zero-ETL = buang kerja bina pipeline Aurora→Redshift. AQUA = kurangkan network movement + CPU load (auto, no overhead).

AQUA vs Redshift Spectrum — DUA-DUA "Redshift feature", JANGAN keliru

Aspect	AQUA	Redshift Spectrum
Masalah yang diselesai	Cluster slow sebab NETWORK + CPU bottleneck	Nak query data S3 external tanpa load ke cluster
Cara kerja	Push compute (filter/aggregate) DEKAT storage → kurang data bergerak	Compute layer asing query data dalam S3 (external table)
Kesan pada network	KURANGKAN data lalu network (dalam cluster)	Boleh TAMBAH trafik (tarik data dari S3)
Setup / overhead	🟢 Auto-managed — Redshift decide sendiri, no toggle	Kena define external schema + external table
Kos	No extra charge (RA3)	Bayar per data DI-SCAN dalam S3

Ingat: Keyword pembeza: "network bandwidth + CPU processing limits" → AQUA. "query data dalam S3 / data lake / external table tanpa load" → Spectrum. Dua-dua bunyi "Redshift + performance", tapi AQUA selesai bottleneck DALAM cluster; Spectrum extend query KELUAR ke S3. AQUA config status kini RETIRED — Redshift auto-tentukan (zero ops overhead).

⚡ Quick Sifir — hafal ni

▪Redshift = data warehouse OLAP (columnar + compress + MPP). RDS/Aurora = OLTP.
▪Cluster = 1 Leader Node (plan/agih) + banyak Compute Node (proses) → pecah jadi Slices (unit MPP).
▪RA3 + Managed Storage = compute & storage berasingan, scale independent.
▪Spectrum = query S3 TERUS tanpa load. Athena = serverless ad-hoc (no cluster).
▪Concurrency Scaling = auto-tambah cluster bila ramai user serentak.
▪Zero-ETL (Aurora→Redshift) = analytics near-real-time tanpa bina pipeline ETL.
▪AQUA = push compute dekat storage → kurang NETWORK + CPU bottleneck. Auto-managed, RA3, no extra charge. BUKAN Spectrum (Spectrum = query S3 external).

💡 Exam Scenario

"Dashboard BI berulang atas berbilion baris sales, perlu join kompleks laju & konsisten" → Redshift. "Banyak analyst query serentak waktu puncak, jangan beratur" → Concurrency Scaling. "Nak query data dalam Redshift DAN data lake S3 dalam satu SQL" → Redshift Spectrum. "Beban analytics tak menentu, tak nak urus cluster" → Redshift Serverless. "Redshift slow sebab NETWORK BANDWIDTH + CPU processing limits, minimize overhead/cost" → AQUA (BUKAN Spectrum, BUKAN ElastiCache). Bukan Athena (ad-hoc/jarang), bukan RDS (OLTP).

🪤 Perangkap Soalan

Q: App transaksi e-commerce (banyak insert/update order satu-satu, low latency). Database paling sesuai?

⚠ Umpan: Redshift — sebab dia handle data besar & SQL kompleks, nampak powerful.

✓ Betul: RDS/Aurora — keyword 'transaksi / single-row insert/update / OLTP' = RDS/Aurora. Redshift columnar, teruk untuk write satu-satu baris (itu OLAP, bukan OLTP).

Q: Analyst jarang-jarang nak run SQL ad-hoc atas log dalam S3, tak nak urus apa-apa cluster. Service?

⚠ Umpan: Redshift — sebab dia 'analytics + SQL', nampak macam jawapan analytics default.

✓ Betul: Athena — keyword 'ad-hoc / jarang / no setup / no cluster / data dalam S3' = Athena (serverless). Redshift perlu provision cluster, overkill untuk query jarang-jarang.

Q: Redshift cluster makin slow bila data membesar — punca disebut 'network bandwidth and CPU processing limits'. Nak improve query performance, minimize operational overhead & cost. Pilih?

⚠ Umpan: Redshift Spectrum — sebab data ada dalam warehouse + S3, Spectrum nampak macam jawapan 'Redshift + performance' default. SALAH: Spectrum cuma extend query ke S3 external; ia TAK selesai bottleneck CPU/network DALAM cluster, malah boleh TAMBAH trafik network dari S3.

✓ Betul: AQUA (Advanced Query Accelerator) — push compute dekat storage → kurang data lalu network + kurang beban CPU compute node. Tepat dua-dua keyword 'network bandwidth + CPU'. Auto-managed (Redshift decide sendiri) = zero operational overhead, no extra charge pada RA3.

Q: Nak laju query analytics Redshift, ada yang cadang ElastiCache Memcached untuk cache. Betul tak?

⚠ Umpan: ElastiCache Memcached — sebab 'caching = laju' nampak masuk akal.

✓ Betul: BUKAN — ElastiCache untuk cache OLTP/key-value (depan RDS/DynamoDB), bukan beban analytics columnar Redshift. Untuk percepat Redshift + kurang CPU/network = AQUA. Keyword 'Redshift query performance' ≠ ElastiCache.

🧠 Cara Mudah Ingat

→Redshift = OLAP/analytics (columnar, aggregate berjuta baris). RDS/Aurora = OLTP (transaksi, single-row read/write). Jangan keliru — soalan "transactional app" JANGAN jawab Redshift.
→Columnar storage + compression = sebab Redshift laju untuk SUM/AVG/GROUP BY atas data besar, tapi teruk untuk insert/update satu-satu baris.
→Redshift Spectrum vs Athena: dua-dua query S3. Spectrum = kau dah ada Redshift cluster, nak extend query ke S3. Athena = standalone, no cluster, ad-hoc.
→RA3 nodes = compute & managed storage berasingan (scale independent). DC2 = compute+storage bercantum (legacy, dataset kecil).
→Concurrency Scaling: auto add transient cluster untuk handle spike concurrent queries — per-second billing, ada free credits harian.
→Multi-AZ: Redshift RA3 boleh Multi-AZ untuk HA. Cross-region snapshot untuk DR.
→Zero-ETL integration: Aurora/RDS → Redshift tanpa bina ETL pipeline sendiri (near-real-time analytics atas data transaksi).
→AQUA (Advanced Query Accelerator): lapisan cache + hardware-accelerated compute antara compute node RA3 dengan Managed Storage. Push-down filter/aggregate dekat data → kurangkan data yang lalu network + kurangkan beban CPU compute node. Tepat untuk soalan "performance issues due to network bandwidth and CPU processing limits".
→AQUA vs Spectrum (trap!): AQUA selesaikan bottleneck CPU/network DALAM cluster; Spectrum extend query KELUAR ke S3 (boleh tambah trafik network). Soalan sebut "network bandwidth + CPU limits" → AQUA, bukan Spectrum.
→AQUA kini AUTO-MANAGED: parameter aqua-configuration-status sudah RETIRED — Redshift sendiri tentukan bila guna AQUA. Available pada RA3, no extra charge. Sebab itu AQUA = pilihan "minimize operational overhead".
→AQUA ≠ ElastiCache: ElastiCache cache OLTP/key-value (depan RDS/DynamoDB), bukan untuk beban analytics columnar Redshift.
→PRICING/COST: Manual snapshots TAK auto-delete (kekal selamanya sampai kau padam sendiri) → caj storage berterusan, jadi cluster jangka panjang kos naik kalau tak padam yang lama. Automated snapshots auto-delete lepas retention (default 1, max 35 hari). Jimat kos cluster lama: padam manual snapshots tak perlu + set automated retention rendah. BUKAN Spot (Redshift cluster guna Reserved Instance), BUKAN instance store (data cluster = managed storage, instance store ephemeral = data hilang).

Guna Bila

Data warehouse: complex/recurring analytics atas structured data berskala besar

data warehouseOLAPcolumnarMPPRA3managed storageRMSRedshift SpectrumRedshift Serverlessconcurrency scalingzero-ETLclusterleader nodecompute nodenode slicesdistribution keyDISTKEYsort keySORTKEYzone mapspetabyteBI analyticsAQUAAdvanced Query Acceleratornetwork bandwidthCPU processing limitspush down computequery acceleratorAQUA vs Spectrummanual snapshots costpricing

📎 Redshift Spectrum 📎 Concurrency Scaling 📎 Query performance tuning

D3 · High-Performing

QuickSight

Amazon QuickSight

"QuickSight = panel meter kereta: sensor (data) → meter cluster (SPICE) → pemandu nampak sekali pandang. Serverless BI, drag-drop, dia visualize."

🎯 Sebab Apa Wujud

QuickSight wujud sebab nak tunjuk data sebagai dashboard/carta untuk boss & business user, kalau bina sendiri kau kena setup BI server, urus scaling, dan dashboard jadi lambat bila ramai user pukul database. QuickSight serverless BI + ada SPICE (in-memory cache) supaya dashboard laju tanpa pukul sumber tiap kali, plus ML Insights bagi forecast/anomaly tanpa perlu pasukan data science.

Apa Dia

Serverless BI service. Connect terus ke S3, RDS, Redshift, Athena, atau sumber lain. Bina dashboard & visualisasi tanpa server. SPICE = enjin in-memory yang cache data import → query dashboard laju & berulang tanpa pukul sumber tiap kali. ML Insights bagi anomaly detection, forecasting (seasonality/trend), dan auto-narratives — tiada infrastruktur ML berasingan. Edition Enterprise tambah row-level security, akses AD/SSO, dan embedded analytics.

QuickSight — komponen utama

Data sources → S3, Athena, Redshift, RDS/Aurora, sumber luar (databases, SaaS); QuickSight baca TERUS, tak payah ETL dulu

SPICE → Super-fast Parallel In-memory Calculation Engine; cache data import dalam memori untuk dashboard laju & repeatable. Alternatif: Direct Query (pukul sumber live)

Analysis → kanvas tempat kau bina visual (carta, jadual, peta)

Dashboard → analysis yang dah di-publish, read-only, untuk dikongsi

ML Insights → anomaly detection, forecasting, auto-narrative (built-in, no ML setup)

Edition → Standard (asas) vs Enterprise (row-level security, AD/SSO, encryption-at-rest, embedded, fine-grained access)

Analogi Panel Meter Kereta — apa QuickSight buat

Rendering diagram…

QuickSight = panel meter kereta. Sensor enjin/tangki (raw data dalam S3/Redshift/Athena) hantar bacaan. SPICE = meter cluster yang baca sekali & simpan dalam memori supaya jarum bergerak laju tanpa pukul enjin tiap saat (in-memory cache, dashboard berulang laju). Panel pemandu (Dashboard) tunjuk semua sekali pandang untuk "boss"/executive. Lampu amaran (ML Insights) = anomaly detection + forecasting ("usage akan naik bulan depan"). INGAT exam: "dashboard + forecasting, minimal ops" → QuickSight, BUKAN Redshift + custom ML.

QuickSight vs alat analytics lain — siapa buat apa

Servis	Peranan dalam pipeline	Trigger exam
QuickSight	VISUALIZE — dashboard, carta, BI	"executive dashboard", "visualization", "forecast trend"
Athena	QUERY — SQL ad-hoc atas S3	"ad-hoc SQL", "query S3 no setup"
Glue	TRANSFORM — ETL + Data Catalog	"ETL serverless", "catalog data lake"
Redshift	STORE/ANALYZE — data warehouse OLAP	"recurring complex analytics", "data warehouse"

Ingat: Pipeline serverless klasik: S3 (store) → Glue (catalog/ETL) → Athena (query) → QuickSight (visualize). QuickSight = lapisan PALING HUJUNG (visual). Soalan sebut "dashboard / chart / visualize / report untuk boss" → QuickSight. Sebut "SQL" → Athena. Sebut "transform" → Glue.

SPICE vs Direct Query — dua cara QuickSight baca data

Aspect	SPICE (import)	Direct Query
Di mana data	Cache dalam memori QuickSight	Kekal di sumber, pukul live
Kelajuan dashboard	🟢 Laju, konsisten	Ikut kelajuan sumber
Data freshness	Sefresh refresh terakhir (jadual)	🟢 Real-time, sentiasa terkini
Beban ke sumber	🟢 Rendah (baca sekali)	Tinggi (tiap query pukul sumber)
Guna bila	Dashboard kerap dibuka, data tak perlu real-time	Data mesti terkini detik ini

Ingat: SPICE = laju + jimat sumber tapi data sefresh refresh terakhir. Direct Query = sentiasa terkini tapi bergantung & bebankan sumber. Soalan "dashboard laju untuk ramai user" → SPICE. "Mesti real-time" → Direct Query.

QuickSight Standard vs Enterprise edition

Ciri	Standard	Enterprise
Row-level / column-level security	🔴 Tiada	🟢 Ada
Akses AD / SSO (IAM Identity Center)	🔴 Tiada	🟢 Ada
Encryption at rest (SPICE)	🔴 Tiada	🟢 Ada
Embedded analytics dalam app	Terhad	🟢 Penuh
ML Insights	🔴 Tiada	🟢 Ada

Ingat: Soalan sebut "row-level security", "restrict data per user", "Active Directory", "embed dashboard", atau "ML Insights/forecast" → mesti Enterprise edition. Standard = BI asas sahaja.

⚡ Quick Sifir — hafal ni

▪QuickSight = serverless BI — lapisan PALING HUJUNG (visualize). Athena=query, Glue=ETL, Redshift=store.
▪SPICE = in-memory cache → dashboard laju, jimat sumber (data sefresh refresh terakhir).
▪Direct Query = pukul sumber live → real-time tapi bebankan sumber.
▪ML Insights (forecast + anomaly + auto-narrative) = Enterprise edition.
▪Row/column-level security + AD/SSO + embedded + encryption-at-rest = Enterprise edition.
▪Connect terus ke S3/Athena/Redshift/RDS — tak payah load ke warehouse dulu.

💡 Exam Scenario

"Executive dashboards dari IoT data dalam S3 dengan forecasting, no data warehouse" → QuickSight + S3 direct + ML Insights. "Dashboard laju untuk ramai user, data tak perlu real-time" → SPICE. "Setiap analyst nampak hanya baris data region dia" → QuickSight Enterprise + row-level security. QuickSight BUKAN untuk ad-hoc SQL (→ Athena), BUKAN untuk ETL (→ Glue). Keywords: dashboard, forecast, trend, BI, visualization.

🪤 Perangkap Soalan

Q: Executive nak dashboard dari data IoT dalam S3 dengan usage forecasting, operasi minimal, tiada data warehouse. Penyelesaian?

⚠ Umpan: Load ke Redshift + bina custom ML model untuk forecast — nampak 'proper' untuk analytics + ML.

✓ Betul: QuickSight (connect S3 direct) + ML Insights forecasting. Keyword 'dashboard + forecast + minimal ops' = QuickSight, bukan Redshift + custom ML/SageMaker.

Q: Setiap analyst regional hanya boleh nampak baris data region mereka dalam dashboard yang sama. Apa diperlukan?

⚠ Umpan: QuickSight Standard edition + buat dashboard berasingan tiap region — leceh & tak scale.

✓ Betul: QuickSight Enterprise + row-level security. Keyword 'restrict data per user / row-level' = Enterprise edition (Standard tiada RLS).

🧠 Cara Mudah Ingat

→QuickSight connect TERUS ke S3/Athena/Redshift/RDS — tak payah load ke Redshift dulu untuk dashboard. Lapisan paling hujung dalam pipeline analytics.
→SPICE = in-memory cache → dashboard laju & berulang. Direct Query = pukul sumber live (real-time tapi bebankan sumber). "Dashboard laju ramai user" → SPICE.
→ML Insights = built-in forecasting + anomaly detection + auto-narrative. "Usage trends + forecasting, minimal ops" → QuickSight ML Insights (Enterprise), bukan SageMaker.
→Row-level security, Active Directory/SSO, encryption-at-rest, embedded analytics, ML Insights = SEMUA Enterprise edition. Soalan sebut mana-mana ni → Enterprise.
→Exam trap: "dashboards + forecasting, minimal ops" → QuickSight (bukan Redshift + custom ML). "Visualize" / "report untuk eksekutif" = QuickSight.

Guna Bila

Business intelligence dashboards, data visualization, ML-powered analytics

QuickSightBIdashboardforecastingML InsightsvisualizationS3 directSPICEDirect Queryrow-level securityEnterprise editionembedded analyticsanomaly detectionauto-narrativeIoT analyticsActive Directory

📎 What is Amazon QuickSight 📎 Importing data into SPICE

D3 · High-Performing

Athena

Amazon Athena

"SQL terus pada S3 — serverless, bayar per TB scan"

🎯 Sebab Apa Wujud

Athena wujud sebab kadang kau cuma nak run SQL sekali-sekala atas data dalam S3 (cth analyze CloudTrail/ALB logs) — tak berbaloi setup & bayar Redshift cluster 24/7 untuk query jarang-jarang. Athena serverless: point ke S3, tulis SQL, bayar HANYA per TB data di-scan, tiada cluster untuk urus atau idle cost.

Apa Dia

Serverless interactive query service. Point ke S3, tulis SQL, dapat results. Bayar per TB data yang di-scan. Sokong CSV, JSON, Parquet, ORC. Pair dengan Glue Data Catalog sebagai metadata store.

Serverless analytics pattern (100% serverless, no cluster)

raw data lake

Glue Crawler

+ Data Catalog (schema)

Athena

SQL, pay per scan

QuickSight

dashboard / BI

S3 simpan data → Glue Crawler discover schema isi Data Catalog → Athena query guna SQL → QuickSight visualize. Tiada server/cluster untuk urus. Ini "serverless analytics" classic — kalau soalan kata "no infrastructure / serverless / query S3 directly", ini jawapannya (bukan Redshift/EMR).

Athena — cara kurangkan kos (kos = data di-scan)

Teknik	Kenapa jimat	Kesan
Columnar format (Parquet/ORC)	Athena scan kolum yang perlu sahaja, bukan seluruh baris	🟢 Scan ↓ banyak + laju
Partition data (by date/region)	Athena langkau partition yang tak kena filter WHERE	🟢 Scan ↓ ikut partition pruning
Compress (gzip/Snappy)	Saiz fail kecil = byte di-scan kurang	🟢 Kos ↓
SELECT kolum tertentu (bukan SELECT *)	Kurangkan kolum dibaca	🟢 Scan ↓

Ingat: Kos Athena = jumlah data DI-SCAN, bukan bilangan query. Parquet/ORC + partition + compress = combo standard untuk turunkan kos & masa. Soalan "kurangkan kos Athena" → convert ke columnar + partition.

Athena vs Redshift Spectrum vs Redshift

Aspect	Athena	Redshift Spectrum	Redshift
Perlu cluster?	🟢 Tidak (serverless)	🔴 Ya (Redshift cluster)	🔴 Ya
Data di mana	S3	S3 (extend dari cluster)	Loaded dalam cluster
Guna bila	Ad-hoc, jarang, no setup	Dah ada Redshift, nak query S3 sekali	Recurring complex analytics
Kos model	Per TB di-scan	Per TB di-scan (+ cluster)	Per node/jam atau RPU

Ingat: Ad-hoc tanpa cluster → Athena. Dah ada Redshift, nak gabung query warehouse + S3 → Spectrum. Recurring/complex dengan prestasi konsisten → Redshift loaded. Dua-dua Athena & Spectrum query S3, beza pada "ada cluster ke tak".

⚡ Quick Sifir — hafal ni

▪Athena = serverless SQL atas S3, bayar per TB DI-SCAN (bukan per query, bukan per jam).
▪Kurangkan kos: Parquet/ORC (columnar) + partition (date/region) + compress + SELECT kolum.
▪Pair dengan Glue Data Catalog sebagai metadata/schema store.
▪Pipeline serverless: S3 → Glue (catalog/ETL) → Athena (query) → QuickSight (visualize).
▪Athena vs Spectrum: dua-dua query S3; Athena no cluster, Spectrum perlu Redshift cluster.

💡 Exam Scenario

"Analyse CloudTrail logs atau ALB access logs dalam S3 guna SQL" → Athena. "Ad-hoc analysis tanpa setup database" → Athena. "Kurangkan kos query S3" → convert ke Parquet + partition. Bukan Redshift (yang untuk structured, recurring analytics dengan dedicated cluster).

🪤 Perangkap Soalan

Q: Team nak analyze CloudTrail/ALB logs dalam S3 guna SQL, hanya sekali-sekala, tak nak urus infrastruktur. Service?

⚠ Umpan: Redshift — sebab 'SQL analytics' nampak macam kerja data warehouse.

✓ Betul: Athena — keyword 'ad-hoc / sekali-sekala / serverless / query S3 directly / no setup' = Athena. Redshift perlu cluster 24/7, overkill untuk query jarang.

Q: Kos query Athena terlalu tinggi. Cara paling berkesan kurangkan kos?

⚠ Umpan: Tambah lebih banyak query nodes / upgrade — silap konsep, Athena serverless tiada node.

✓ Betul: Convert data ke Parquet/ORC (columnar) + partition + compress. Keyword: kos Athena = jumlah data DI-SCAN, jadi kurangkan byte yang di-scan.

🧠 Cara Mudah Ingat

→Kurangkan kos Athena: data di-scan = duit. Guna columnar format (Parquet/ORC) + PARTITION data (by date/region) + compress → scan kurang, murah + laju
→Serverless analytics pipeline: S3 (store) → Glue (catalog/ETL) → Athena (query) → QuickSight (visualize). Hafal urutan ni
→Athena = ad-hoc/interactive SQL pada S3. Redshift = data warehouse dedicated untuk recurring/complex analytics. Redshift Spectrum = Redshift query terus S3
→PRICING: Athena = $5.00 per TB data DI-SCAN (round up 10MB minimum per query). Convert ke Parquet + partition + compress boleh jimat 30-90% sebab byte di-scan turun. DDL (CREATE/ALTER) FREE. Athena Provisioned Capacity (per-DPU) ada untuk workload besar predictable. Cost discriminator: kos ikut byte scan, BUKAN bilangan query — jadi columnar+partition = lever utama.

Guna Bila

Ad-hoc SQL analysis of data in S3 without loading to a database

serverless SQLS3 queriespay per scanParquetORCGlue Cataloglog analysisad-hocpartitioncolumnarQuickSightserverless analytics patternpricing

📎 What is Amazon Athena?📎 Top 10 performance tuning tips for Athena

D3 · High-Performing

Glue

AWS Glue

"Crawler endus schema → Catalog simpan → ETL Job transform. Glue = gam yang sambung data lake."

🎯 Sebab Apa Wujud

Glue wujud sebab data lake berselerak (S3, RDS, Redshift) — tanpa katalog pusat, Athena/EMR/Redshift tak tahu schema apa wujud, dan transform data (CSV→Parquet) secara manual perlu jaga cluster Spark sendiri. Glue selesaikan dua-dua: Crawler auto-endus schema isi Data Catalog (satu sumber kebenaran), dan ETL Job serverless transform data tanpa cluster — jadi 'gam' yang sambung data lake.

Apa Dia

Glue = serverless ETL + metadata layer untuk data lake. Glue Data Catalog = kedai metadata pusat (table/column) untuk semua data assets (S3, RDS, Redshift) — Athena, EMR, Redshift Spectrum semua baca katalog yang sama. Glue Crawler = auto-endus (discover) schema dari sumber dan ISI Data Catalog. Glue ETL Job = serverless Spark job untuk transform data (cth CSV → Parquet). Bayar per DPU-hour, tiada cluster untuk diurus.

Glue — komponen utama

Data Catalog → kedai metadata pusat (database, table, column, partition). Satu sumber kebenaran untuk Athena, EMR, Redshift Spectrum, Lake Formation

Crawler → connect ke sumber (S3/JDBC/DynamoDB), infer schema, ISI Data Catalog. Boleh jadual berkala

Classifier → kenal pasti format data (CSV/JSON/Parquet/custom grok); Crawler panggil Classifier untuk tentukan schema

ETL Job → serverless Spark (atau Python shell) yang transform data; auto-generate kod, bayar per DPU-hour

Trigger / Workflow → orchestration: jalankan job ikut jadual, event, atau on-demand (rantai crawler → job)

DataBrew → visual data prep, no-code (250+ transformation) untuk pembersihan data

Glue Studio → antara muka visual drag-drop untuk bina ETL pipeline

Anatomy komponen Glue (crawler → catalog → job)

Rendering diagram…

Aliran: Crawler connect ke sumber → panggil Classifier untuk kenal format → ISI Data Catalog dengan schema. Lepas tu ETL Job (Spark) baca katalog, transform data, tulis ke target. Athena/EMR/Redshift Spectrum semua kongsi Data Catalog yang sama. INGAT exam: komponen yang ISI Data Catalog = Crawler (bukan Job, bukan Table).

Analogi Jus Mangga — apa itu ETL (Extract · Transform · Load)

Rendering diagram…

ETL = Extract → Transform → Load, macam buat jus mangga. EXTRACT = petik mangga mentah dari pokok (tarik raw data dari sumber). TRANSFORM = kupas, buang biji, blend (bersihkan + tukar format, cth CSV→Parquet) — ini bahagian paling berat & guna Spark. LOAD = tuang jus siap dalam botol (tulis data bersih ke target supaya boleh terus "minum"/query). Glue buat ketiga-tiga langkah ni secara serverless — kau tak payah beli & jaga "blender" (cluster) sendiri.

Glue — komponen mana buat apa

Komponen	Peranan	Trigger exam
Data Catalog	Kedai metadata pusat (table/column/partition)	"central metadata store", "schema registry untuk data lake"
Crawler	Endus schema dari sumber & ISI Data Catalog	"determine schema & populate Data Catalog" → Crawler
Classifier	Kenal pasti format data; dipanggil oleh Crawler	"recognize custom/log format" → custom Classifier
ETL Job	Serverless Spark transform (CSV→Parquet)	"transform/convert data tanpa urus server" → Glue Job
Trigger / Workflow	Orchestrate job ikut jadual/event	"automate & schedule ETL pipeline"
DataBrew	Visual no-code data cleaning	"clean/prep data tanpa tulis kod"

Ingat: Crawler ISI katalog · Classifier kenal format · Job transform · Catalog simpan metadata. Soalan "apa populate Data Catalog?" → Crawler. "Convert CSV ke Parquet serverless?" → Glue ETL Job.

Glue terminology — Data Store vs Data Source vs Data Target vs Catalog vs Table (jangan keliru)

Istilah	Maksud	Trigger exam
Data Store	Repository tempat data SEBENAR duduk (S3 bucket, RDS, DB). Ini yang Crawler CRAWL	"repository for persistently storing the data", "data store yang di-crawl"
Data Source	Data Store dalam peranan INPUT kepada job/transform	"data store used as input to a transform" → Data Source
Data Target	Data Store dalam peranan OUTPUT — tempat job TULIS hasil	"a process/transform writes to" → Data Target
Data Catalog	Persistent metadata store (1 per account/region) — simpan SEMUA metadata dari data store yang di-crawl	"persistent metadata store", "central metadata store"
Table	Metadata definition yang wakili data: nama column, jenis data, partition. Schema SAHAJA	"metadata definition representing the data" → Table

Ingat: Data Store = tempat data betul-betul duduk (yang di-CRAWL). Data Source/Target = Data Store yang SAMA, cuma beza peranan (input vs output). Data Catalog = kedai metadata pusat. Table = definisi schema dalam Catalog. PENTING: Table & Database simpan metadata SAHAJA — data sebenar kekal dalam data store asal. Ini trap terminology Glue yang exam suka uji.

Glue vs EMR vs Athena — bila guna yang mana (ETL/query)

Aspect	Glue	EMR	Athena
Fungsi utama	Serverless ETL + Catalog	Cluster Hadoop/Spark	Serverless SQL on S3
Urus server?	🟢 Tidak (serverless)	🔴 Ya (kau urus cluster)	🟢 Tidak (serverless)
Guna untuk	Transform & catalog data	Custom big-data, ML, full control	Ad-hoc query data S3
Kos	Per DPU-hour	Per instance (Spot jimat)	Per TB di-scan
Trigger soalan	"ETL serverless + Data Catalog"	"Hadoop/Spark cluster, full control"	"SQL ad-hoc, no setup"

Ingat: ETL ringkas + katalog tanpa cluster → Glue. Custom Spark/Hadoop dengan kawalan penuh → EMR. Query SQL ad-hoc atas S3 → Athena. Ketiga-tiga boleh kongsi Glue Data Catalog.

⚡ Quick Sifir — hafal ni

▪Glue = serverless ETL + Data Catalog (metadata pusat untuk Athena/EMR/Redshift Spectrum).
▪Crawler = auto-endus schema dari sumber → ISI Data Catalog (boleh jadual berkala).
▪Data Catalog = metadata SAHAJA (apa data wujud); data sebenar kekal di S3/RDS.
▪Glue = serverless (no cluster). EMR = managed cluster, full control Spark/Hadoop.
▪Lake Formation = kawalan akses (siapa) atas Glue Data Catalog (apa).
▪Bayar per DPU-hour; ETL Job = serverless Spark (auto-generate code).

💡 Exam Scenario

"Transform raw CSV dalam S3 ke Parquet format untuk Athena" → Glue ETL job. "Auto-catalog all data sources for data lake" → Glue Crawler + Data Catalog. "Determine schema dari DynamoDB & populate Data Catalog" → Crawler. "Clean & prep data tanpa tulis kod" → DataBrew. Keywords: ETL, data lake, data catalog, transform, Spark, serverless.

🪤 Perangkap Soalan

Q: Perlu auto-determine schema dari data dalam S3/DynamoDB dan populate Glue Data Catalog. Komponen Glue mana?

⚠ Umpan: Glue Classifier — sebab dia kenal pasti format data, nampak macam dia yang tentukan schema.

✓ Betul: Glue Crawler — keyword 'discover schema + populate Data Catalog' = Crawler (orchestrator). Classifier cuma BANTU Crawler kenal format custom; Crawler yang panggil Classifier & isi catalog.

Q: Pasukan nak transform raw CSV dalam S3 ke Parquet dengan kawalan PENUH atas Spark cluster & framework. Service?

⚠ Umpan: AWS Glue — sebab dia memang ETL Spark, nampak terus padan.

✓ Betul: EMR — keyword 'full control over Spark/cluster / pilih framework' = EMR. Glue serverless, Spark config diabstrak (kawalan terhad). Kalau soalan kata 'serverless ETL, no cluster' baru Glue.

🧠 Cara Mudah Ingat

→Glue Crawler specifically = the component that connects to a data source, infers schema, and POPULATES the Glue Data Catalog with table metadata
→Exam pattern: "determine schema from DynamoDB/S3 and populate Glue Data Catalog" → Crawler (not a Table, not a Classifier)
→Glue Classifier helps the Crawler recognize custom data formats — but Crawler is the orchestrator that calls Classifiers
→Glue Data Catalog = metadata sahaja (apa data wujud). Lake Formation = kawalan akses (siapa boleh akses) atas katalog yang sama
→Serverless analytics pattern: S3 (store) → Glue (crawl + catalog + ETL) → Athena (query) → QuickSight (visualize). Hafal urutan ni
→Glue = serverless (no cluster). Kalau soalan tekankan "full control" / "Hadoop/Spark cluster" → itu EMR, bukan Glue
→Terminology trap: Data Store = repository simpan data (yang di-CRAWL). Data Source = Data Store sebagai INPUT job. Data Target = Data Store yang job TULIS hasil. Sama benda, beza peranan
→Glue Table = metadata definition (column, jenis data, partition) dalam Data Catalog. PENTING: Table & Database simpan metadata SAHAJA — data sebenar kekal dalam data store asal (S3/RDS). Table ≠ data

Guna Bila

ETL jobs, data catalog for data lake, prepare and transform data for analytics

ETLdata catalogSparkserverlesscrawlerclassifierETL jobDataBrewGlue Studioworkflowdata laketransformParquetschema discoveryDynamoDBschema inferenceDPUdata storedata sourcedata targetGlue tablemetadata definitiondatabase

📎 AWS Glue components 📎 Glue Crawlers

D3 · High-Performing

Lake Formation

AWS Lake Formation

"Pengawal keselamatan kolam (data lake) — row, column, cell level access + bina secure lake laju"

🎯 Sebab Apa Wujud

Lake Formation wujud sebab dua pain: (1) Glue Data Catalog cuma simpan metadata (table/column wujud) — dia TAK boleh kawal siapa boleh baca baris/kolum mana; dan (2) bina data lake selamat guna S3+IAM+Glue+KMS secara manual itu azab & ambil berminggu (setup IAM satu-satu, pening encryption, cuci data kotor). Lake Formation selesai dua-dua: fine-grained access (analyst Region A nampak baris Region A je, kolum gaji disorok) DARI SATU TEMPAT, plus blueprint/automation yang percepat bina secure lake.

Apa Dia

Senang cerita: Data Lake = KOLAM simpan data (S3). Lake Formation = PENGAWAL KESELAMATAN + KONTRAKTOR untuk kolam tu. Ia duduk atas S3 + Glue Data Catalog dan (1) enforce fine-grained permissions hingga row, column, cell level, dan (2) automate kerja susah bina data lake (IAM, encryption, cleansing, catalog) supaya secure lake siap dalam HARI, bukan minggu. Glue Data Catalog cuma simpan table/column metadata — Lake Formation yang enforce ACTUAL access control.

Analogi Kolam + Pengawal Keselamatan — Data Lake vs Lake Formation

Rendering diagram…

Data Lake = kolam takungan data (S3). Lake Formation = pengawal keselamatan pintar depan pintu kolam: tapis siapa boleh sauk air, bahagian mana boleh ambil (row/column/cell), + tolong cuci & catalog kolam automatik. INGAT exam: "fine-grained / row/column/cell-level access pada data lake" ATAU "simplify/accelerate secure data lake" → Lake Formation.

Aliran sebenar — Data Lake → cuci → Data Warehouse → BI

Rendering diagram…

Aliran lazim syarikat besar: data mentah → Data Lake (S3, simpan semua) → Glue/Batch cuci & susun → Data Warehouse (Redshift) untuk BI. INGAT exam: "store all/any data types raw, any scale" → Data Lake/S3; "complex SQL + BI on structured data" → Redshift; "process massive raw logs/big data" → EMR/Batch; "row/column/cell access on the lake" → Lake Formation.

3 Zon dalam Data Lake — Raw → Cleanse → Curated (elak jadi Data Swamp)

Rendering diagram…

Data lake matang biasa pecah 3 zon: Raw/Landing (mentah, simpan original tak diubah) → Cleanse/Processed (Glue ETL buang duplicate/anomali, tukar Parquet) → Curated/Analytics (kemas, sedia BI/ML). Kalau data dihumban masuk tanpa Glue Data Catalog + Lake Formation governance, lake reput jadi DATA SWAMP — ada data tapi tiada siapa boleh cari/percaya/guna. INGAT exam: "raw zone/landing zone" = data as-is; "curated/analytics zone" = data sedia guna; "data lake jadi tak terurus/tak boleh cari" = Data Swamp → fix dengan Catalog + Lake Formation governance.

Data Lake vs Data Warehouse — exam paling suka kelirukan

Aspect	Data Lake (S3)	Data Warehouse (Redshift)
Jenis data	SEMUA: struct + unstruct + semi (mentah)	Structured sahaja (dah bersih & modeled)
Analogi	Gudang lambakan 📦	Perpustakaan tersusun 📚
Tujuan	Simpan semua / staging awal	OLAP analytics, BI, laporan berulang
Schema	Schema-on-read (tafsir masa baca)	Schema-on-write (kena kemas dulu)
Service	Amazon S3 (+ Athena / Spectrum query)	Amazon Redshift
Keyword exam	"store all/any data types, any scale, raw"	"complex SQL + BI on structured data"

Ingat: Lake = mentah, semua jenis, simpan dulu (S3). Warehouse = structured kemas untuk BI (Redshift). "any/all data raw" → Data Lake/S3; "complex SQL + BI on structured" → Redshift. Lake Formation BUKAN storage — ia pengawal akses ATAS data lake.

Glue Data Catalog vs Lake Formation — metadata vs akses

Aspect	Glue Data Catalog	Lake Formation
Simpan apa	Metadata: apa data wujud (table/column/schema)	Access control: siapa boleh akses apa
Kawal akses?	❌ Tidak (metadata sahaja)	✅ Row / column / cell-level
Keyword	"central metadata / schema catalog"	"fine-grained / row/column/cell security"

Ingat: Catalog = WHAT data exists. Lake Formation = WHO can access (sampai row/column/cell). "fine-grained access to data lake" → Lake Formation, bukan Glue Catalog, bukan IAM S3 (object-level je).

⚡ Quick Sifir — hafal ni

▪Data Lake = KOLAM (S3, simpan data). Lake Formation = PENGAWAL kolam (kawal akses + bina lake).
▪Glue Data Catalog = metadata (apa data wujud). Lake Formation = ACCESS CONTROL (siapa boleh akses).
▪Lake Formation = row-level + column-level + cell-level security (IAM/Glue sahaja tak boleh).
▪Keyword 'fine-grained / row/column/cell-level access' untuk data lake → Lake Formation.
▪Keyword 'simplify / accelerate creation of a SECURE data lake' → Lake Formation.
▪Lake Formation sendiri PERCUMA — bayar service bawah (S3, Glue, Athena) je.
▪3 zon data lake: Raw/Landing (mentah, as-is) → Cleanse/Processed (dah bersih, Parquet) → Curated/Analytics (sedia BI/ML).
▪Humban data tanpa catalog + governance = DATA SWAMP (ada data tapi tak boleh cari/guna) — Glue Catalog + Lake Formation yang elak swamp.

🪤 Perangkap Soalan

Q: Data lake dalam S3 perlu analyst hanya boleh baca kolum tertentu & baris tertentu (sembunyi data sensitif macam gaji/No IC). Penyelesaian?

⚠ Umpan: Glue Data Catalog + IAM policy atas bucket S3 — nampak boleh kawal akses.

✓ Betul: Lake Formation — keyword 'row/column/cell-level fine-grained access untuk data lake' = Lake Formation. Glue Catalog metadata sahaja; IAM S3 cuma object-level (whole-file), bukan row/column dalam table.

Q: Syarikat nak bina data lake SELAMAT dengan cepat — auto setup permission, encryption, data cleansing, catalog. Service mana?

⚠ Umpan: Setup S3 + IAM + Glue + KMS manual satu-satu — 'memang boleh buat sendiri'. Nampak betul sebab 'semua komponen ada'. SALAH: manual = berminggu, banyak silap config, bukan 'simplify/accelerate'.

✓ Betul: AWS Lake Formation — keyword 'simplify / accelerate creation of a SECURE data lake' = Lake Formation (blueprint, permission, encryption, cleansing, semua sekali tempat).

Q: Analyst nak run complex SQL + BI dashboard BERULANG atas data jualan yang dah tersusun (structured). Simpan & analisis kat mana?

⚠ Umpan: Data Lake (S3) + Athena — 'boleh query S3 guna SQL'. Separa betul, tapi untuk BI berulang + complex join atas structured data, Athena ad-hoc kurang sesuai & data lake bukan optimized untuk ni.

✓ Betul: Data Warehouse = Amazon Redshift — keyword 'complex SQL + BI on structured data, berulang' = Redshift (OLAP, columnar, MPP). Data Lake (S3) = simpan SEMUA data mentah; Warehouse = structured dah kemas untuk BI.

🧠 Cara Mudah Ingat

→Glue Data Catalog = metadata store (what data exists). Lake Formation = access control (who can access what data)
→Lake Formation supports row-level, column-level, dan cell-level security — Glue / IAM S3 je tak boleh buat ni (IAM S3 = object-level whole-file)
→Use case: data lake dengan sensitive data, analysts boleh access hanya specific columns/rows
→Exam: "fine-grained access control" atau "row/column/cell-level security" untuk data lake → Lake Formation
→Exam: "simplify / accelerate the creation of a secure data lake" → Lake Formation (blueprint, automate IAM/encryption/cleansing)
→Lake Formation juga support data cleansing, data catalog, secure data sharing — one-stop data lake governance
→Data Lake (S3) vs Data Warehouse (Redshift): Lake = raw, semua jenis; Warehouse = structured untuk BI. Lake Formation governs the Lake
→PRICING: AWS Lake Formation sendiri PERCUMA (no additional charge) — kau bayar service bawah je: S3 storage ($0.023/GB-mo), Glue ETL/crawler ($0.44/DPU-hr), Athena query ($5/TB scanned). Jadi governance tak tambah kos extra

Guna Bila

Fine-grained access control on data lake (row/column/cell) + simplify & accelerate creation of a secure data lake

Lake Formationrow-level securitycolumn-levelcell-levelfine-grained accessdata lakeGlue Data Catalogsecure data lakeaccelerate data lakesimplify data lakedata lake vs data warehousedata warehousegovernancecentralized access controlblueprintdata cleansingpricingdata swampraw zonelanding zonecleanse zonecurated zoneanalytics zonethree-zone data lakedata lake zones

📎 What is AWS Lake Formation?📎 Lake Formation fine-grained access control

D3 · High-Performing

EMR

Amazon EMR

"EMR = Elastic + Master node + Ramai-pekerja: Elastic cluster, Master node (ketua agih kerja), Ramai pekerja (Core/Task). Hadoop/Spark, kau urus cluster (BUKAN serverless)."

🎯 Sebab Apa Wujud

EMR wujud sebab proses data berskala petabyte (log, ML, transform berat) guna satu EC2 mustahil — kau perlu cluster Spark/Hadoop, tapi setup & urus cluster Hadoop sendiri (provision, config, scale, patch) itu sangat susah. EMR managed-kan cluster big-data tu: kau pilih saiz/framework, AWS handle setup, dan boleh guna Spot + transient cluster (mati lepas job, data kekal di S3) untuk jimat besar.

Apa Dia

Managed cluster platform untuk big data frameworks (Spark, Hadoop, Hive, Presto, HBase). Kau choose cluster size, instance types, frameworks. Cluster ada 3 jenis node: Master (urus cluster), Core (run task + simpan HDFS), Task (run task sahaja, no HDFS). EMRFS benarkan EMR guna S3 sebagai storage layer → decouple compute dari storage. Lebih control dari Glue — untuk complex/custom big-data jobs.

EMR cluster — node roles

Master Node

urus + coordinate

Core Node

task + HDFS data

Core Node

task + HDFS data

Task Node (Spot)

compute only, no data

Task Node (Spot)

compute only, no data

S3 via EMRFS

durable storage layer

Master = otak cluster. Core nodes simpan HDFS data + run tasks → JANGAN letak atas Spot (hilang node = hilang data). Task nodes compute sahaja → selamat & jimat atas Spot. EMRFS guna S3 sebagai storage supaya cluster boleh transient (mati lepas job, data kekal di S3).

Analogi Kilang Kerupuk Lekor — EMR cluster nodes

Rendering diagram…

EMR = kilang besar proses ikan pukal (big data) sebab satu blender rumah (1 EC2) tak larat. Ketua pekerja (Master) agih kerja, tak potong sendiri. Pekerja Tetap (Core) potong + ada meja simpan ikan (HDFS) → jangan Spot, hilang meja = hilang ikan. Pekerja Sambilan (Task) tolong potong je, pinjam meja → selamat & jimat atas Spot, halau bila siap. Gudang S3 (EMRFS) simpan hasil walau kilang dah tutup (transient cluster).

HDFS dalaman — NameNode vs DataNode (Stor Ikan kilang)

Rendering diagram…

HDFS = "hard disk maya" merentas banyak komputer. Fail gergasi dipecah jadi Blocks (~128MB). NameNode (otak, atas Master) cuma pegang BUKU REKOD (metadata) — blok mana di node mana, dia TAK simpan data sebenar. DataNodes (atas Core nodes) yang betul-betul simpan blok. Tiap blok disalin ×3 ke node berbeza → satu node terbakar pun data selamat (fault tolerance). INGAT exam: Core node = pegang HDFS → JANGAN Spot. Dalam AWS, ramai tukar HDFS → S3 (via EMRFS): serverless, murah, tak payah jaga DataNode.

EMR node types — apa boleh Spot, apa tak

Node	Peranan	Simpan data?	Spot ok?
Master	Urus & coordinate cluster	Tidak	🔴 Tak (mati = cluster mati)
Core	Run tasks + simpan HDFS	✅ Ya (HDFS)	🟡 Berisiko (hilang data)
Task	Run tasks sahaja	Tidak	🟢 Ya — jimat selamat

Ingat: Letak Task nodes atas Spot untuk jimat tanpa risiko data. Core nodes pegang HDFS — guna On-Demand/RI. Master tak boleh Spot langsung.

EMR vs Glue — bila guna yang mana

Aspect	EMR	Glue
Model	Managed cluster (kau urus)	Serverless ETL (no cluster)
Control	🟢 Penuh — pilih framework, tune	Terhad (Spark abstracted)
Framework	Spark, Hadoop, Hive, Presto, HBase	Spark (managed)
Best bila	Custom/complex big-data, ML, kawalan penuh	ETL ringkas + Data Catalog
Kos	Per instance (Spot jimat)	Per DPU-hour, no idle cost

Ingat: Nak kawalan penuh framework / custom Spark / ML besar → EMR. Nak ETL serverless tanpa urus cluster → Glue. Soalan sebut "Hadoop/Spark cluster" atau "full control" → EMR.

Redshift vs EMR — gaya analisis (SQL vs Kod)

Ciri	Amazon Redshift	Amazon EMR
Kategori	Data Warehouse (OLAP)	Big-data framework (Hadoop/Spark)
Jenis data	Berstruktur — table, columnar	Bebas — log, teks, video, unstructured
Bahasa	SQL sahaja (SELECT, JOIN, SUM)	Kod (Python, Scala, Spark, Hive)
Infra	Managed cluster / Serverless	EC2 cluster (Master/Core/Task)
Analogi	📚 Perpustakaan tersusun — rujuk katalog, terus dapat	🏭 Kilang kitar semula — ramai pekerja proses sampah mentah
Guna bila	Laporan BI / dashboard atas data dah kemas	Bersih & proses data gergasi mentah, Data Science/ML

Ingat: Mereka TAK bergantung antara satu sama lain — dua enjin berbeza untuk gaya kerja berbeza. Data dah kemas + nak SQL → Redshift. Data mentah berterabur + perlu kod Spark/Hadoop → EMR. Boleh kolaborasi dalam satu pipeline (lihat tip), tapi EMR BUKAN buat analisis "di dalam" Redshift — itu salah faham biasa.

Glue vs EMR Serverless — dua-dua serverless, beza di mana?

Aspect	AWS Glue	EMR Serverless
Tujuan utama	ETL + Data Catalog (data integration)	Run Spark/Hive jobs serverless (analytics engine)
Framework	Spark (abstracted, Glue uruskan)	Pilih sendiri: Spark atau Hive, kawal versi
Kawalan Spark	Terhad — config diabstrak	🟢 Lebih — tune Spark config, sizing
Catalog & Crawler	🟢 Built-in (Crawler, DataBrew, Studio)	Tiada — guna Glue Data Catalog bila perlu
Best bila	ETL pipeline, catalog data lake, no-code prep	Migrate Spark/Hive job sedia ada, tak nak urus cluster
Kos	Per DPU-hour	Per vCPU + memori (per saat, auto-scale)

Ingat: Dua-dua serverless (tiada cluster). Bezanya: Glue = alat integrasi data lengkap (Crawler + Catalog + ETL) untuk bina pipeline. EMR Serverless = enjin Spark/Hive je dengan lebih kawalan — sesuai kalau kau dah ada Spark job dan cuma nak run tanpa jaga cluster. Soalan "migrate existing Spark job, no cluster management" → EMR Serverless. "ETL + auto data catalog" → Glue.

⚡ Quick Sifir — hafal ni

▪EMR = managed cluster big-data (Spark/Hadoop/Hive/Presto/HBase) — kau urus cluster (BUKAN serverless).
▪3 node: Master (urus), Core (task + HDFS data), Task (task sahaja).
▪Spot: Task = selamat (no data). Core = berisiko (pegang HDFS). Master = jangan Spot langsung.
▪EMRFS = guna S3 sebagai storage → transient cluster (mati lepas job, data kekal di S3).
▪EMR vs Glue: EMR = full control framework. Glue = serverless ETL, no cluster.
▪EMR vs Redshift: EMR = kod (Spark/Python) atas data mentah. Redshift = SQL atas data kemas (OLAP).

💡 Exam Scenario

"Process petabytes of log data using custom Spark jobs" → EMR. "Migrate on-prem Hadoop/Spark cluster ke AWS" → EMR. "Jimatkan kos batch processing yang fault-tolerant" → Task nodes atas Spot. Glue = serverless ETL (simpler, less control). EMR = full cluster (more control). Keywords: Hadoop, Spark, big data cluster, full control.

🪤 Perangkap Soalan

Q: Untuk jimat kos batch big-data yang fault-tolerant atas EMR, node mana paling sesuai letak atas Spot Instances?

⚠ Umpan: Core nodes atas Spot — sebab core paling ramai, jimat paling banyak kalau Spot.

✓ Betul: Task nodes atas Spot. Keyword: Core pegang HDFS data (hilang node = hilang data, berisiko); Task compute-only → selamat & jimat atas Spot. Master tak boleh Spot.

Q: Migrate on-prem Hadoop/Spark cluster ke AWS dengan kawalan penuh atas framework & versi. Service?

⚠ Umpan: AWS Glue — sebab dia ETL Spark serverless, nampak macam pengganti.

✓ Betul: EMR — keyword 'Hadoop/Spark cluster / full control / migrate existing cluster' = EMR. Glue serverless & Spark diabstrak (kawalan terhad), bukan untuk migrate cluster sedia ada dengan kawalan penuh.

🧠 Cara Mudah Ingat

→Tiga node: Master (urus), Core (task + HDFS data), Task (task sahaja). Core pegang data → bahaya atas Spot; Task tak pegang data → selamat & jimat atas Spot.
→EMRFS = guna S3 sebagai storage layer (bukan HDFS lokal). Ini benarkan TRANSIENT cluster: cluster mati lepas job habis, data kekal di S3 → jimat kos.
→Transient cluster = auto-terminate lepas step habis (batch jobs). Long-running cluster = kekal hidup untuk interactive/ad-hoc query.
→Exam trigger "managed Hadoop/Spark", "custom big-data framework", atau "full control over cluster" → EMR (bukan Glue, bukan Athena).
→EMR Serverless = run Spark/Hive tanpa urus cluster langsung. EMR on EKS = jalankan EMR atas cluster Kubernetes sedia ada.
→Instance fleets vs instance groups = dua cara provision capacity; fleets bagi flexibility pilih banyak instance type + Spot strategy.
→Pipeline sebenar (EMR + Redshift KOLABORASI, bukan bergantung): S3 (log mentah) → EMR/Spark bersih & transform (ETL) → load ke Redshift → analyst query SQL + QuickSight dashboard. EMR = pembersih data mentah; Redshift = warehouse untuk analisis akhir. Jangan fikir "EMR analyze dalam Redshift".
→HDFS dalaman: NameNode (Master) = pegang metadata/buku rekod (blok mana di node mana, TAK simpan data). DataNode (Core) = simpan blok sebenar. Block ~128MB, replication ×3 → fault tolerant. Sebab tu Core node bahaya atas Spot.

Guna Bila

Process petabyte-scale data with Spark, Hadoop, Hive, Presto — full control

HadoopSparkHivePrestoHBasebig dataclusterpetabytemanagedSpot instancesmaster nodecore nodetask nodeHDFSNameNodeDataNodeblockreplicationEMRFSETL pipelinetransient clusterEMR ServerlessEMR on EKS

📎 EMR cluster node types 📎 EMRFS

D3 · High-Performing

OpenSearch

Amazon OpenSearch Service

"Enjin carian dan log analytics — Elasticsearch dalam AWS"

🎯 Sebab Apa Wujud

Wujud sebab database biasa (RDS, DynamoDB) teruk untuk full-text search — nak cari 'kasut merah saiz 9' dengan typo + ranking + relevance, RDS LIKE '%...%' lembap dan tak boleh rank. OpenSearch index data jadi inverted index supaya search jadi laju + fuzzy + boleh aggregate log real-time, tanpa kau urus Elasticsearch cluster sendiri (AWS handle broker, patching, scaling).

Apa Dia

Managed OpenSearch (Elasticsearch fork) cluster. Ingestion via Amazon Data Firehose, OpenSearch Ingestion, atau Lambda. Visualise dengan OpenSearch Dashboards (Kibana). Guna untuk search yang perlu relevance ranking, fuzzy matching, atau aggregations real-time. Storage tiers (hot / UltraWarm / cold) untuk imbang kos vs kelajuan.

Log analytics ingestion pipeline

Sources

apps, CloudWatch Logs, IoT

Amazon Data Firehose

buffer + deliver, no code

OpenSearch

index + search + aggregate

OpenSearch Dashboards

Kibana visualise

Pola klasik log/observability: sumber → Firehose hantar ke OpenSearch (no code) → index untuk full-text search + aggregation → Dashboards visualize. Kalau soalan sebut "real-time log analytics + search + dashboard", ini jawapannya.

OpenSearch vs servis "cari/log" yang mengelirukan

Servis	Untuk apa	Bila pilih
OpenSearch	Full-text search + log analytics + dashboard	Search dengan ranking/fuzzy, atau log analytics real-time
CloudWatch Logs Insights	Ad-hoc query atas log CloudWatch	Cuma nak query log AWS sedia ada, no cluster
Amazon Kendra	Enterprise document search (ML, natural language)	Soalan natural language atas dokumen/PDF
Athena	SQL ad-hoc atas data S3	Query log dalam S3 guna SQL, jarang-jarang
DynamoDB	Key-value exact lookup	Lookup guna primary key — BUKAN full-text search

Ingat: Full-text/fuzzy search + log analytics + dashboard → OpenSearch. Natural-language document search → Kendra. Query log CloudWatch cepat → Logs Insights. Exact key lookup → DynamoDB (bukan OpenSearch).

⚡ Quick Sifir — hafal ni

▪OpenSearch = full-text/fuzzy SEARCH + log analytics, BUKAN exact key lookup (itu DynamoDB)
▪Storage tiers: Hot (laju+mahal) > UltraWarm (S3-backed, baca lambat sikit) > Cold (arkib termurah)
▪Firehose = jawapan default untuk 'stream logs ke OpenSearch tanpa code'
▪3 dedicated master nodes = quorum untuk stabilkan cluster besar
▪OpenSearch Dashboards = fork Kibana (soalan lama sebut 'Kibana')
▪Natural-language document search = Kendra, BUKAN OpenSearch

💡 Exam Scenario

"E-commerce product search dengan fuzzy matching dan relevance ranking" → OpenSearch. "Ingest dan search application logs real-time dengan visualisation" → OpenSearch + Firehose + Dashboards. "Natural-language search atas dokumen korporat" → Kendra (bukan OpenSearch). DynamoDB = exact key lookups. Keywords: search engine, log analytics, Elasticsearch.

🪤 Perangkap Soalan

Q: App perlu kekalkan 2 tahun log untuk audit tapi log lama jarang dibaca — nak kurangkan kos OpenSearch. Apa pilihan?

⚠ Umpan: Terus delete log lama / pindah ke Glacier — nampak murah, tapi soalan kata log MASIH perlu boleh disearch untuk audit, jadi delete/Glacier hilang kebolehan query.

✓ Betul: Pindah ke UltraWarm (atau Cold) tier — masih dalam OpenSearch, boleh search, tapi S3-backed jadi jauh lebih murah dari Hot. Keyword: 'kekal log lama + kos rendah + masih searchable'.

Q: Syarikat nak natural-language search atas dokumen korporat (PDF, wiki, FAQ) — pekerja tanya soalan biasa dan dapat jawapan tepat. Pilih apa?

⚠ Umpan: OpenSearch — nampak macam 'search engine' jadi orang pilih, tapi OpenSearch keyword/full-text, bukan faham intent/konteks soalan.

✓ Betul: Amazon Kendra — ML semantic search yang faham natural-language query + jawab terus dari FAQ. Keyword: 'natural language', 'understand intent', 'enterprise document'.

🧠 Cara Mudah Ingat

→Ingestion: Amazon Data Firehose (managed, no code), OpenSearch Ingestion (Data Prepper), atau Lambda. Firehose = jawapan paling kerap untuk "stream logs ke OpenSearch tanpa code".
→Storage tiers untuk jimat kos: Hot (laju, mahal) → UltraWarm (S3-backed, baca lambat sikit) → Cold (arkib, paling murah). Soalan "kekal log lama untuk kos rendah" → UltraWarm/Cold.
→Dedicated master nodes (3 untuk quorum) = stabilkan cluster besar; berbeza dari data nodes yang simpan/index data.
→Multi-AZ with standby untuk HA production. OpenSearch Serverless = auto-scale tanpa urus cluster.
→Exam confusion: OpenSearch = full-text/fuzzy SEARCH + log analytics. Kendra = ML natural-language document search. Jangan keliru dua ni.
→OpenSearch Dashboards = fork Kibana — soalan lama mungkin sebut "Kibana" untuk visualisation.

Guna Bila

Full-text search, real-time log/event analytics, dashboard visualisation

search enginelog analyticsElasticsearch compatibleKibanaOpenSearch Dashboardsreal-time analyticsfull-text searchfuzzyUltraWarmcold storagededicated masterFirehose ingestionrelevance ranking

📎 OpenSearch UltraWarm storage

D3 · High-Performing

MSK

Amazon MSK

"Kafka dalam AWS — managed, tak payah urus brokers"

🎯 Sebab Apa Wujud

Wujud sebab pasukan yang dah ada Apache Kafka (on-prem atau guna Kafka API) tak nak rewrite code mereka ke Kinesis, tapi pening urus broker, ZooKeeper, patching, dan HA sendiri. MSK ambil alih kerja operasi Kafka — kau guna Producer/Consumer API yang SAMA, AWS yang jaga infra.

Apa Dia

Fully managed Apache Kafka service. AWS manage brokers, ZooKeeper, patching. Kau guna Kafka Producer/Consumer API yang sama. Cross-AZ untuk HA.

MSK vs Kinesis Data Streams — streaming kembar (pilih yang mana?)

Aspect	Amazon MSK	Kinesis Data Streams
API	Apache Kafka API (open-source)	AWS proprietary (Kinesis API/KCL)
Bila pilih	Dah ada Kafka / nak migrate / ada Kafka expertise	Mula baru, nak simpler, AWS-native
Ops beban	Pilih broker size & count (atau Serverless)	Pilih shard count (atau On-Demand)
Unit skala	Partition (per topic)	Shard (1MB/s in, 2MB/s out setiap satu)
Retention	Boleh unlimited (tiered storage)	Default 24h, max 365 hari
Trigger Lambda	Perlu Event Source Mapping (ESM)	Perlu Event Source Mapping (ESM)

Ingat: Keyword "Apache Kafka / migrate Kafka / Kafka API / minimum code change" → MSK. Keyword "AWS-native streaming, simpler, tak nak urus Kafka" → Kinesis Data Streams. Dua-dua AWS-only (BUKAN multi-cloud). Lihat card Kinesis untuk Data Streams vs Firehose vs Video.

⚡ Quick Sifir — hafal ni

▪MSK = managed Apache Kafka; Kinesis = AWS-native proprietary (no Kafka API)
▪Pilih MSK bila ada Kafka expertise / nak migrate existing Kafka; pilih Kinesis kalau nak simpler
▪TIADA SSH / direct access ke broker — AWS manage infra
▪Lambda + MSK perlu Event Source Mapping (ESM), tak auto-trigger
▪MSK Serverless = auto-scale compute + storage, no broker capacity management
▪MSK = AWS-only, BUKAN multi-cloud

💡 Exam Scenario

"Migrate on-premises Apache Kafka cluster ke AWS" → Amazon MSK. Atau streaming pipeline yang perlu Kafka API compatibility. Kinesis = AWS-native proprietary. MSK = Kafka-compatible (for migration or Kafka expertise teams).

🪤 Perangkap Soalan

Q: Syarikat nak migrate on-prem Apache Kafka cluster ke AWS dengan minimum code change. Servis mana?

⚠ Umpan: Amazon Kinesis Data Streams — nampak macam 'streaming jadi pilih', tapi Kinesis bukan Kafka-compatible, jadi kena rewrite producer/consumer code = bukan minimum change.

✓ Betul: Amazon MSK — Kafka API compatible, code Kafka sedia ada jalan terus. Keyword: 'migrate Kafka', 'Kafka API', 'minimum code change'.

Q: Lambda function tak trigger walaupun ada message masuk topic MSK. Kenapa?

⚠ Umpan: Ingat Lambda auto-poll MSK macam SNS/SQS — sebab tu orang tak setup apa-apa, sangka ia automatik.

✓ Betul: Kena configure Event Source Mapping (ESM) — Lambda TAK auto pick up MSK events. Keyword: 'event source mapping', 'Lambda MSK trigger'.

🧠 Cara Mudah Ingat

→MSK is managed BUT does NOT provide SSH/direct access to Kafka brokers — AWS manages the underlying infrastructure.
→Lambda + MSK integration REQUIRES configuring an Event Source Mapping (ESM) — Lambda does not automatically pick up MSK events.
→MSK Auto Scaling: automatically expands broker storage based on utilization threshold. MSK Serverless: auto-scales compute AND storage, no broker capacity management.
→MSK is NOT multi-cloud — AWS-only service. Does NOT span other cloud providers.
→Kinesis vs MSK: Kinesis = AWS-native, simpler, no Kafka expertise needed. MSK = Kafka API compatible, for teams with Kafka expertise or migrating existing Kafka workloads.
→PRICING: (approx, us-east-1, no free tier)MSK Provisioned = bayar per broker-hour ikut instance type (kafka.m5.large ~$0.21/broker-hr) + storage $0.10/GB-mo + data transfer. Broker jalan 24/7 = bayar walau idle. MSK Serverless = $0.75/cluster-hr + ~$0.0015/partition-hr + throughput ($0.10/GB in). Exam: workload spiky/tak tentu → MSK Serverless (no broker capacity planning); steady high-throughput → Provisioned lagi murah.

Guna Bila

Real-time event streaming dengan Kafka API — migrate or build Kafka workloads

Kafkamanagedstreamingevent streamingmigrationKafka APIreal-time pipelinebrokersno SSHevent source mappingMSK Serverlessauto scaling storagepartitionpricingbroker-hour

📎 What is Amazon MSK 📎 MSK Serverless

D3 · High-Performing

Kendra

Amazon Kendra

"Google-like ML search for your enterprise documents"

🎯 Sebab Apa Wujud

Wujud sebab keyword search (OpenSearch) tak faham SOALAN — kalau pekerja tanya 'berapa hari cuti tahunan saya?', keyword search cuma match perkataan, bukan beri jawapan. Kendra guna ML untuk faham intent + konteks dan pulangkan jawapan tepat dari dokumen unstructured (PDF, wiki, email) merentas pelbagai repositori, hilangkan kerja bina enjin NLP sendiri.

Apa Dia

ML-powered enterprise search. Indexes PDFs, Word docs, HTML, emails, FAQs across S3, SharePoint, Confluence, databases. Understands natural language queries to return precise answers, not just keyword matches.

Kendra vs OpenSearch vs Comprehend — jangan keliru tiga "text" service ni

Aspect	Kendra	OpenSearch	Comprehend
Buat apa	ML "find the answer" search	Keyword/full-text search engine	NLP — analisa kandungan text
Input → Output	Soalan natural → jawapan tepat	Query → senarai dokumen match	Text → sentiment/entiti/topik
Guna bila	Cari jawapan dlm dokumen syarikat	Product/log search, high-volume	Sentiment review, extract entiti
Kos	MAHAL (per-hour edition, ~$810+/mo)	Sederhana (per node-hour)	Murah (per unit text dianalisa)
Keyword exam	"natural language", "ask a question", "FAQ"	"spell-check", "synonym", "high-volume"	"sentiment", "entities", "key phrases"

Ingat: Kendra = FAHAM soalan & bagi JAWAPAN (mahal); OpenSearch = MATCH keyword & senarai dokumen (volume tinggi); Comprehend = ANALISA text (sentiment/entiti), bukan search. INGAT exam: "natural language enterprise search" → Kendra; tapi kalau soalan tekan "high-volume / cost-sensitive product search" → OpenSearch sebab Kendra mahal.

⚡ Quick Sifir — hafal ni

▪Kendra = ML semantic 'find the answer' search; OpenSearch = scalable keyword/full-text
▪Faham natural-language query + intent, bukan sekadar match keyword
▪Index unstructured: PDF, Word, PowerPoint, HTML, email, wiki
▪Native handle FAQ → boleh bagi jawapan terus question-answer
▪Connector ke S3, SharePoint, Confluence, database
▪'enterprise document search + natural language' = Kendra

💡 Exam Scenario

"Enterprise wants to search across internal docs, FAQs, emails, PDFs with natural language queries" → Kendra. "E-commerce product search with spell-check/synonyms" → OpenSearch (keyword search engine). Kendra = understanding context + intent. OpenSearch = scalable keyword/full-text search.

🪤 Perangkap Soalan

Q: E-commerce nak product search berskala besar dengan spell-check, synonym, dan ranking untuk jutaan query/hari. Pilih apa?

⚠ Umpan: Kendra — nampak macam 'intelligent search jadi mesti Kendra', tapi Kendra mahal + dioptimumkan untuk 'find the answer' enterprise docs, bukan high-volume keyword product search.

✓ Betul: OpenSearch — scalable keyword/full-text search dengan fuzzy + relevance ranking untuk volume tinggi. Keyword: 'product search', 'high-volume', 'spell-check/synonym'.

🧠 Cara Mudah Ingat

→Kendra vs OpenSearch: Kendra = ML semantic search for enterprise "find the answer" use cases. OpenSearch = scalable keyword full-text search for high-volume queries (e-commerce, log analytics).
→Kendra natively handles FAQs — can provide direct question-answer responses from FAQ documents.
→Kendra indexes unstructured content: PDFs, Word, PowerPoint, HTML, emails, wikis.
→PRICING: (approx, us-east-1)Developer Edition ~$1.125/hr (~$810/mo, max 10K docs / 4K queries/day) ; Enterprise Edition ~$1.40/hr (~$1,008/mo, production HA) + connector sync hours. NO real free tier (Developer Edition free 750 hrs first 30 days je). Exam discriminator: Kendra MAHAL & bil per-jam walau idle — kalau soalan tekan kos atau high-volume keyword search, jawapan selalunya OpenSearch, BUKAN Kendra.

Guna Bila

Intelligent enterprise search across diverse document repositories

Kendraenterprise searchML searchsemantic searchnatural language queryFAQsunstructured documentsintelligent searchpricingexpensive

📎 What is Amazon Kendra

D3 · High-Performing

Data Exchange

AWS Data Exchange

"AWS marketplace untuk beli/subscribe third-party data"

🎯 Sebab Apa Wujud

Wujud sebab dapatkan data pihak ketiga (market data, cuaca, regulatory filing) selalunya leceh — kena urus kontrak lesen, cara delivery, dan update berkala secara manual. Data Exchange jadi marketplace: kau subscribe, data terus masuk S3 account kau, dan AWS handle licensing + subscription automatik.

Apa Dia

Marketplace for external data products. Providers publish datasets (market data, financial data, regulatory filings, weather, etc.). Subscribers browse, subscribe, data delivered directly to S3. Handles licensing and subscription management automatically.

⚡ Quick Sifir — hafal ni

▪Data Exchange = marketplace beli/subscribe data PIHAK KETIGA (external)
▪Data dihantar terus ke S3 bucket kau
▪AWS handle licensing + subscription management automatik
▪Kinesis = data REAL-TIME kau sendiri; Data Exchange = data luar orang lain jual
▪Contoh data: market/financial data, regulatory filing, cuaca, ekonomi

💡 Exam Scenario

"Company wants to subscribe to market data, economic indicators, and regulatory filings from third-party providers and deliver them to their AWS accounts for analytics" → AWS Data Exchange. Kinesis = your own real-time data. Data Exchange = external third-party data products.

🪤 Perangkap Soalan

Q: Firma analitik nak subscribe market data + economic indicator dari penyedia luar dan masukkan ke AWS untuk analisis. Servis mana?

⚠ Umpan: Amazon Kinesis / S3 + custom ingest — orang ingat kena bina pipeline ingest sendiri, tapi itu untuk data KAU sendiri, bukan beli data pihak ketiga berlesen.

✓ Betul: AWS Data Exchange — marketplace third-party data, delivery terus ke S3, licensing diuruskan. Keyword: 'third-party data', 'subscribe dataset', 'data products'.

🧠 Cara Mudah Ingat

→Data Exchange vs Kinesis/Glue/DataSync: itu semua untuk gerak/proses data KAU SENDIRI. Data Exchange = beli data PIHAK KETIGA (provider luar jual). Keyword "third-party / subscribe to a dataset / data product" → Data Exchange.
→Delivery: subscriber dapat data terus ke S3 (file-based), atau akses via API / Redshift datashare. AWS uruskan entitlement + licensing automatik.
→PRICING: subscriber bayar harga langganan yang DITETAPKAN OLEH PROVIDER (ada produk percuma, ada bayar bulanan/tahunan) — tiada markup AWS tetap. Kau juga bayar storan S3 biasa untuk data yang dihantar. Provider bayar kos publish/deliver. Exam: kos Data Exchange = ikut harga provider, bukan tarif AWS tetap.

Guna Bila

Subscribe to and access third-party datasets for analytics

Data Exchangethird-party datadata marketplacedata subscriptionmarket datafinancial datadata productsS3 deliverylicensingentitlementpricing

📎 What is AWS Data Exchange

D3 · High-Performing

AWS AI/ML Services

AWS AI Services — Polly, Rekognition, Lex, Comprehend, Textract, Transcribe, Translate

"Polly = cakap. Transcribe = dengar. Lex = faham + balas. Rekognition = nampak. Comprehend = baca. Textract = scan dokumen. Translate = tukar bahasa"

🎯 Sebab Apa Wujud

Wujud sebab kebanyakan syarikat tak ada data scientist atau data untuk latih model ML sendiri, tapi masih nak guna AI (transkrip suara, detect muka, analisa sentiment). AWS dah pre-train model ni dan expose sebagai API ready-to-use — kau call API terus, no training, no infra ML.

Apa Dia

AWS menyediakan pelbagai AI services ready-to-use: speech, vision, NLP, document processing.

AI Services Comparison

Amazon Polly → Text-to-Speech (TTS): convert text jadi audio (natural voice)

Amazon Transcribe → Speech-to-Text (STT): convert audio/video jadi text

Amazon Lex → Conversational chatbot: NLU + ASR, maintains context, integrates Lambda (powers Alexa)

Amazon Rekognition → Image/Video analysis: face detection, object/scene detection, labels

Amazon Comprehend → NLP: sentiment analysis, entities, key phrases, language detection

Amazon Textract → Document OCR: extract text, forms, tables from PDFs/images

Amazon Translate → Neural machine translation: convert text between languages, real-time + batch

Kinesis Video Streams → Ingest, store, process video/audio streams (for Rekognition real-time analysis)

Pilih AI service — input apa → output apa (decision tree)

Rendering diagram…

Pre-built API (no training) untuk audio/imej/text/chatbot; nak model custom → SageMaker. Pipeline klasik multilingual: Transcribe → Translate → Comprehend → Polly. INGAT exam: padankan input→output dengan keyword soalan.

Match the AI service to the use case (exam guna keyword input→output)

Service	Input → Output	Keyword soalan
Polly	Text → Speech (audio)	"read aloud", "voice", "audiobook"
Transcribe	Speech → Text	"transcribe", "caption", "meeting notes"
Translate	Text → Text (bahasa lain)	"translate", "multilingual", "localize"
Comprehend	Text → Insight (NLP)	"sentiment", "entities", "key phrases", "topic"
Lex	Text/Speech → Dialog (chatbot)	"chatbot", "conversation", "intent", "Alexa"
Rekognition	Image/Video → Labels/Faces	"face detection", "object", "moderation", "CCTV"
Textract	Document → Structured text	"OCR", "scanned PDF", "forms", "tables", "invoice"
SageMaker	Data → Custom model	"train your own", "custom ML", "hyperparameter"

Ingat: Pre-built API (Polly/Transcribe/Translate/Comprehend/Lex/Rekognition/Textract) = no training, call API terus. SageMaker = bina/latih model sendiri. Pipeline klasik: Transcribe → Translate → Comprehend → Polly. CCTV real-time = Rekognition + Kinesis VIDEO Streams.

⚡ Quick Sifir — hafal ni

▪Pre-built API (Polly/Transcribe/Translate/Comprehend/Lex/Rekognition/Textract) = NO training, call terus
▪Polly = Text→Speech; Transcribe = Speech→Text (jangan terbalik)
▪Comprehend = NLP sentiment/entity; Translate = tukar bahasa
▪Lex = chatbot (NLU+ASR, powers Alexa); Rekognition = image/video labels+faces
▪Rekognition sub-feature: Object/Scene Detection = pre-trained labels GENERIK (objek/haiwan/scene, NO training); Custom Labels = train managed model untuk objek/SPESIES khusus (Rekognition urus training, bukan from-scratch); Facial Analysis = muka MANUSIA (emosi/umur/jantina) BUKAN haiwan
▪Textract = OCR borang/jadual/invois dari PDF/imej
▪CCTV real-time = Rekognition + Kinesis VIDEO Streams
▪Latih model sendiri = SageMaker (bukan servis pre-built ni)

💡 Exam Scenario

"Baca soalan kuiz dengan suara" → Polly. "Transkrip meeting jadi nota" → Transcribe. "Chatbot customer service / Alexa" → Lex. "Detect muka dalam CCTV real-time" → Rekognition + Kinesis Video Streams. "Extract data dari invois/borang scan" → Textract. "Analisa sentiment review pelanggan" → Comprehend. "Translate app ke banyak bahasa" → Translate. "Latih model ramalan sendiri" → SageMaker.

🪤 Perangkap Soalan

Q: App perlu extract data berstruktur (nama, jumlah, tarikh) dari ribuan invois scan PDF. Servis mana?

⚠ Umpan: Amazon Rekognition — orang ingat 'analisa imej jadi Rekognition', tapi Rekognition untuk muka/objek/scene, bukan baca teks/borang/jadual dari dokumen.

✓ Betul: Amazon Textract — OCR khas untuk extract teks, forms, tables dari dokumen. Keyword: 'OCR', 'scanned PDF', 'forms/tables', 'invoice'.

Q: Bina pipeline: terima audio meeting bahasa Jepun → simpan transkrip Inggeris → analisa sentiment. Servis mana ikut turutan?

⚠ Umpan: Guna satu servis je macam Comprehend untuk semua — Comprehend NLP sahaja, tak boleh transkrip audio atau translate.

✓ Betul: Transcribe (audio→teks) → Translate (JP→EN) → Comprehend (sentiment). Setiap langkah satu pre-built API. Keyword: 'transcribe + translate + sentiment pipeline'.

Q: Wildlife conservation org nak analyze camera trap images (dalam S3) untuk monitor SPESIES haiwan, automatik, TANPA build & train custom model FROM SCRATCH. Servis mana?

⚠ Umpan: B. Rekognition Object and Scene Detection — umpan kuat sebab frasa 'no custom model' buat orang terus pilih pre-trained DetectLabels. Tapi DetectLabels bagi label GENERIK ('Animal', 'outdoor') — TAK boleh kenal SPESIES tertentu (harimau Malaya vs harimau Benggala). Keyword 'species' tu yang buka jawapan. (C Facial Analysis pun umpan kedua — tu untuk muka MANUSIA.)

✓ Betul: A. Rekognition Custom Labels — train model dengan imej berlabel spesies, Rekognition URUS training & hosting (managed, tak perlu kepakaran ML). Beza halus: 'from scratch' = SageMaker (D, yang di-exclude), BUKAN Custom Labels (managed training, masih kira 'tanpa bina dari awal'). Keyword: 'identify specific species + no from-scratch build' → Custom Labels. Generik label → DetectLabels.

🧠 Cara Mudah Ingat

→Polly = text → speech. Transcribe = speech → text. INGAT: P=produce speech, T=transcribe speech
→Lex = chatbot dengan context awareness. Kalau soalan sebut "chatbot", "natural language", "conversation turns" → Lex
→Rekognition + Kinesis VIDEO Streams = real-time video analysis (e.g. CCTV face mask detection, surveillance cameras)
→Rekognition + Kinesis DATA Streams = SALAH untuk video. Data Streams untuk text/structured records
→Rekognition ada 3 mod utama (jangan keliru): (1) Object & Scene Detection = pre-trained labels GENERIK (objek, haiwan, scene) — NO training; (2) Custom Labels = train MANAGED model dengan imej berlabel kau sendiri (untuk objek/SPESIES khusus — Rekognition urus training & hosting, BUKAN from-scratch); (3) Facial Analysis = muka MANUSIA (emosi, umur, jantina) BUKAN haiwan. Keyword mapping: "generik label + no training" → Object/Scene Detection; "specific species / custom object + no from-scratch build" → Custom Labels; "build model from scratch" → SageMaker. PERANGKAP: exam kata "without building/training a custom model FROM SCRATCH" — "from scratch" tu exclude SageMaker (D), BUKAN Custom Labels (A, managed). Dan "species" perlukan Custom Labels, bukan DetectLabels generik.
→Textract vs Comprehend: Textract = extract text FROM documents (OCR, forms, tables). Comprehend = analyze/understand text content (sentiment, entities)
→Exam shortcut: "read quiz questions aloud" → Polly. "CCTV face detection" → Rekognition + Kinesis Video Streams. "Chatbot" → Lex
→Polly StartSpeechSynthesisTask = async TTS: starts a long synthesis job and writes audio directly to S3 (use for large text files; SynthesizeSpeech is synchronous/streaming only)
→"scanned PDFs → audiobook" = Textract (extract text) + Polly (text → audio)
→Kinesis Video Streams = INGESTION layer (secure ingest from cameras/devices). Rekognition Video = ANALYSIS layer. You need both for a real-time surveillance pipeline.
→Comprehend = NLP text ANALYSIS (sentiment, entities, key phrases, topic modeling). Use for support tickets, social media, reviews. NOT for document OCR (use Textract) and NOT for chatbots (use Lex).
→Comprehend Medical = versi khusus perubatan — extract medical entities (penyakit, ubat, dosage, PHI) dari clinical text (nota doktor, rekod pesakit), HIPAA-eligible. Soalan sebut "medical records / clinical notes / extract PHI" → Comprehend Medical, bukan Comprehend biasa.
→Textract = EXTRACT structured data from scanned documents. Key-value pairs from forms, data from tables, dates and amounts from invoices/contracts. Goes beyond basic OCR.
→Lex = CONVERSATIONAL chatbot with multi-turn dialogue, intent recognition, slot filling. Powers Amazon Alexa. Manages conversation state.
→Enterprise search across PDFs/Word/email with natural language? → Amazon Kendra (see Kendra card). Product search with spell-check/synonyms? → OpenSearch.
→Translate = neural machine translation (text → text, different language). NOT speech (combine with Transcribe/Polly for audio) dan NOT sentiment analysis (use Comprehend)
→"Multilingual chatbot/support ticket pipeline": Transcribe (speech→text) → Translate (translate text) → Comprehend (analyze sentiment) → Polly (text→speech in target language)

Guna Bila

AI/ML services untuk audio, video, text, image analysis without training models

PollyTranscribeLexRekognitionComprehendComprehend MedicalTextractTranslateKinesis Video Streamstext-to-speechspeech-to-textchatbotimage analysisStartSpeechSynthesisTaskaudiobookOCRNLPsentiment analysisentity recognitiondocument extractionmachine translationPHIHIPAA

📎 Amazon Polly — text to speech 📎 Amazon Comprehend (& Comprehend Medical)📎 Amazon Rekognition 📎 Amazon Lex V2

D3 · High-Performing

SageMaker

Amazon SageMaker

"Custom ML end-to-end — train, tune, deploy your own models"

🎯 Sebab Apa Wujud

Wujud sebab bila masalah kau UNIK (ramal churn ikut data syarikat sendiri), servis AI pre-built (Rekognition, Comprehend) tak cukup — kau perlu latih model atas data sendiri. Tapi setup infra ML penuh (data prep, training cluster, tuning, deploy endpoint, MLOps) sangat leceh. SageMaker bungkus semua langkah end-to-end jadi platform managed.

Apa Dia

End-to-end managed ML platform: data prep (Data Wrangler, Feature Store), training (built-in algorithms, custom code in any framework), AutoML (Autopilot), HPO, model registry, and deployment (real-time, serverless, batch, async endpoints). Supports CI/CD via SageMaker Pipelines.

SageMaker — 4 cara deploy model (pilih ikut traffic pattern)

Endpoint type	Bila guna	Bil (cost)	Keyword exam
Real-time	Inference segera, traffic steady 24/7	Per instance-hour, jalan & bil non-stop walau idle	"low-latency", "always-on API"
Serverless Inference	Traffic intermittent/spiky, ada idle period	Per inference (auto scale → 0 bila tiada traffic)	"intermittent", "unpredictable", "scale to zero"
Batch Transform	Score dataset besar sekali-sekala (no live endpoint)	Per job (instance jalan masa job je)	"batch", "offline scoring", "whole dataset"
Async Inference	Payload besar / inference lama (sampai 1GB, beratur)	Auto scale, queue request, scale→0 bila kosong	"large payload", "long processing", "queue"

Ingat: Steady traffic → Real-time. Spiky/idle → Serverless Inference (scale to zero, jimat). Score banyak data sekali → Batch Transform. Payload besar/lama → Async. INGAT exam: real-time endpoint bil 24/7 walau tiada request — kalau soalan tekan "intermittent traffic + cost", jawapan Serverless Inference.

⚡ Quick Sifir — hafal ni

▪SageMaker = latih/deploy MODEL CUSTOM sendiri (data kau, algorithm kau)
▪Pre-built AI (Rekognition/Polly/Lex/Comprehend) = no training, call API je
▪Autopilot = AutoML: auto cuba algorithm + hyperparameter, pilih terbaik
▪Deploy options: real-time, serverless, batch, async endpoint
▪'train custom model on company data' → SageMaker; 'detect faces' → Rekognition
▪Pipelines = CI/CD untuk ML (MLOps)

💡 Exam Scenario

"Build a churn prediction model from historical data using custom Python code, tune hyperparameters, and deploy to a real-time endpoint" → SageMaker. Pre-built AI services (Polly, Lex, Rekognition, Comprehend) = no training needed, call the API. SageMaker = you control the model.

🪤 Perangkap Soalan

Q: Pasukan nak bina model ramalan churn guna kod Python custom, tune hyperparameter, dan deploy ke real-time endpoint. Servis mana?

⚠ Umpan: Amazon Comprehend / servis AI pre-built — nampak 'AI/ML jadi pilih AI service', tapi pre-built tak boleh latih model custom atas data kau atau tune hyperparameter.

✓ Betul: Amazon SageMaker — platform ML end-to-end untuk train custom + tune + deploy. Keyword: 'custom model', 'train your own', 'hyperparameter tuning'.

🧠 Cara Mudah Ingat

→SageMaker vs pre-built AI services: SageMaker = custom model training (your data, your algorithm). Rekognition/Polly/Lex/Comprehend = pre-trained, call API directly. Lihat card "AWS AI/ML Services" untuk decision tree pre-built vs custom.
→SageMaker Autopilot = AutoML: automatically tries different algorithms and hyperparameters, picks the best model
→"Train a custom model on company data" → SageMaker. "Detect faces in images" → Rekognition (no training needed)
→PRICING: (approx, us-east-1)tiada flat fee — bayar per ML instance-hour untuk setiap fasa (Notebook/Studio, Training job, Endpoint) + storage + data processing. ml.m5.xlarge ~$0.23/hr. Free tier 2 bulan pertama (jam terhad). Exam discriminator: real-time endpoint bil SELAGI ia "InService" (walau 0 request) — workload intermittent → guna Serverless Inference / matikan endpoint untuk jimat.

Guna Bila

Build, train, and deploy custom ML models with full control

SageMakercustom MLtrainingAutoMLAutopilothyperparameter tuningmodel deploymentMLOpsFeature StorePipelinesreal-time endpointserverless inferencebatch transformasync inferencepricing

📎 What is Amazon SageMaker AI 📎 Deploy models for inference