Google Cloud Data Lifecycle

Ingest

Stream

• Pub/Sub

Batch

• Cloud Storage
  • Storage Transfer Service—moving large amounts of data from on-prem
• BigQuery
  • BigQuery Transfer Service—moving from SaaS e.g. YouTube, Google Ads
• Big Data (e.g. PBs)
  • Storage Transfer Appliance

Application

• Cloud Logging
• Pub/Sub
• Cloud SQL
• Cloud Firestore—noSQL
• Cloud Bigtable—large amounts of noSQL
• Cloud Spanner—large amounts of structured data

Storage

Objects

• Cloud Storage
• Cloud Storage for Firebase

Databases

• SQL
  • Cloud SQL
  • Cloud Spanner
• noSQL
  • Cloud Bigtable
  • Cloud Firestore

Warehouse

• BigQuery

Process / Analyse

Compute

• Compute Engine
• Kubernetes Engine
• App Engine

Large Scale

• Dataproc—Hadoop, Spark etc.
• Dataflow—Apache Beam
• Dataprep

Analyse

• BigQuery—PBs of data analysis at speed

Explore / Visualize

Science

• Cloud Datalab—Jupyter notebooks

Visualize

• BigQuery BI
• Looker Studio
• Looker

References

• Data Lifecycle