Dataflow

Overview

• Streaming analytics service
• Transform and enrich data in stream and batch modes with a single model
  • Avoid Training-Serving Skew
• Serverless—managed service, horizontal scaling
• NoOps—automated maintenance, monitoring and scaling
• Execution engine for Apache Beam
  • Open source
  • Beam SDK—Java, Python, Go
  • Dataflow is the backend for the Beam API

Templates

• Streaming
  • Pub/Sub to BigQuery
  • Pub/Sub to Cloud Storage
  • Datastream to BigQuery
  • Pub/Sub to MongoDb
• Batch
  • BigQuery to Cloud Storage
  • Cloud Bigtable to Cloud Storage
  • Cloud Storage to BigQuery
  • Cloud Spanner to Cloud Storage
• Utility
  • Bulk compression in Cloud Storage
  • Cloud Firestore bulk delete
  • File format conversion

References

• Google Cloud Decision Trees