Categories
Software development Technology

GCP Data Fusion

Reading Time: < 1 minute

(Update) Tried running couple more pre-set pipelines from google. It took a while to run (don’t know why). More on permissions (IAMs): need to add “Dataproc Worker” role to “Compute Engine default service account”. Continue added Service Account User to “Cloud Data Fusion Service Account / Cloud Data Fusion API Service Agent”

Couple tutorials

Targeting campaign pipeline

Creating a reusable pipeline

Permission issue (note the exact error will depends on the setup of network as well, for example, this DF service account needs to have network access to run the pipeline, and it needs that role if applicable).

Cost: the developer edition for data fusion instance costs 35 cents per hour. The basic edition is 1.80 per hour but comes with first 120 hours free, this is 5 days free usage and recommended. Also, there is ways in GCP to set up budgets and alerts.

Leave a Reply

Your email address will not be published. Required fields are marked *