Sample DAG with SDK #2
Conversation
| ) | ||
| def sample_list_ch_clusters(): | ||
| @task | ||
| def get_project_id(): |
There was a problem hiding this comment.
I suggest changing this could to a dag parameter instead, see how it is used in https://github.com/doublecloud/sample-airflow-dags/blob/trunk/dags/sample_ch_insert.py#L25-L28
When using decorator, dag/params are accessible through context variables, see
| )) | ||
| print("Your CH clusters are:") | ||
| for cluster in response.clusters: | ||
| print(cluster) |
There was a problem hiding this comment.
This is good, but I was thinking if we can go even further and create connections for the user?
Here's how that is done with gcp sql query
https://github.com/apache/airflow/blob/8f7616c47d5a89da83dc0a565d86fd3f618f5612/tests/system/providers/google/cloud/cloud_sql/example_cloud_sql_query.py#L442
If we can do something similar, it would make it super-useful
Alternatively, if we want to just make this as a "print all stuff we have" I'd suggest adding more similar tasks to report clusters and other entities in the account — kafka, clickhouse, transfer endpoints
There was a problem hiding this comment.
Could you please also add a short note in README with a screenshot of a DAG structure?
Added a simple DAG to show Integration with our SDK