export/import datafusion pipeline schedule

[vemulakarthik]

Hello All,

I am trying to export and import a datafusion pipeline which is scheduled in DEV to STAGE, however when i export i am able to see the schedule in Json file, but when i import the Json file to stage i am able to import only the pipeline but not the schedule, why is it so? any one has any other alternative to export/import datafusion pipeline including scheduling, its real pain to schedule in STAGE one after one and will be helpful if i can import including schedule.

I tried with HTTP get which is working as expcted, but HTTP PUT method is not working

if any one show me some insight on how to export and import triggers from dev to stage ? i would like to avoid manual intervention

[ms4446]

The issue you're encountering with Google Cloud Data Fusion is likely because the pipeline export and import functionality does not automatically include schedules or triggers. When you export a pipeline, the schedules are typically stored in a separate part of the configuration and are not automatically applied during import.

To resolve this, you can try the following approach:

Steps to Export and Import Triggers in Data Fusion:

1. Export the Pipeline and Schedules:

   • Use the HTTP GET method to export both the pipeline and schedule configuration. You’ve mentioned this works as expected, so continue to retrieve the schedule information in a separate API call.

2. Manually Apply the Schedules:

   • When moving the pipeline to STAGE, you may need to use the Data Fusion REST API to create the schedules manually since the pipeline import does not include them. This can be done by:

     • Retrieving the schedule details from the JSON using HTTP GET.

     • Creating a new schedule in STAGE using the HTTP PUT or POST method to apply the same schedule.

3. Automating Schedule Application with the Data Fusion API:

   • Export the Trigger: Use GET on the Data Fusion /{namespace}/pipelines/{pipeline}/schedules endpoint in DEV to export the schedule.

   • Recreate the Trigger in STAGE: After importing the pipeline into STAGE, use the PUT or POST method on /{namespace}/pipelines/{pipeline}/schedules to recreate the schedule. You may need to adjust the PUT method, ensuring the correct parameters and payload structure are used when calling it.

4. Script Automation: If you want to avoid manual intervention:

   • Write a script that automates these HTTP requests:

     • Export the pipeline and schedule configuration from DEV.

     • Import the pipeline to STAGE.

     • Reapply the schedule using the extracted schedule JSON through the API.

[jangemmar]

@vemulakarthik In addition, feel free to explore the 3 ways to restrict access to your Cloud Data Fusion instance and pipelines and Building a Cloud Data Fusion pipeline for your future reference.

I hope this helps.