Dataform: "Procedure is not found" error occurring more frequently

[ntrgdata24]

Hi all,

I'm running multiple Dataform workflows across projects for many months now. But over the last week or so, this particular error which says "Procedure not found: <procedure name>" is increasing in frequency. I am guessing it refers to the SQL procedure Dataform creates to execute its actions.

While I have a recovery mechanism, I would of course like to avoid encountering the error in the first place. Can someone please help me understand why it happens and how can I avoid this error?

[NorieRam]

Hi @ntrgdata24,

Welcome to Google Cloud Community!

The "Procedure not found: <procedure name>" error in Dataform usually means Dataform can't find the temporary procedure it creates to execute your actions.

One potential solution is to add the workflow invocation ID to procedure names in the generated SQL. This should help prevent name collisions across different executions and ideally reduce those "Procedure Not Found" errors.

You may also try this workarounds as well, for it may help resolve the problem:

• Retry your Dataform actions with increasing delays to handle temporary network or warehouse issues.
• Avoid running Dataform workflows during peak load times on your data warehouse to minimize resource contention.
• For simpler actions, consider rewriting them as direct SQL queries within Dataform to bypass procedure creation entirely.
• Proactively monitor your data warehouse performance and Dataform logs to identify any patterns or underlying issues.

If the issue persists, reach out to Google Cloud support for assistance. They can investigate further using more information that you have.

I hope the above information is helpful.