Dataform: Exceeded rate limits: too many updates for this routine.

[lucasferreira13]

Hello All,

Im running a dataform workflow with about 35 actions, majority of them has no dependency between each other. And I am facing the following error:
Exceeded rate limits: too many updates for this routine. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors

Its valid to point out that all of them are using incremental with merge approach. Looking at information_schema.jobs table inside bigquery i found the following:

Apparently its always relate to procedure operations when we create and drop it. I tried inside docs found out some quota related to this but the only one inside routine part that mentioned the update rate is the following:

Is that my case here? And if its how can I  avoid that? Since dataform use procedures for incremental tables for me this seems to be a huge blocker.
I tried to limit concurrent actions like what was mentioned here:
https://www.googlecloudcommunity.com/gc/Data-Analytics/Dataform-Exceeded-rate-limits-too-many-datase...
But It had no effect at all.

[Steexyz]

We’ve also had this issues for the last few days and we actually didn’t deployed anything new. There are only a couple of actions in the failing repository.

[ntrgdata24]

We've had the same issue too a few days ago. Did you take any remedial actions?

[spencerg]

We're having the same issue. We have quite a few Dataform pipelines running throughout the day, and every now and again it just runs one of the actions within a pipeline 2 or 3 times in immediate succession which is  causing UPDATE/MERGE failures and concurrent update failures.

However it was only when I went into the BQ  project history I figured out what was  happening, as Dataform itself is just showing a single execution with no evidence of scripts running multiple times.

I started noticing issues on Friday afternoon and they continued yesterday. There doesn't appear to be any rhyme or reason for which pipelines  it's  affecting though. and is only affecting a small number of processes.

[dalirmw]

Hello, 

We are facing the same issue. Our Dataform pipelines are randomly executing some tables more than once (2-3 times), and it’s not the same table each time.

[spencerg]

I have opened a case with Google to see if they have any idea what is happening.

[mpinlac]

Hi @lucasferreira13,

As the case has been resolved before, and as I see most are having issues with it, I would like to suggest you to Create new BigQuery Issue and you can track progress at the List of open BigQuery issue.

You can contact Google Cloud Support for a more in-depth analysis. When contacting them, please provide comprehensive details and include screenshots. This will help them better understand and address your issue.

I hope this helps.