Delayed Data Ingestion from Cloud Storage to Datastore: How Long Should It Take?

[pedropcamellon]

I've set up automatic data ingestion from a Cloud Storage bucket to a Datastore following the instructions in the official documentation (https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-es#storage-periodic-sync). However, I'm experiencing some issues and would appreciate your insights:

1. Expected vs. Actual Ingestion Time:
- I set up the ingestion yesterday, but as of today, no txt files have been imported as documents in the Datastore.
- What's the typical timeframe for this process to complete?
- In the documentation, the default is 1 hour, but this was not the case.
```
After you set up your data source and import data the first time, data is synced from that source at a frequency that you select during setup. About an hour after the data connector is created, the first sync occurs.
```
- Is there a way to check the progress or status of the ingestion?
- Connector state is Active.

2. Troubleshooting Steps:
- Are there any common issues that might cause delays in ingestion?
- What logs or metrics should I check to diagnose potential problems?

3. Configuration Verification:
- How can I verify that my ingestion setup is correct?
- Are there any specific permissions or settings that are often overlooked?

In the documentation, it is specified that data cannot be manually refreshed.

Any guidance or insights would be greatly appreciated. Thank you in advance for your help!

[ruthseki]

Hi @pedropcamellon,

Welcome to Google Cloud Community!

Here are some insights into the issues you’re facing with the delayed data ingestion from Cloud Storage to Datastore:

1. Expected vs. Actual Ingestion Time:

According to this documentation:

Depending on the size of your data, ingestion can take several minutes to several hours.
After you set up your data source and import data the first time, data is synced from that source at a frequency that you select during setup. About an hour after the data connector is created, the first sync occurs. The next sync then occurs around 24 hours, 72 hours, or 120 hours later.

---

Data ingestion time can vary depending on factors like:

• Data Size: The volume of data in your Cloud Storage bucket. Larger datasets take longer.
• File Size: Individual file sizes can impact processing time.
• Network Bandwidth: Network latency and transfer rates play a role.
• Datastore Capacity: Datastore's current load and resource availability.

For checking the progress of ingestion process, you may try the following: 

• Cloud Logging: Check Cloud Logging for any errors or warnings related to the Datastore connector or the cloud-gen-ai-app-builder service.
• Datastore Console: Monitor the count of documents in your Datastore collection. If the number isn't changing, it suggests ingestion hasn't started.
• Cloud Storage Access: Ensure your Datastore connector has appropriate permissions to read files from the Cloud Storage bucket. Check that the bucket is accessible and not locked down.

2. Troubleshooting Steps:

Here are some common issues that might cause delays in ingestion:

• Incorrect Permissions: The Datastore connector needs sufficient permissions to read data from Cloud Storage. Verify access control lists (ACLs) for your bucket and make sure the connector's service account has the necessary roles.
• File Format Issues: Datastore expects specific file formats (e.g., .txt, .json, .csv). Ensure your files adhere to the expected structure.
• Datastore Limits: Check for any limits on Datastore operations, such as a limit on write operations or an overall data size limit.
• Data Connector Settings: Review the connector's settings. Ensure you've correctly selected the Cloud Storage bucket, the file format, and the synchronization frequency (if not defaulting to 1 hour).
• Datastore Region: Ensure that the Datastore location and the Cloud Storage bucket location are consistent.

Here are some logs and metrics to check to diagnose potential problems:

• Datastore Logs: Check the Cloud Logging section for the Datastore service. Search for errors or warnings related to ingestion or synchronization.
• Cloud Storage Access Logs: Examine Cloud Storage access logs to see if the connector is attempting to access the bucket and files.
• Datastore Connector Configuration:
  • Permissions Verification: Review the service account associated with your Datastore connector. Make sure it has the following roles or equivalent permissions:
    • Cloud Storage Object Viewer: Allows reading data from Cloud Storage.
    • Datastore Owner: Allows full access to Datastore.
  • Datastore Connection: Check if the Datastore connector has a valid connection to Datastore. Review your Datastore settings, including the connection string and the dataset name.

3. Configuration Verification:

• Cloud Console: Review the Datastore connector configuration in the Cloud Console to verify all settings are correct, including the Cloud Storage bucket, file format, and synchronization frequency.
• Permissions: Review the permissions assigned to the connector's service account. It should have permissions to read files from the Cloud Storage bucket and write data to Datastore.
• Datastore Settings: Ensure the correct Datastore dataset and namespace are specified in the connector configuration.

Here are some additional tips to consider:

• Test With a Small Dataset: Start by ingesting a small sample of data to identify any configuration issues or potential errors.
• Consider a Different Approach: If manual refresh isn't an option, explore alternative methods like:
• Cloud Functions: Trigger a function that imports data into Datastore whenever new files are added to the Cloud Storage bucket.
• Dataflow: Utilize Dataflow pipelines to perform data transformations and load data into Datastore.

I hope the above information is helpful.

[gerson_neto]

Is there a way to force the SYNC?
I had erros one day and it stopped updating the data

[ruthseki]

Hi @gerson_neto , 

For connecting to Cloud Storage with periodic syncing:

Data is synced periodically to the entity data store. You can specify synchronization daily, every three days, or every five days.

---

I hope this helps.

[gerson_neto]

Hi @ruthseki !

I have it set up to update daily, but since I had an error because I uploaded the wrong extension it is not updating anymore.
That is why I asked if is there a way to force update it.
Thanks for the