Data Analytics

Build a hybrid data processing footprint using Dataproc on Google Distributed Cloud

May 31, 2024

Antonio Scaramuzzino

Senior Product Manager

Chris Nauroth

Senior Staff Software Engineer

Try Gemini 1.5 models

Google's most advanced multimodal models in Vertex AI

Try it

Google Cloud customers interested in building or modernizing their data lake infrastructure often need to maintain at least part of their workloads and data on-premises, because of regulatory or operational requirements.

Thanks to Dataproc on Google Distributed Cloud, introduced in preview at Google Cloud Next ‘24, you can now fully modernize your data lake with cloud-based technology, while building hybrid data processing footprints that allow you to store and process on-prem data that you can’t move to the cloud.

Dataproc on Google Distributed Cloud lets you run Apache Spark processing workloads on-prem, using Google-provided hardware located within your data center, while maintaining consistency between the technology you use in the cloud and locally.

For example, a large telecommunications company in Europe is modernizing their data lake on Google Cloud, while keeping Personally Identifiable Information (PII) data on-prem, on Google Distributed Cloud, to satisfy regulatory requirements.

In this blog, we will show how to use Dataproc on Google Distributed Cloud to read on-prem PII data, calculate aggregate metrics, and upload the resulting dataset to the data lake on the cloud using Google Cloud Storage.

Aggregate and anonymize sensitive data on-prem

In our demo scenario, the customer is a telecommunications company storing event logs that record users’ calls:

customer_id	customer_name	call_duration	call_type	signal_strength	device_type	location
1	<redacted>	141	Voice	379	LG Q6	Tammieview, PA
2	<redacted>	26	Video	947	Kyocera Hydro Elite	New Angela, FL
3	<redacted>	117	Voice	625	Huawei Y5	Toddville, MO
4	<redacted>	36	Video	382	iPhone X	Richmondview, NV
5	<redacted>	110	Video	461	HTC 10 evo	Cowanchester, KS
6	<redacted>	0	Video	326	Galaxy S7	Nicholsside, NV
7	<redacted>	200	Data	448	Kyocera Hydro Elite	New Taramouth, AR
8	<redacted>	178	Data	475	Galaxy S7	South Heather, CT
9	<redacted>	200	Voice	538	Oppo Reno6 Pro+ 5G	Gregoryburgh, ID
10	<redacted>	113	Voice	878	ZTE Axon 30 Ultra 5G	Karaview, NV
11	<redacted>	200	Data	722	Huawei P10 Lite	Petersonstad, IA
12	<redacted>	200	Voice	1	HTC 10 evo	West Danielport, CO
13	<redacted>	169	Voice	230	Samsung Galaxy S10+	North Jose, SD
14	<redacted>	198	Voice	1	Kyocera DuraForce	East Matthewmouth, AS
15	<redacted>	155	Data	757	Oppo Find X	Tuckerchester, MD
16	<redacted>	0	Data	1	ZTE Axon 30 Ultra 5G	New Tammy, NC
17	<redacted>	200	Data	656	Galaxy Note 7	East Jeanside, NJ
18	<redacted>	15	Data	567	Huawei Y5	Lake Patrickburgh, OH

This dataset contains PII. For regulatory compliance, PII must remain on-prem in their own data center. To satisfy this requirement, the customer S3-compatible object storage on-premise to store this data. However, now the customer would like to use their broader data lake in Google Cloud to analyze signal_strength by location and identify the best areas for new infrastructure investments.

To integrate with Google Cloud Data Analytics while still satisfying compliance requirements, Dataproc on Google Distributed Cloud supports full local execution of Spark jobs that can perform an aggregation on signal_quality. Consider this sample Spark code:

import argparse

from pyspark.sql import SparkSession
import pyspark.sql.functions as F

parser = argparse.ArgumentParser()

parser.add_argument("--input", default="s3a://event-logs/gdc-demo/dataset.tsv")
parser.add_argument("--output", default="")

args = parser.parse_args()

# Create SparkSession
spark = SparkSession.builder.appName("demo-query").getOrCreate()

# Read data
print("Reading data from %s" % args.input)
df = spark.read.csv(args.input, sep=r"\t", header=True)

# Find weighted average signal strength by location
out = df.select("call_duration", "signal_strength", "location").withColumn(
    "adj_duration", F.col("call_duration") + 1
).withColumn(
    "signal_x_duration", F.col("adj_duration") * F.col("signal_strength")
).groupBy(
    "location"
).agg(
    F.sum("adj_duration").alias("total_call_duration"),
    F.sum("signal_x_duration").alias("total_signal_x_duration"),
).withColumn(
    "weighted_avg_signal_strength",
    F.col("total_signal_x_duration") / F.col("total_call_duration"),
).select(
    "location", "weighted_avg_signal_strength"
).orderBy(
    F.asc("weighted_avg_signal_strength")
)

out.show()

if args.output:
  print("Saving output to %s" % args.output)
  out.coalesce(1).write.option("delimiter", "\t").csv(args.output)

Dataproc on GDC exposes custom resources in the Kubernetes Resource Manager API to support Spark application submission. First, users obtain credentials to the GDC cluster:

Then, users can run the job shown above by creating a SparkApplication custom resource and specifying the input location from local object storage and the output location to Cloud Storage:

The resulting output in Cloud Storage identifies several areas of low signal quality:

Location	Value
Georgefurt, MS	1.0
Scottside, MA	1.0
Monroemouth, FL	1.0
Lake Robert, OH	1.0
East Lauren, VA	1.0
Shelleyburgh, CT	1.0
Buckville, ID	1.0
Garzaton, WI	3.32
North Danielle, NY	3.99
Port Natalie, ID	5.43

This data set is now available in Cloud Storage, with PII removed, as part of the customer’s broader GCP data lake strategy. This opens the possibility of additional analysis, such as trending over time, or using multiple data analytics products such as BigQuery and Dataproc Serverless.

Learn more

In this blog, we saw how you can leverage Dataproc on Google Distributed Cloud to create hybrid data processing footprints, processing on-prem sensitive data that needs to remain in your datacenter, and moving the rest of your data to the cloud. Dataproc on Google Distributed Cloud lets you modernize your data lake while respecting regulatory and operational data residency requirements. To learn more about Dataproc and Google Distributed Cloud, please visit:

Posted in

Data Analytics

How to simplify building RAG pipelines in BigQuery with Document AI Layout Parser

By Jeff Nelson • 5-minute read

Data Analytics

Getting started with NL2SQL (natural language to SQL) with Gemini and BigQuery

By Bernard Chang • 8-minute read

Data Analytics

A practical guide to synthetic data generation with Gretel and BigQuery DataFrames

By Jiaxun Wu • 4-minute read

Data Analytics

BigQuery's AI-assisted data preparation is now in preview

By Tim Bezold • 4-minute read

Build a hybrid data processing footprint using Dataproc on Google Distributed Cloud

Antonio Scaramuzzino

Chris Nauroth

Try Gemini 1.5 models

Aggregate and anonymize sensitive data on-prem

Learn more

Related articles

How to simplify building RAG pipelines in BigQuery with Document AI Layout Parser

Getting started with NL2SQL (natural language to SQL) with Gemini and BigQuery

A practical guide to synthetic data generation with Gretel and BigQuery DataFrames

BigQuery's AI-assisted data preparation is now in preview