Python Powers Up: The Rise of the Python API for Earth Engine
By Justin Braaten, Developer Relations; Alex Merose, Software Engineer; & Nate Schmitz, Software Engineer
For the past 10 years, the JavaScript Code Editor has been the leading character in the Earth Engine story. Its purpose-built user interface, integrated map, drawing tools, console, inspector, and extensive documentation have overshadowed the Python API, its unadorned sidekick. However, a plot twist is brewing: the Earth Engine Python API is revealing new powers… 🦸🏽♀️⚡
Earth Engine has long had two primary interfaces: the JavaScript Code Editor and the Python client library. They share essentially the same language for describing data classes and methods, but the user experience can be vastly different. And, despite the appearance, the Python API has historically had limited interoperability with other Python libraries and data models. We recognize that the Python environment is where many of you are doing analyses and we want to meet you there. Over this last year, the Earth Engine team and its friends have been working to improve the EE Python experience and expand its reach within the ecosystem. At the Geo for Good 2023 summit we showcased three new super powers of the Python API.
Interactive development with geemap
We’ve partnered with Dr. Qiusheng Wu to support the amazing geemap library and adopt it as the UI for Python API examples in the Earth Engine Developer Guide. Until recently, the lack of a Python development environment has limited EE Python’s use. Dr. Wu solved the problem with geemap, a Python package that makes it easy to visualize and explore data from Earth Engine in a Jupyter notebook. We know a lot of you are already using it and we’re excited to now include geemap throughout the EE Developer site as the tool for displaying interactive maps and rich object representations.
Earlier this year, Dr. Wu moved the geemap repository to the gee-community organization and we began contributing pull requests to add unit tests, refactor features, and implement a core module, which provides the essential base classes and functionality for the map interface. Core functionality includes displaying map tiles, inspecting map tiles, drawing tools, tile and base layer management, and rich object representation. We anticipate adding more to the core module in the future and would love to hear what functionality is most important to you.
Using geemap, we’ve been able to translate many code examples in the EE Developer site to Python, including guide pages, dataset examples, and API reference snippets. We’ve also been working with the Google Colab team to make Colab a convenient notebook environment for interacting with Earth Engine and geemap, both of which are now pre-installed. And, with Colab now offering enterprise features, it’s a great cloud-based notebook solution for operational and research applications. Give it a try!
Earth Engine as Arrays
We’re happy to announce the launch of Xee, a new way of using Earth Engine. With Xee, you can work with Earth Engine’s data catalog right from your favorite tool for managing n-dimensional arrays — Xarray. We’re most excited about what this means for bringing Earth Engine to the scientific Python ecosystem, where Xarray has grown in importance for geosciences.
We believe that Xarray offers the most convenient way to work with pixels from Earth Engine: it allows you to operate on Earth Engine ImageCollections as Xarray Datasets. Using Xarray’s lazy evaluation (powered by Dask), you can stream in data from Earth Engine’s vast data catalog “just in time” for analysis. Now, there’s no need for exporting Earth Engine data to get the full power of the SciPy or Pangeo Data ecosystem. Further, Xee allows you to integrate with the standard Python client, allowing server-side processing within EE. This lets you use your favorite parts of each API — Xarray or Earth Engine’s compute platform — seamlessly together.
Speaking of seamless, this new Earth Engine client melds well with Xarray-Beam. Together, it takes about ~25 lines of code to export any Earth Engine ImageCollection to Zarr (a cloud-optimized format for big data). When run in parallel on Dataflow, you can export terabytes of Earth Engine data incredibly fast. Whereas before, the 3000-task limit for batch Earth Engine export made orchestrating data egress cumbersome, you now can effortlessly leverage EE’s high-volume API (via computePixels). Instead of days to export a dataset, it now takes hours. Xarray-Beam makes this process simple, flexible, and powerful.
Data converters
At the beginning of August, we launched two new methods in the Python client library, getPixels and computePixels (Pixels to the People!). We’re excited to announce the next step in our Data Extraction API journey: data converters!
Data converters are client-side conversion capabilities built into getPixels, computePixels, listFeatures, and computeFeatures. They provide a better base interface for working with the scientific Python ecosystem. By specifying a compatible fileFormat, these methods can now return data in Python-native formats like structured NumPy arrays for rasters and Pandas DataFrames or GeoPandas GeoDataFrames for vectors. In the case of vectors, the listFeatures and computeFeatures methods will actually make several network requests to fetch all the pages of the table before returning the Python object. Fetching entire tables without going through the batch stack is now easier and more scalable.
With these new converters you can seamlessly access and reduce data with Earth Engine, leverage Pandas’ intuitive and expressive API for powerful table operations and charting, perform fast geo filtering and visualization with GeoPandas’ comprehensive geospatial toolkit, and harness NumPy’s blazing-fast array operations for pixel-level analysis. We put together a Colab notebook to help you get started. We’ll be interested to hear how you use these converters and what additional formats you’d like in the future.
If you’re a dedicated Code Editor user, we hope these enhancements to the EE Python experience will entice you to try it, and if you’re already working with the Python API, we hope the new interoperability tools will help supercharge your analyses.
Stay tuned for the next chapter where we continue the story of the rise of the Earth Engine Python API.
