Neptune Fetcher is designed to separate data retrieval capabilities from the regular neptune package. This separation makes data fetching more efficient and improves performance.

pip install neptune-fetcher

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

for run in project.list_runs():
    print(run)  # dicts with identifiers

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

for experiment in project.list_experiments():
    print(experiment)  # dicts with identifiers

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

runs_df = project.fetch_runs_df(
    columns=["sys/custom_run_id", "sys/modification_time"],
    columns_regex="tree/.*",  # added to columns specified with the "columns" parameter
)

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

for run in project.fetch_read_only_runs(with_ids=["RUN-1", "RUN-2"]):
    run.prefetch(["parameters/optimizer", "parameters/init_lr"])

    print(run["parameters/optimizer"].fetch())
    print(run["parameters/init_lr"].fetch())

from neptune_fetcher import ReadOnlyProject, ReadOnlyRun

project = ReadOnlyProject("workspace/project")
run = ReadOnlyRun(project, with_id="TES-1")

run.prefetch(["parameters/optimizer", "parameters/init_lr"])
run.prefetch_series_values(["metrics/loss", "metrics/accuracy"], use_threads=True)

print(run["parameters/optimizer"].fetch())
print(run["parameters/init_lr"].fetch())
print(run["metrics/loss"].fetch_values())
print(run["metrics/accuracy"].fetch_values())

Representation of a Neptune project in a limited read-only mode.

Initialize with the ReadOnlyProject class constructor:

project = ReadOnlyProject("workspace/project", api_token="...")

Parameters:

Lists all runs of a project.

Each run is identified by Neptune ID (sys/id), custom ID (sys/custom_run_id) and, if set, name (sys/name).

Returns: Iterator of dictionaries with Neptune run identifiers, custom identifiers and names.

Example:

project = ReadOnlyProject()

for run in project.list_runs():
    print(run)

Lists all experiments of a project.

Each experiment is identified by:

• Neptune ID: sys/id
• (If set) Custom ID: sys/custom_run_id
• Name: sys/name

Example:

for experiment in project.list_experiments():
    print(experiment)

Returns: Iterator of dictionaries with Neptune experiment identifiers, custom identifiers and names.

Fetches a table containing Neptune IDs, custom run IDs and names of runs in the project.

Returns: pandas.DataFrame pandas.DataFrame with three columns (sys/id, sys/name and sys/custom_run_id) and one row for each run.

Example:

project = ReadOnlyProject()
df = project.fetch_runs()

Fetches a table containing Neptune IDs, custom IDs and names of experiments in the project.

Example:

df = project.fetch_experiments()

Returns: pandas.DataFrame with three columns (sys/id, sys/custom_run_id, sys/name) and one row for each experiment.

Fetches the runs' metadata and returns them as a pandas DataFrame.

Parameters:

Returns: pandas.DataFrame: A pandas DataFrame containing metadata of the fetched runs.

Specifically, you can fetch a data frame with a maximum of:

• 5000 columns, when using columns or columns_regex to filter columns.

Examples:

Fetch all runs, with specific columns:

project = ReadOnlyProject()

runs_df = project.fetch_runs_df(
    columns=["sys/modification_time", "training/lr"]
)

Fetch all runs, with specific columns and extra columns that match a regex pattern:

runs_df = project.fetch_runs_df(
    columns=["sys/modification_time"],
    columns_regex="tree/.*",
)

Fetch runs by specific ID:

specific_runs_df = my_project.fetch_runs_df(custom_ids=["nostalgic_shockley", "high_albattani"])

Fetch runs by names that match a regex pattern:

specific_runs_df = my_project.fetch_runs_df(
    names_regex="tree_3[2-4]+"
)

Fetch runs with a complex query:

runs_df = my_project.fetch_runs_df(query="(accuracy: float > 0.88) AND (loss: float < 0.2)")

Fetches the experiments' metadata and returns them as a pandas DataFrame.

Parameters:

Returns: pandas.DataFrame: A pandas DataFrame containing metadata of the fetched experiments.

Specifically, you can fetch a data frame with a maximum of:

• 5000 columns, when using columns or columns_regex to filter columns.

Examples:

Fetch all experiments with specific columns:

experiments_df = project.fetch_experiments_df(
    columns=["sys/custom_run_id", "sys/modification_time", "training/lr"]
)

Fetch all experiments with specific columns and extra columns that match a regex pattern:

experiments_df = project.fetch_experiments_df(
    columns=["sys/custom_run_id", "sys/modification_time"],
    columns_regex="tree/.*",
)

Fetch experiments by specific IDs:

specific_experiments_df = my_project.fetch_experiments_df(
    custom_ids=["nostalgic_shockley", "high_albattani"]
)

Fetch experiments with a complex query:

experiments_df = my_project.fetch_experiments_df(query="(accuracy: float > 0.88) AND (loss: float < 0.2)")

List runs of the project in the form of ReadOnlyRun.

Parameters:

Returns: Iterator of ReadOnlyRun objects.

Example:

project = ReadOnlyProject()

for run in project.fetch_read_only_runs(custom_ids=["nostalgic_shockley", "high_albattani"]):
    ...

Lists experiments of the project in the form of ReadOnlyRun.

Parameters:

Returns: Iterator of ReadOnlyRun objects.

Example:

project = ReadOnlyProject()

for run in project.fetch_read_only_experiments(names=["yolo-v2", "yolo-v3"]):
    ...

Representation of a Neptune run in a limited read-only mode.

Can be created

• with the class constructor:

  project = ReadOnlyProject()
  run = ReadOnlyRun(project, with_id="TES-1")

• or as a result of the fetch_read_only_runs() method of the ReadOnlyProject class:

  for run in project.fetch_read_only_runs(
      custom_ids=["nostalgic_shockley", "high_albattani"]):
      ...

Parameters:

Example:

from neptune_fetcher import ReadOnlyProject, ReadOnlyRun

project = ReadOnlyProject("workspace-name/project-name", api_token="...")
run = ReadOnlyRun(project, custom_id="high_albattani")

List of run field names.

A field is the location where a piece of metadata is stored in the run.

Returns: Iterator of run fields as strings.

Example:

for run in project.fetch_read_only_runs(custom_ids=["nostalgic_shockley", ...]):
    print(list(run.field_names))

Used to access a specific field of a run. See Available types.

Returns: An internal object used to operate on a specific field.

Example:

run = ReadOnlyRun(...)
custom_id = run["sys/custom_run_id"].fetch()

Pre-fetches a batch of fields to the internal cache.

Improves the performance of access to consecutive field values.

Supported Neptune field types:

• Boolean
• Datetime
• Float
• FloatSeries
• Integer
• ObjectState
• String
• StringSet

Parameters:

Example:

run = ReadOnlyRun(...)
run.prefetch(["parameters/optimizer", "parameter/init_lr"])
# No more calls to the API
print(run["parameters/optimizer"].fetch())
print(run["parameter/init_lr"].fetch())

Prefetches a batch of series to the internal cache.

Improves the performance of access to consecutive field values. Works only for series (FloatSeries).

To speed up the fetching process, this method can use multithreading. To enable it, set the use_threads parameter to True.

By default, the maximum number of workers is 10. You can change this number by setting the NEPTUNE_FETCHER_MAX_WORKERS environment variable.

Parameters:

Example:

run.prefetch_series_values(["metrics/loss", "metrics/accuracy"])
# No more calls to the API
print(run["metrics/loss"].fetch_values())
print(run["metrics/accuracy"].fetch_values())

This section lists the available field types and data retrieval operations.

Retrieves a bool value either from the internal cache (see prefetch()) or from the API.

Example:

status = run["sys/failed"].fetch()

Retrieves a datetime.datetime value either from the internal cache (see prefetch()) or from the API.

Example:

created_at = run["sys/creation_time"].fetch()

Retrieves a float value either from the internal cache (see prefetch()) or from the API.

Example:

f1 = run["scores/f1"].fetch()

Retrieves the last value of a series, either from the internal cache (see prefetch()) or from the API.

Returns: Optional[float]

Example:

loss = run["loss"].fetch_last()

Retrieves all series values either from the internal cache (see prefetch_series_values()) or from the API.

Parameters:

Returns: pandas.DataFrame

Example:

values = run["loss"].fetch_values()

Retrieves an int value either from the internal cache (see prefetch()) or from the API.

Example:

batch_size = run["batch_size"].fetch()

Retrieves the state of a run either from the internal cache (see prefetch()) or from the API.

Returns: str

Example:

state = run["sys/state"].fetch()

Retrieves a str value either from the internal cache (see prefetch()) or from the API.

Example:

token = run["token"].fetch()

Retrieves a dict of str values either from the internal cache (see prefetch()) or from the API.

Example:

groups = run["sys/group_tags"].fetch()

This project is licensed under the Apache License Version 2.0. For more details, see Apache License Version 2.0.