You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The intent was for metadata to be browsable via Dataset Viewer, in addition to each individual dataset, and to allow datasets to be loaded by specifying the config/name to load_dataset.
While testing load_dataset I encountered the following error:
The correct file is downloaded, however the incorrect builder type is detected; parquet due to other content of the repository. It would appear that the config needs to be taken into account.
Note that I have removed the additional configs from the repository because of this issue and there is a limit of 3000 configs anyway so the Dataset Viewer doesn't work as I intended. I'll add them back in if it assists with testing.
The text was updated successfully, but these errors were encountered:
Having looked into this further it seems the core of the issue is with two different formats in the same repo.
When the parquet config is first, the WebDatasets are loaded as parquet, if the WebDataset configs are first, the parquet is loaded as WebDataset.
A workaround in my case would be to just turn the parquet into a WebDataset, although I'd still need the Dataset Viewer config limit increasing. In other cases using the same format may not be possible.
Following documentation I had defined different configs for
Dataception
, a dataset of datasets:The intent was for metadata to be browsable via Dataset Viewer, in addition to each individual dataset, and to allow datasets to be loaded by specifying the config/name to
load_dataset
.While testing
load_dataset
I encountered the following error:The correct file is downloaded, however the incorrect builder type is detected;
parquet
due to other content of the repository. It would appear that the config needs to be taken into account.Note that I have removed the additional configs from the repository because of this issue and there is a limit of 3000 configs anyway so the Dataset Viewer doesn't work as I intended. I'll add them back in if it assists with testing.
The text was updated successfully, but these errors were encountered: