161
questions
2
votes
0
answers
15
views
How to handle frequently refreshed OIDC token in DVC repo
I have a DVC repo that is thought for storing data. I have a WebDAV remote. The authentication for the remote is done via oidc. I store the oidc token in the local.config file with
dvc remote modify -...
- 21
1
vote
1
answer
31
views
Incremental add with DVC tracked directories
While I know that the main use case of DVC comes after the "data engineering" parts, I have written something that works quite nice for me but is missing one feature.
So nightly i run a ...
- 129
1
vote
1
answer
29
views
remove file from GCS with DVC
Is there any way to remove a credential file that I've 'dvc pushed' to GCS?
I can't access the GCS interface via the browser either
I tried to do dvc rm data/jsons/creds.json but it says that 'dvc....
- 11
1
vote
2
answers
76
views
How to register dependencies programmatically in a Python DVC pipeline
I want to run a sequence of experiments and each experiment will use certain input data files (dependencies), each of which I want to prepare when an experiment is run. (Some experiments will use the ...
- 11.5k
0
votes
0
answers
32
views
Can you have a foreach loop above stages in a Python DVC pipeline?
I want to use the foreach-do construct to iterate over a series of experiments.
From the docs, I understand the standard approach is as follows:
dvc.yaml
stages:
prepare_data:
foreach: ${...
- 11.5k
1
vote
1
answer
43
views
DCV ERROR: Failed to import "file" due to SCM error: "github.com/thursday/myrepo". name 'urllib3' is not defined
I tried using dvc to import my zipped data using the command:
`dvc import https://github.com/thursday/myrepo xyz.zip -o data/myrepo/xyz.zip'.
This is my github :"https://github.com/...
- 11
0
votes
0
answers
55
views
Module not found error using dvc repro, issues with pandas and pyenv
OS: windows 10
I cloned my teammate's git repo, but when I run dvc repro I get an error saying that there is no module named pandas.
project_path (main)
$ dvc repro
Running stage 'data_collection':
&...
- 415
0
votes
0
answers
47
views
Can I create a dynamic variable that gets a new value everytime for dvc.yaml
I'm aiming to incorporate a unique identifier into my dvc.yaml file every time I execute dvc repro. I'm using a Python script, generate_uuid.py, to generate a UUID and store it in a JSON file. Then, ...
- 99
1
vote
1
answer
126
views
Extract current running stage from dvc
I'm conducting an experiment using 'dvc repro -f', where multiple stages are executed according to the dvc.yaml configuration. For instance:
Stages:
Training:
foreach:
-cycle: 0
-cycle: 1
...
- 99
2
votes
1
answer
264
views
How can I download data from just one of the DVC repositories?
I have a project that uses several databases, to avoid versioning huge files in git, I used DVC to manage it on gdrive.
I followed the following step by step on DVC
Start DVC (dvc init)
dvc add #...
3
votes
1
answer
104
views
Adding data using dagshub.upload.Repo(USER_NAME,REPO_NAM)
I want to add a raw dataset file to my dagshub repo (my first repo, and its being used alongside an MLflow tutorial)
This is the line that is giving me trouble:
repo = dagshub.upload.Repo(USER_NAME,...
- 195
1
vote
0
answers
95
views
DVC using cached run although parameter changed
I am trying to perform pipeline tracking using dvc. The problem is, that if i change for example the size parameter from the params.yaml, it does not rerun the stage but simply uses a cached run, ...
- 11
1
vote
1
answer
61
views
Paramater-based dependecies and outs in DVC from constants file
I am trying to define a single-source set of paths such that it can be modified if necessary from a single spot rather than modifying it in various places across many scripts. I am doing this by ...
- 1,575
0
votes
0
answers
57
views
conda can't activate existing virtual enivrement
I'm trying to learn how to automate data science project using dvc and cookiecutter as project structure, I made conda envirument and installed all my libraries in venv file.
everything was working ...
1
vote
0
answers
39
views
dvc push: local variable referenced before assigment
Error while pushing files to DVC:
dvc push
ERROR: unexpected error - local variable 'paths' referenced before assignment
Having any troubles? Hit us ...
- 11