Skip to content

Commit

Permalink
Update docs (#257)
Browse files Browse the repository at this point in the history
  • Loading branch information
iesahin authored Jul 15, 2024
1 parent fab1536 commit 1fd1e98
Show file tree
Hide file tree
Showing 20 changed files with 76 additions and 49 deletions.
3 changes: 3 additions & 0 deletions book/src/ref/xvc-file-copy.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,11 @@ If the source files you specify are changed, Xvc cancels the copy operation. Ple
$ perl -i -pe 's/a/ee/g' data.txt

$ xvc file copy data.txt data5.txt
? 1
[ERROR] File Error: Sources have changed, please carry-in or recheck following files before copying:
data.txt
Error: FileError { source: AnyhowError { source: Sources have changed, please carry-in or recheck following files before copying:
data.txt } }

```

Expand Down
2 changes: 2 additions & 0 deletions book/src/ref/xvc-file-list.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,9 @@ a repository yet, it reports an error.

```console
$ xvc file list
? 1
[ERROR] File Error: [E2004] Requires xvc repository.
Error: FileError { source: RequiresXvcRepository }

```

Expand Down
8 changes: 4 additions & 4 deletions book/src/ref/xvc-file-send.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@
$ xvc file send --help
Send (push, upload) files to external storages

Usage: xvc file send [OPTIONS] --remote <REMOTE> [TARGETS]...
Usage: xvc file send [OPTIONS] --storage <STORAGE> [TARGETS]...

Arguments:
[TARGETS]... Targets to send/push/upload to storage

Options:
-r, --storage <REMOTE> Storage name or guid to send the files
--force Force even if the files are already present in the storage
-h, --help Print help
-s, --storage <STORAGE> Storage name or guid to send the files
--force Force even if the files are already present in the storage
-h, --help Print help

```
15 changes: 6 additions & 9 deletions book/src/ref/xvc-file-share.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@
$ xvc file share --help
Share a file from S3 compatible storage for a limited time

Usage: xvc file share [OPTIONS] --storage <REMOTE> <TARGET>
Usage: xvc file share [OPTIONS] --storage <STORAGE> <TARGET>

Arguments:
<TARGET> File to send/push/upload to storage

Options:
-r, --storage <REMOTE> Storage name or guid to send the files
-s, --storage <STORAGE> Storage name or guid to send the files
-d, --duration <DURATION> Period to send the files to. You can use s, m, h, d, w suffixes [default: 24h]
-h, --help Print help

Expand Down Expand Up @@ -58,30 +58,27 @@ $ xvc storage new s3 --name backup --bucket-name xvc-test --region eu-central-1
You must first send files to the remote storage.

```console
$ xvc file send --remote backup dir-0001/
$ xvc file send --storage backup dir-0001/
```

Now you can share the files. It will create a URL for you to share that file. (Here we use cut to make the command repeatable)

```console
$ zsh -cl 'xvc file share --remote backup dir-0001/file-0001.bin | cut -c -50'
https://xvc-test.s3.eu-central-1.amazonaws.com/xvc
$ zsh -cl 'xvc file share --storage backup dir-0001/file-0001.bin | cut -c -50'

```

Note that the default period is 24 hours. You can set another period with `--duration`.

```console
$ zsh -cl 'xvc file share --duration 1h --remote backup dir-0001/file-0002.bin | cut -c -50'
https://xvc-test.s3.eu-central-1.amazonaws.com/xvc
$ zsh -cl 'xvc file share --duration 1h --storage backup dir-0001/file-0002.bin | cut -c -50'

```

You can get another URL for a shared file with a different period.

```console
$ zsh -cl 'xvc file share --duration 1m --remote backup dir-0001/file-0002.bin | cut -c -50'
https://xvc-test.s3.eu-central-1.amazonaws.com/xvc
$ zsh -cl 'xvc file share --duration 1m --storage backup dir-0001/file-0002.bin | cut -c -50'

```

Expand Down
2 changes: 2 additions & 0 deletions book/src/ref/xvc-pipeline-import.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,9 @@ If you don't supply the `--overwrite` option, Xvc will report an error and quit.

```console
$ xvc pipeline import --file pipeline.yaml
? 1
[ERROR] Pipeline Error: Pipeline default already found
Error: PipelineError { source: PipelineAlreadyFound { name: "default" } }

```

Expand Down
6 changes: 4 additions & 2 deletions book/src/ref/xvc-pipeline-run.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,10 @@ You can run the default pipeline without specifying its name.
```console
$ xvc pipeline run
[OUT] [hello] hello

[DONE] hello (echo hello)


```

Note that, when a step has no dependencies, it's set to always run if it's not set to run never explicitly.
Expand All @@ -83,7 +84,8 @@ $ xvc pipeline --pipeline-name my-pipeline step new --step-name my-hello --comma
```console
$ xvc pipeline run --pipeline-name my-pipeline
[OUT] [my-hello] hello from my-pipeline

[DONE] my-hello (echo 'hello from my-pipeline')


```
9 changes: 6 additions & 3 deletions book/src/ref/xvc-pipeline-step-dependency-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@ When you run the command, it will print `data.txt has changed` if the file `data
```console
$ xvc pipeline run
[OUT] [file-dependency] data.txt has changed

[DONE] file-dependency (echo data.txt has changed)


```

You can add multiple dependencies to a step with multiple invocations.
Expand All @@ -43,9 +44,10 @@ A step will run if any of its dependencies have changed.
```console
$ xvc pipeline run
[OUT] [file-dependency] data.txt has changed

[DONE] file-dependency (echo data.txt has changed)


```

By default, they are not run if none of the dependencies have changed.
Expand All @@ -66,7 +68,8 @@ Now the step will run even if none of the dependencies have changed.
```console
$ xvc pipeline run
[OUT] [file-dependency] data.txt has changed

[DONE] file-dependency (echo data.txt has changed)


```
9 changes: 6 additions & 3 deletions book/src/ref/xvc-pipeline-step-dependency-generic.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,10 @@ The step won't run until tomorrow, when `date +%F` changes.
```console
$ xvc pipeline run
[OUT] [morning-message] Good Morning!

[DONE] morning-message (echo 'Good Morning!')


```

You can mimic all kinds of pipeline behavior with this generic dependency.
Expand All @@ -50,9 +51,10 @@ $ xvc pipeline step dependency --step-name directory-contents --generic 'ls'

$ xvc pipeline run
[OUT] [directory-contents] Files changed

[DONE] directory-contents (echo 'Files changed')


```

When you add a file to the directory, the step is invalidated and run again:
Expand All @@ -63,7 +65,8 @@ $ xvc pipeline run
$ xvc-test-helper generate-random-file new-file.txt
$ xvc pipeline run
[OUT] [directory-contents] Files changed

[DONE] directory-contents (echo 'Files changed')


```
9 changes: 6 additions & 3 deletions book/src/ref/xvc-pipeline-step-dependency-glob-items.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,10 @@ dir-0002/file-0003.bin

### Changed Files:


[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}")


$ xvc pipeline run

```
Expand All @@ -81,9 +82,10 @@ $ xvc pipeline run
dir-0001/file-0001.bin
### Changed Files:


[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}")


```

When you change a file, it's printed in both added and removed files:
Expand All @@ -98,7 +100,8 @@ $ xvc pipeline run

### Changed Files:
dir-0001/file-0002.bin

[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}")


```
6 changes: 4 additions & 2 deletions book/src/ref/xvc-pipeline-step-dependency-glob.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,10 @@ The step is invalidated when a file described by the glob is added, removed or c
```console
$ xvc pipeline run
[OUT] [files-changed] Files have changed.

[DONE] files-changed (echo 'Files have changed.')


$ xvc pipeline run

```
Expand All @@ -60,7 +61,8 @@ $ rm dir-0001/file-0001.bin

$ xvc pipeline run
[OUT] [files-changed] Files have changed.

[DONE] files-changed (echo 'Files have changed.')


```
6 changes: 4 additions & 2 deletions book/src/ref/xvc-pipeline-step-dependency-lines.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,10 @@ $ xvc pipeline run
"Gwen", "F", 26, 64, 121
"Hank", "M", 30, 71, 158
"Ivan", "M", 53, 72, 175

[DONE] print-top-10 (head people.csv)


``````

When you run the pipeline again, the step is not run because the specified lines didn't change.
Expand Down Expand Up @@ -99,8 +100,9 @@ $ xvc pipeline run
"Gwen", "F", 26, 64, 121
"Ferzan", "M", 30, 71, 158
"Ivan", "M", 53, 72, 175
[DONE] print-top-10 (head people.csv)
```

9 changes: 6 additions & 3 deletions book/src/ref/xvc-pipeline-step-dependency-param.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,14 @@ Run for the first time, as initially all dependencies are invalid:
```console
$ xvc pipeline run
[OUT] [read-hyperparams] Update Hyperparameters

[DONE] read-hyperparams (echo "Update Hyperparameters")

[OUT] [read-database-config] Updated Database Configuration

[DONE] read-database-config (echo "Updated Database Configuration")


```

For the second time, it won't read the configuration as nothing is changed:
Expand All @@ -74,9 +76,10 @@ $ perl -pi -e 's/5432/9876/g' myparams.yaml

$ xvc pipeline run
[OUT] [read-database-config] Updated Database Configuration

[DONE] read-database-config (echo "Updated Database Configuration")


```

Note that, `read-hyperparams` is not invalidated, though the values are in the same file.
9 changes: 6 additions & 3 deletions book/src/ref/xvc-pipeline-step-dependency-regex-items.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,9 @@ $ xvc pipeline run
"Neil", "M", 36, 75, 160
"Omar", "M", 38, 70, 145
"Quin", "M", 29, 71, 176

[DONE] new-males (echo "New Males:/n ${XVC_ADDED_REGEX_ITEMS}")

[OUT] [new-females] New Females:
"Elly", "F", 30, 66, 124
"Fran", "F", 33, 66, 115
Expand All @@ -86,9 +87,10 @@ $ xvc pipeline run
"Myra", "F", 23, 62, 98
"Page", "F", 31, 67, 135
"Ruth", "F", 28, 65, 131

[DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}")


``````

When you run the pipeline again, the steps are not run because the regexes didn't change.
Expand Down Expand Up @@ -129,7 +131,8 @@ $ cat people.csv
$ xvc pipeline run
[OUT] [new-females] New Females:
"Asude", "F", 12, 55, 110

[DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}")


```
6 changes: 4 additions & 2 deletions book/src/ref/xvc-pipeline-step-dependency-regex.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,10 @@ When you run the pipeline initially, the steps are run.
```console
$ xvc pipeline run
[OUT] [count-females] 7

[DONE] count-females (grep -c '"F",' people.csv)


``````

When you run the pipeline again, the step is not run because the regex result didn't change.
Expand Down Expand Up @@ -98,7 +99,8 @@ $ cat people.csv

$ xvc pipeline run
[OUT] [count-females] 8

[DONE] count-females (grep -c '"F",' people.csv)


```
9 changes: 6 additions & 3 deletions book/src/ref/xvc-pipeline-step-dependency-step.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,14 @@ When run, the dependency will be run first and the step will be run after.
```console
$ xvc pipeline run
[OUT] [hello] hello

[DONE] hello (echo hello)

[OUT] [world] world

[DONE] world (echo world)


```

If the dependency is not run, the dependent step won't run either.
Expand All @@ -46,7 +48,8 @@ If you want to run the dependent always, you can set it to run always explicitly
$ xvc pipeline step update --step-name world --when always
$ xvc pipeline run
[OUT] [world] world

[DONE] world (echo world)


```
3 changes: 2 additions & 1 deletion book/src/ref/xvc-pipeline-step-dependency-url.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,10 @@ The step is invalidated when the page is updated.
```console
$ xvc pipeline run
[OUT] [xvc-docs-update] Xvc docs updated!

[DONE] xvc-docs-update (echo 'Xvc docs updated!')


```

The step won't run again until a new version of the page is published.
Expand Down
Loading

0 comments on commit 1fd1e98

Please sign in to comment.