-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
81 changed files
with
764 additions
and
294 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,3 +15,4 @@ Cargo.lock | |
| # Use plain vim for large files and let it create swp | ||
| *.swp | ||
| flamegraph.svg | ||
| workflow_tests/docs | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| # xvc pipeline step remove | ||
|
|
||
| ## Purpose | ||
|
|
||
| Remove a step and all its dependencies and outputs from the pipeline. | ||
|
|
||
| ## Synopsis | ||
|
|
||
| ```console | ||
| $ xvc pipeline step remove --help | ||
| Remove a step from a pipeline | ||
|
|
||
| Usage: xvc pipeline step remove --step-name <STEP_NAME> | ||
|
|
||
| Options: | ||
| -s, --step-name <STEP_NAME> Name of the step to remove | ||
| -h, --help Print help | ||
|
|
||
| ``` | ||
|
|
||
| ## Examples | ||
|
|
||
| This command works only in Xvc repositories. | ||
|
|
||
| ```console | ||
| $ git init | ||
| ... | ||
| $ xvc init | ||
| ``` | ||
|
|
||
| Let's create a few steps and make them depend on each other. | ||
| ```console | ||
| $ xvc pipeline step new --step-name hello --command 'echo hello >> hello.txt' | ||
|
|
||
| $ xvc pipeline step new --step-name world --command 'echo world >> world.txt' | ||
|
|
||
| $ xvc pipeline step new --step-name from --command 'echo from >> from.txt' | ||
|
|
||
| $ xvc pipeline step new --step-name xvc --command 'echo xvc >> xvc.txt' | ||
|
|
||
| ``` | ||
| Let's specify the outputs as well. | ||
| ```console | ||
| $ xvc pipeline step output --step-name hello --output-file hello.txt | ||
|
|
||
| $ xvc pipeline step output --step-name world --output-file world.txt | ||
|
|
||
| $ xvc pipeline step output --step-name from --output-file from.txt | ||
|
|
||
| $ xvc pipeline step output --step-name xvc --output-file xvc.txt | ||
|
|
||
| ``` | ||
|
|
||
| Now we can add dependencies between them. | ||
| ```console | ||
| $ xvc pipeline step dependency --step-name xvc --step from | ||
|
|
||
| $ xvc pipeline step dependency --step-name from --file world.txt | ||
|
|
||
| $ xvc pipeline step dependency --step-name world --step hello | ||
|
|
||
| ``` | ||
|
|
||
|
|
||
| Now the pipeline looks like this: | ||
| ```console | ||
| $ xvc pipeline step list | ||
| hello: echo hello >> hello.txt (by_dependencies) | ||
| world: echo world >> world.txt (by_dependencies) | ||
| from: echo from >> from.txt (by_dependencies) | ||
| xvc: echo xvc >> xvc.txt (by_dependencies) | ||
|
|
||
| $ xvc pipeline dag --format mermaid | ||
| flowchart TD | ||
| n0["hello"] | ||
| n1["hello.txt"] --> n0 | ||
| n2["world"] | ||
| n0["hello"] --> n2 | ||
| n3["world.txt"] --> n2 | ||
| n4["from"] | ||
| n3["world.txt"] --> n4 | ||
| n5["from.txt"] --> n4 | ||
| n6["xvc"] | ||
| n4["from"] --> n6 | ||
| n7["xvc.txt"] --> n6 | ||
|
|
||
|
|
||
| ``` | ||
|
|
||
| When we remove a step, all its dependencies and outputs are removed as well. | ||
| ```console | ||
| $ xvc -vv pipeline step remove --step-name from | ||
| [INFO] Removing dep: file(world.txt) | ||
| [INFO] Removing dep step(from) from xvc | ||
| [INFO] Removing output: File | ||
| [INFO] Removing step: from | ||
|
|
||
| ``` | ||
|
|
||
| ```console | ||
| $ xvc pipeline step list | ||
| hello: echo hello >> hello.txt (by_dependencies) | ||
| world: echo world >> world.txt (by_dependencies) | ||
| xvc: echo xvc >> xvc.txt (by_dependencies) | ||
|
|
||
| $ xvc pipeline dag --format mermaid | ||
| flowchart TD | ||
| n0["hello"] | ||
| n1["hello.txt"] --> n0 | ||
| n2["world"] | ||
| n0["hello"] --> n2 | ||
| n3["world.txt"] --> n2 | ||
| n4["xvc"] | ||
| n5["xvc.txt"] --> n4 | ||
|
|
||
|
|
||
| ``` | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| # xvc storage new s3 | ||
|
|
||
| ## Purpose | ||
|
|
||
| Configure an S3 (or a compatible) service as an Xvc storage. | ||
|
|
||
| ## Synopsis | ||
|
|
||
| ```console | ||
| $ xvc storage new rsync --help | ||
| Add a new rsync storage | ||
|
|
||
| Uses rsync in separate processes to communicate with the server. | ||
|
|
||
| This can be used when you already have an SSH/Rsync connection. | ||
|
|
||
| It doesn't prompt for any passwords. The connection must be set up with ssh keys beforehand. | ||
|
|
||
| Usage: xvc storage new rsync [OPTIONS] --name <NAME> --host <HOST> --storage-dir <STORAGE_DIR> | ||
|
|
||
| Options: | ||
| -n, --name <NAME> | ||
| Name of the storage. | ||
| Recommended to keep this name unique to refer easily. | ||
|
|
||
| --host <HOST> | ||
| Hostname for the connection in the form host.example.com (without @, : or protocol) | ||
|
|
||
| --port <PORT> | ||
| Port number for the connection in the form 22. Doesn't add port number to connection string if not given | ||
|
|
||
| --user <USER> | ||
| User name for the connection, the part before @ in [email protected] (without @, hostname). User name isn't included in connection strings if not given | ||
|
|
||
| --storage-dir <STORAGE_DIR> | ||
| storage directory in the host to store the files | ||
|
|
||
| -h, --help | ||
| Print help (see a summary with '-h') | ||
|
|
||
| ``` | ||
|
|
||
| ## Examples | ||
|
|
||
| You must setup an SSH connection | ||
|
|
||
| The command works only in Xvc repositories. | ||
|
|
||
| ```console | ||
| $ git init | ||
| ... | ||
| $ xvc init | ||
|
|
||
| $ xvc-test-helper create-directory-tree --directories 1 --files 3 --seed 20230211 | ||
|
|
||
| $ tree dir-0001 | ||
| dir-0001 | ||
| ├── file-0001.bin | ||
| ├── file-0002.bin | ||
| └── file-0003.bin | ||
|
|
||
| 1 directory, 3 files | ||
|
|
||
| ``` | ||
|
|
||
| Xvc only sends and receives tracked files. | ||
|
|
||
| ```console | ||
| $ xvc file track dir-0001 | ||
| ``` | ||
|
|
||
| You can define a storage bucket as storage and begin to use it. | ||
|
|
||
| ```console | ||
| $ xvc storage new rsync --name backup --host one.emresult.com --user iex --storage-dir /tmp/xvc-backup/ | ||
|
|
||
| ``` | ||
|
|
||
| Send files to this storage. | ||
|
|
||
| ```console | ||
| $ xvc file send dir-0001 --to backup | ||
|
|
||
| ``` | ||
|
|
||
| You can remove the files you sent from your cache and workspace. | ||
|
|
||
| ```console | ||
| $ xvc file remove --from-cache dir-0001/ | ||
| [DELETE] [CWD]/.xvc/b3/3c6/70f/e91055c2be2e87890dba1e952d656d1e70dd196bf5530d379243c6e4aa/0.bin | ||
| [DELETE] [CWD]/.xvc/b3/3c6/70f/e91055c2be2e87890dba1e952d656d1e70dd196bf5530d379243c6e4aa | ||
| [DELETE] [CWD]/.xvc/b3/3c6/70f | ||
| [DELETE] [CWD]/.xvc/b3/3c6 | ||
| [DELETE] [CWD]/.xvc/b3/7aa/354/0225bd33702c239454b63b31d1ea25721cbbfb491d6139d0b85b82d15d/0.bin | ||
| [DELETE] [CWD]/.xvc/b3/7aa/354/0225bd33702c239454b63b31d1ea25721cbbfb491d6139d0b85b82d15d | ||
| [DELETE] [CWD]/.xvc/b3/7aa/354 | ||
| [DELETE] [CWD]/.xvc/b3/7aa | ||
| [DELETE] [CWD]/.xvc/b3/d7d/629/677c6d8df55ab3a1d694453c59f3ca0df494d3dc190aeef1e00abd96eb/0.bin | ||
| [DELETE] [CWD]/.xvc/b3/d7d/629/677c6d8df55ab3a1d694453c59f3ca0df494d3dc190aeef1e00abd96eb | ||
| [DELETE] [CWD]/.xvc/b3/d7d/629 | ||
| [DELETE] [CWD]/.xvc/b3/d7d | ||
| [DELETE] [CWD]/.xvc/b3 | ||
|
|
||
| $ rm -rf dir-0001/ | ||
| ``` | ||
|
|
||
| Then get back them from the storage. | ||
|
|
||
| ```console | ||
| $ xvc file bring --from backup dir-0001 | ||
|
|
||
| $ tree dir-0001 | ||
| dir-0001 | ||
| ├── file-0001.bin | ||
| ├── file-0002.bin | ||
| └── file-0003.bin | ||
|
|
||
| 1 directory, 3 files | ||
|
|
||
| ``` | ||
|
|
||
| If you want to remove a file and all of its versions from a storage, you can use `xvc file remove` command. | ||
|
|
||
| ```console | ||
| $ xvc file remove --from-storage backup dir-0001/ | ||
|
|
||
| ``` |
Oops, something went wrong.