Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3store: Allow customizing upload ID, object ID and bucket #1167

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
12 changes: 10 additions & 2 deletions docs/_advanced-topics/hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,9 +209,17 @@ Below you can find an annotated, JSON-ish encoded example of a hook response:
// When the filestore is used, the Path property defines where the uploaded file is saved.
// The path may be absolute or relative, and point towards a location outside of the directory
// defined using the `-dir` flag. If it's relative, the path will be resolved relative to `-dir`.
"Path": "./upload-e7a036dc-33f4-451f-9520-49032b87e952/presentation.pdf"
"Path": "./upload-e7a036dc-33f4-451f-9520-49032b87e952/presentation.pdf",

// Other storages, such as S3Store, GCSStore, and AzureStore, do not support the Storage
// When the S3 storage is used, the Key property defines the key under which the uploaded file is saved.
// If the key is not set, it defaults to using the upload ID as the key. In addition, the Bucket property
// defines the bucket where the uploaded file is saved. Note that the info and part files will still
// be stored in the bucket that the S3 storage has been configured for.
// Please consult https://tus.github.io/tusd/storage-backends/aws-s3/#custom-storage-location for more details.
"Key": "upload-e7a036dc-33f4-451f-9520-49032b87e952/presentation.pdf",
"Bucket": "customer-ABC"

// Other storages, such as GCSStore, and AzureStore, do not support the Storage
// property yet.
}
},
Expand Down
28 changes: 28 additions & 0 deletions docs/_storage-backends/aws-s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,38 @@ By default, the objects are stored at the root of the bucket. For example the ob
- `abcdef123`: File object
- `abcdef123.part`: Temporary object

For details on customizing the storage location, please read the next section below.

{: .note }

The file object is not visible in the S3 bucket before the upload is finished because the transferred file data is stored in the associated S3 multipart upload. Once the upload is complete, the chunks from the S3 multipart are reassembled into the file, creating the file object and removing the S3 multipart upload. In addition, the S3 multipart upload is not directly visible in the S3 bucket because it does not represent a complete object. Please don't be confused if you don't see the changes in the bucket while the file is being uploaded.

### Custom storage location

The locations of the three objects mentioned above can be fully customized using the [pre-create hook]({{ site.baseurl }}/advanced-topics/hooks/). The keys of the `.info` and `.part` objects are derived from the upload ID, which can be customized by the pre-create hook using the [`ChangeFileInfo.ID` setting]({{ site.baseurl }}/advanced-topics/hooks/#hook-requests-and-responses). Both objects will always be saved in the bucket configured in the `-s3-bucket` flag. Similarly, the location where the file object containing the uploaded data is saved is by default derived from the upload ID, but can be fully customized using the [`ChangeFileInfo.Storage.Key` and `ChangeFileInfo.Storage.Bucket` settings]({{ site.baseurl }}/advanced-topics/hooks/#hook-requests-and-responses).

For example, consider that the pre-create hook returns the following hook response and tusd is configured with `-s3-bucket=upload-info`:

```js
{
"ChangeFileInfo": {
"ID": "project-123/abc",
"Storage": {
"Key": "project-123/abc/presentation.pdf",
"Bucket": "customer-ABC"
}
},
}
```

Then the following objects will be created during the upload:

- An informational object at `project-123/abc.info` in the `upload-info` bucket.
- A temporary object at `project-123/abc.part` in the `upload-info` bucket (if needed).
- The file object containing the uploaded data at `project-123/abc/presentation.pdf` in the `customer-ABC` bucket.

If an object prefix with the `-s3-object-prefix` flag is configured, the prefix is prepended to the keys of all three objects.

### Metadata

If [metadata](https://tus.io/protocols/resumable-upload#upload-metadata) is associated with the upload during creation, it will be added to the file object once the upload is finished. Because the metadata on S3 objects must only contain ASCII characters, tusd will replace every non-ASCII character with a question mark. For example, "Menü" will become "Men?".
Expand Down
Loading
Loading