-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: new job to backup pins to s3 #1844
Open
joshghent
wants to merge
69
commits into
main
Choose a base branch
from
feat/794-copy-pins-to-eips
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Commits on Oct 12, 2022
-
Configuration menu - View commit details
-
Copy full SHA for ca658d5 - Browse repository at this point
Copy the full SHA ca658d5View commit details -
Configuration menu - View commit details
-
Copy full SHA for cd7a420 - Browse repository at this point
Copy the full SHA cd7a420View commit details -
Configuration menu - View commit details
-
Copy full SHA for 79219bb - Browse repository at this point
Copy the full SHA 79219bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2cd96c7 - Browse repository at this point
Copy the full SHA 2cd96c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 48cb64d - Browse repository at this point
Copy the full SHA 48cb64dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 55367e7 - Browse repository at this point
Copy the full SHA 55367e7View commit details -
Configuration menu - View commit details
-
Copy full SHA for fdb92d3 - Browse repository at this point
Copy the full SHA fdb92d3View commit details -
Configuration menu - View commit details
-
Copy full SHA for c910b65 - Browse repository at this point
Copy the full SHA c910b65View commit details -
Configuration menu - View commit details
-
Copy full SHA for bff06de - Browse repository at this point
Copy the full SHA bff06deView commit details -
Configuration menu - View commit details
-
Copy full SHA for b34dd12 - Browse repository at this point
Copy the full SHA b34dd12View commit details -
Configuration menu - View commit details
-
Copy full SHA for 14cbd8c - Browse repository at this point
Copy the full SHA 14cbd8cView commit details -
Configuration menu - View commit details
-
Copy full SHA for ef2600e - Browse repository at this point
Copy the full SHA ef2600eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e6f035 - Browse repository at this point
Copy the full SHA 9e6f035View commit details -
Configuration menu - View commit details
-
Copy full SHA for ec58dd2 - Browse repository at this point
Copy the full SHA ec58dd2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f0cbe5 - Browse repository at this point
Copy the full SHA 0f0cbe5View commit details -
Configuration menu - View commit details
-
Copy full SHA for d009be1 - Browse repository at this point
Copy the full SHA d009be1View commit details -
Configuration menu - View commit details
-
Copy full SHA for e63a4a1 - Browse repository at this point
Copy the full SHA e63a4a1View commit details -
Configuration menu - View commit details
-
Copy full SHA for e4e22a0 - Browse repository at this point
Copy the full SHA e4e22a0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 381ba65 - Browse repository at this point
Copy the full SHA 381ba65View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e01bcb - Browse repository at this point
Copy the full SHA 9e01bcbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7271673 - Browse repository at this point
Copy the full SHA 7271673View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1010da8 - Browse repository at this point
Copy the full SHA 1010da8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c0cc8f - Browse repository at this point
Copy the full SHA 5c0cc8fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4b34fdb - Browse repository at this point
Copy the full SHA 4b34fdbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4b59c7b - Browse repository at this point
Copy the full SHA 4b59c7bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1826006 - Browse repository at this point
Copy the full SHA 1826006View commit details -
Configuration menu - View commit details
-
Copy full SHA for e01dda0 - Browse repository at this point
Copy the full SHA e01dda0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3c529e5 - Browse repository at this point
Copy the full SHA 3c529e5View commit details -
feat:
w3 open <cid>
to open cid on w3s.link in browser (#1892)I find myself having to add the https://w3s.link/ipfs prefix a lot. I want a command to do it for me. ```bash w3 open bafybeidluj5ub7okodgg5v6l4x3nytpivvcouuxgzuioa6vodg3xt2uqle w3 open bafybeidluj5ub7okodgg5v6l4x3nytpivvcouuxgzuioa6vodg3xt2uqle/olizilla.png ``` License: MIT Signed-off-by: Oli Evans <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for d417922 - Browse repository at this point
Copy the full SHA d417922View commit details -
feat: put write to cluster behind a flag (#1785)
- Set `ENABLE_ADD_TO_CLUSTER` to send upload to ipfs-cluster after sending response to user. The current behavior is to always try to write user uploads to cluster. With this PR we disable that but leave a feature flag to restore it if we need to. - Adds LinkIndexer to decode and record links from each block as we iterate through them. It allows us to set the DAG structure as 'Complete' where we know the upload contains all the blocks for a complete DAG with no dangling links. - Adds task to call to `LINKDEX_API` after the response is sent where the DAG is `Partial`, to check if it is complete across other CARs under the same upload. - Where we have a complete CAR we set the status as "Pinned" early as it's already in elastic-ipfs. This is debatable, as there is a lag until it's discoverable, but I think it's reasonable as I want to push us towards telling users when we safely have their stuff early. - We write uploads to more than one place, by adding an R2Bucket binding to our worker env called `carpark` where we write cars keyed by their car CID. - R2 only supports an [md5 integrity check on put](https://developers.cloudflare.com/r2/runtime-apis/#r2putoptions), so this PR re-uses the md5 lib that the aws-s3-sdk uses to generate that and also stores it in the r2 metadata so we see what hash was used to verify it later. - CAR CIDs are generated as [w3up-cli does it](https://github.com/web3-storage/w3up-cli/blob/b313c1221a0fb93a7d7cc3ced5fd1d4bce537a7a/src/utils.js#L46-L50) with the [`0x202` code](https://github.com/multiformats/multicodec/blob/955e61fe6221454c682ca9ba4901558bfc1bac96/table.csv#L130), and added to the /car response body. - miniflare is now available as a global in tests so we can make assertions about [r2 bucket contents in tests](https://miniflare.dev/storage/r2#manipulating-outside-workers), which is nice. - we make use of [miniflare's new fetch mocking support](https://miniflare.dev/core/standards#mocking-outbound-fetch-requests) to test calls to the linkdex-api happen when we expect them to. _NOTE: We've been using ipfs-cluster as a DAG completeness check-ing service, and we're using DAG completeness as a proxy for "we got all the CARs and the DAG is stored". We need to do some reframing as it's a feature of DAGs and CARs and IPFS that you could upload some `dab-json` that links to the complete wikipedia archive. We want to inform you when your blocks are safely stored, but we don't want to necessarily also pull in all of wikipedia, nor do we want to avoid telling you your blocks are pinned where they link to some other dataset._ Fixes: #1784 License: MIT Signed-off-by: Oli Evans <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 7815d72 - Browse repository at this point
Copy the full SHA 7815d72View commit details -
Configuration menu - View commit details
-
Copy full SHA for 396222f - Browse repository at this point
Copy the full SHA 396222fView commit details -
Configuration menu - View commit details
-
Copy full SHA for e3a13da - Browse repository at this point
Copy the full SHA e3a13daView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6fad50f - Browse repository at this point
Copy the full SHA 6fad50fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c7bca4 - Browse repository at this point
Copy the full SHA 5c7bca4View commit details -
Configuration menu - View commit details
-
Copy full SHA for f07c61b - Browse repository at this point
Copy the full SHA f07c61bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 36e7e08 - Browse repository at this point
Copy the full SHA 36e7e08View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2d3d56a - Browse repository at this point
Copy the full SHA 2d3d56aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 86b3de6 - Browse repository at this point
Copy the full SHA 86b3de6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7ceeaf9 - Browse repository at this point
Copy the full SHA 7ceeaf9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7368e13 - Browse repository at this point
Copy the full SHA 7368e13View commit details -
Configuration menu - View commit details
-
Copy full SHA for e127dc0 - Browse repository at this point
Copy the full SHA e127dc0View commit details -
Configuration menu - View commit details
-
Copy full SHA for bf8aa7a - Browse repository at this point
Copy the full SHA bf8aa7aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4abd92f - Browse repository at this point
Copy the full SHA 4abd92fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 874830c - Browse repository at this point
Copy the full SHA 874830cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 38b96a4 - Browse repository at this point
Copy the full SHA 38b96a4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6aa93ef - Browse repository at this point
Copy the full SHA 6aa93efView commit details -
Configuration menu - View commit details
-
Copy full SHA for 15fc4fd - Browse repository at this point
Copy the full SHA 15fc4fdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9d2fe99 - Browse repository at this point
Copy the full SHA 9d2fe99View commit details -
Configuration menu - View commit details
-
Copy full SHA for ae3b044 - Browse repository at this point
Copy the full SHA ae3b044View commit details
Commits on Oct 21, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 5bc3522 - Browse repository at this point
Copy the full SHA 5bc3522View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1777df2 - Browse repository at this point
Copy the full SHA 1777df2View commit details -
Configuration menu - View commit details
-
Copy full SHA for dc431fc - Browse repository at this point
Copy the full SHA dc431fcView commit details
Commits on Oct 24, 2022
-
Configuration menu - View commit details
-
Copy full SHA for f621c31 - Browse repository at this point
Copy the full SHA f621c31View commit details -
Configuration menu - View commit details
-
Copy full SHA for a266c32 - Browse repository at this point
Copy the full SHA a266c32View commit details -
Configuration menu - View commit details
-
Copy full SHA for b7aa9db - Browse repository at this point
Copy the full SHA b7aa9dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for c9e4dbb - Browse repository at this point
Copy the full SHA c9e4dbbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6e1f121 - Browse repository at this point
Copy the full SHA 6e1f121View commit details -
Configuration menu - View commit details
-
Copy full SHA for d573121 - Browse repository at this point
Copy the full SHA d573121View commit details -
Configuration menu - View commit details
-
Copy full SHA for fd02b6a - Browse repository at this point
Copy the full SHA fd02b6aView commit details
Commits on Oct 25, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 395dff7 - Browse repository at this point
Copy the full SHA 395dff7View commit details -
Configuration menu - View commit details
-
Copy full SHA for c58990b - Browse repository at this point
Copy the full SHA c58990bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 52984d6 - Browse repository at this point
Copy the full SHA 52984d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 694788e - Browse repository at this point
Copy the full SHA 694788eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9dec63f - Browse repository at this point
Copy the full SHA 9dec63fView commit details
Commits on Oct 31, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 61990b2 - Browse repository at this point
Copy the full SHA 61990b2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 749e20d - Browse repository at this point
Copy the full SHA 749e20dView commit details -
Configuration menu - View commit details
-
Copy full SHA for be24923 - Browse repository at this point
Copy the full SHA be24923View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7a48df2 - Browse repository at this point
Copy the full SHA 7a48df2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8eae8f6 - Browse repository at this point
Copy the full SHA 8eae8f6View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.