Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: new job to backup pins to s3 #1844

Open
wants to merge 69 commits into
base: main
Choose a base branch
from
Open

Commits on Oct 12, 2022

  1. Configuration menu
    Copy the full SHA
    ca658d5 View commit details
    Browse the repository at this point in the history
  2. chore: add aws-sdk

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    cd7a420 View commit details
    Browse the repository at this point in the history
  3. feat: add cron

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    79219bb View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2cd96c7 View commit details
    Browse the repository at this point in the history
  5. chore: added required dependencies

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    48cb64d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    55367e7 View commit details
    Browse the repository at this point in the history
  7. feat: finished job to backup files to s3

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    fdb92d3 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    c910b65 View commit details
    Browse the repository at this point in the history
  9. feat: add github action to run cron job

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    bff06de View commit details
    Browse the repository at this point in the history
  10. feat: added cron pins backup script

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    b34dd12 View commit details
    Browse the repository at this point in the history
  11. chore: removed todo

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    14cbd8c View commit details
    Browse the repository at this point in the history
  12. feat: added test mocks

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    ef2600e View commit details
    Browse the repository at this point in the history
  13. chore: add mocks for export car

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    9e6f035 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    ec58dd2 View commit details
    Browse the repository at this point in the history
  15. feat: swapped to a class tructure

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    0f0cbe5 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    d009be1 View commit details
    Browse the repository at this point in the history
  17. feat: added more logging, updated query

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    e63a4a1 View commit details
    Browse the repository at this point in the history
  18. fix: updated test mocks

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    e4e22a0 View commit details
    Browse the repository at this point in the history
  19. fix: updated to make sql valid

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    381ba65 View commit details
    Browse the repository at this point in the history
  20. Fix issues with updating rows

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    9e01bcb View commit details
    Browse the repository at this point in the history
  21. chore: remove redundant argument

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    7271673 View commit details
    Browse the repository at this point in the history
  22. chore: update variable names

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    1010da8 View commit details
    Browse the repository at this point in the history
  23. chore: update to use npm for the cron

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    5c0cc8f View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    4b34fdb View commit details
    Browse the repository at this point in the history
  25. chore: remove commented code

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    4b59c7b View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    1826006 View commit details
    Browse the repository at this point in the history
  27. chore: updated package lock

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    e01dda0 View commit details
    Browse the repository at this point in the history
  28. Make linter happier

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    3c529e5 View commit details
    Browse the repository at this point in the history
  29. feat: w3 open <cid> to open cid on w3s.link in browser (#1892)

    I find myself having to add the https://w3s.link/ipfs prefix a lot. I
    want a command to do it for me.
    
    ```bash
    w3 open bafybeidluj5ub7okodgg5v6l4x3nytpivvcouuxgzuioa6vodg3xt2uqle
    
    w3 open bafybeidluj5ub7okodgg5v6l4x3nytpivvcouuxgzuioa6vodg3xt2uqle/olizilla.png
    ```
    
    License: MIT
    Signed-off-by: Oli Evans <[email protected]>
    olizilla authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    d417922 View commit details
    Browse the repository at this point in the history
  30. feat: put write to cluster behind a flag (#1785)

    - Set `ENABLE_ADD_TO_CLUSTER` to send upload to ipfs-cluster after
    sending response to user. The current behavior is to always try to write
    user uploads to cluster. With this PR we disable that but leave a
    feature flag to restore it if we need to.
    - Adds LinkIndexer to decode and record links from each block as we
    iterate through them. It allows us to set the DAG structure as
    'Complete' where we know the upload contains all the blocks for a
    complete DAG with no dangling links.
    - Adds task to call to `LINKDEX_API` after the response is sent where
    the DAG is `Partial`, to check if it is complete across other CARs under
    the same upload.
    - Where we have a complete CAR we set the status as "Pinned" early as
    it's already in elastic-ipfs. This is debatable, as there is a lag until
    it's discoverable, but I think it's reasonable as I want to push us
    towards telling users when we safely have their stuff early.
    - We write uploads to more than one place, by adding an R2Bucket binding
    to our worker env called `carpark` where we write cars keyed by their
    car CID.
    - R2 only supports an [md5 integrity check on
    put](https://developers.cloudflare.com/r2/runtime-apis/#r2putoptions),
    so this PR re-uses the md5 lib that the aws-s3-sdk uses to generate that
    and also stores it in the r2 metadata so we see what hash was used to
    verify it later.
    - CAR CIDs are generated as [w3up-cli does
    it](https://github.com/web3-storage/w3up-cli/blob/b313c1221a0fb93a7d7cc3ced5fd1d4bce537a7a/src/utils.js#L46-L50)
    with the [`0x202`
    code](https://github.com/multiformats/multicodec/blob/955e61fe6221454c682ca9ba4901558bfc1bac96/table.csv#L130),
    and added to the /car response body.
    - miniflare is now available as a global in tests so we can make
    assertions about [r2 bucket contents in
    tests](https://miniflare.dev/storage/r2#manipulating-outside-workers),
    which is nice.
    - we make use of [miniflare's new fetch mocking
    support](https://miniflare.dev/core/standards#mocking-outbound-fetch-requests)
    to test calls to the linkdex-api happen when we expect them to.
    
    _NOTE: We've been using ipfs-cluster as a DAG completeness check-ing
    service, and we're using DAG completeness as a proxy for "we got all the
    CARs and the DAG is stored". We need to do some reframing as it's a
    feature of DAGs and CARs and IPFS that you could upload some `dab-json`
    that links to the complete wikipedia archive. We want to inform you when
    your blocks are safely stored, but we don't want to necessarily also
    pull in all of wikipedia, nor do we want to avoid telling you your
    blocks are pinned where they link to some other dataset._
    
    Fixes: #1784
    
    License: MIT
    Signed-off-by: Oli Evans <[email protected]>
    olizilla authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    7815d72 View commit details
    Browse the repository at this point in the history
  31. chore: updated package lock

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    396222f View commit details
    Browse the repository at this point in the history
  32. chore: update package lock

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    e3a13da View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    6fad50f View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    5c7bca4 View commit details
    Browse the repository at this point in the history
  35. chore: added dagula

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    f07c61b View commit details
    Browse the repository at this point in the history
  36. feat: added default to empty array

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    36e7e08 View commit details
    Browse the repository at this point in the history
  37. feat: added default on the init data

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    2d3d56a View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    86b3de6 View commit details
    Browse the repository at this point in the history
  39. chore: updated return types

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    7ceeaf9 View commit details
    Browse the repository at this point in the history
  40. chore: update package lock

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    7368e13 View commit details
    Browse the repository at this point in the history
  41. chore: remove todo comment

    joshghent authored and flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    e127dc0 View commit details
    Browse the repository at this point in the history
  42. Add fetch to cron pin-backup

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    bf8aa7a View commit details
    Browse the repository at this point in the history
  43. Update types

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    4abd92f View commit details
    Browse the repository at this point in the history
  44. Add Minio to cron

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    874830c View commit details
    Browse the repository at this point in the history
  45. Add getS3 to cron utils

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    38b96a4 View commit details
    Browse the repository at this point in the history
  46. update tests to be more e2endy

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    6aa93ef View commit details
    Browse the repository at this point in the history
  47. Comment out size limiting

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    15fc4fd View commit details
    Browse the repository at this point in the history
  48. more type fixes

    flea89 committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    9d2fe99 View commit details
    Browse the repository at this point in the history
  49. Configuration menu
    Copy the full SHA
    ae3b044 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2022

  1. tidy tests up

    flea89 committed Oct 21, 2022
    Configuration menu
    Copy the full SHA
    5bc3522 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1777df2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    dc431fc View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2022

  1. Configuration menu
    Copy the full SHA
    f621c31 View commit details
    Browse the repository at this point in the history
  2. Add abort controller

    flea89 committed Oct 24, 2022
    Configuration menu
    Copy the full SHA
    a266c32 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b7aa9db View commit details
    Browse the repository at this point in the history
  4. Align api package with main

    flea89 committed Oct 24, 2022
    Configuration menu
    Copy the full SHA
    c9e4dbb View commit details
    Browse the repository at this point in the history
  5. Improve logging and more tests

    flea89 committed Oct 24, 2022
    Configuration menu
    Copy the full SHA
    6e1f121 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d573121 View commit details
    Browse the repository at this point in the history
  7. Remove only from test

    flea89 committed Oct 24, 2022
    Configuration menu
    Copy the full SHA
    fd02b6a View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2022

  1. Update logging

    flea89 committed Oct 25, 2022
    Configuration menu
    Copy the full SHA
    395dff7 View commit details
    Browse the repository at this point in the history
  2. Update node modules

    flea89 committed Oct 25, 2022
    Configuration menu
    Copy the full SHA
    c58990b View commit details
    Browse the repository at this point in the history
  3. Update setup node action

    flea89 committed Oct 25, 2022
    Configuration menu
    Copy the full SHA
    52984d6 View commit details
    Browse the repository at this point in the history
  4. Fix dependecies

    flea89 committed Oct 25, 2022
    Configuration menu
    Copy the full SHA
    694788e View commit details
    Browse the repository at this point in the history
  5. Increase cron timeout

    flea89 committed Oct 25, 2022
    Configuration menu
    Copy the full SHA
    9dec63f View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2022

  1. Configuration menu
    Copy the full SHA
    61990b2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    749e20d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    be24923 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7a48df2 View commit details
    Browse the repository at this point in the history
  5. Update cron help text

    flea89 committed Oct 31, 2022
    Configuration menu
    Copy the full SHA
    8eae8f6 View commit details
    Browse the repository at this point in the history