Merge pull request #196 from near/main

Prod Release

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/epic-template-.md

@@ -0,0 +1,30 @@
+---
+name: 'Epic Template '
+about: 'Epics are milestones or groups of alike issues '
+title: "\U0001F537 [Epic] New Epic "
+labels: ''
+assignees: ''
+
+---
+
+### Description
+(Overview of milestone or function governed by this epic)
+### Success Criteria
+(Evaluate how this epic could be considered as complete and success)
+### Resources
+(Relevant documentation, Figma links, and other reference material)
+Item 1
+Item 2
+Item 3
+```[tasklist]
+### Child Issues
+[ ] https://github.com/near/github-project-test/issues/1
+[ ] https://github.com/near/github-project-test/issues/2
+[ ] https://github.com/near/github-project-test/issues/3
+```
+```[tasklist]
+### dependencies/blocked
+[ ] https://github.com/near/github-project-test/issues/1
+[ ] https://github.com/near/github-project-test/issues/2
+[ ] https://github.com/near/github-project-test/issues/3
+```

.github/ISSUE_TEMPLATE/feature-request-.md

@@ -0,0 +1,20 @@
+---
+name: 'Feature Request '
+about: Suggest an idea for this project. If this doesn't look right
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/secondary-focus-area-.md

@@ -0,0 +1,23 @@
+---
+name: 'Secondary Focus Area '
+about: This issue serves to help us propose and organize support for impactful work,
+  as a secondary priority to epics & planned roadmap items. If this doesn't look right
+title: "\U0001F525  [Secondary Focus Area] <Add a brief Title>"
+labels: ''
+assignees: ''
+
+---
+
+**Motivation**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
+
+**Open questions**

.github/workflows/deploy-dev-widgets.yml

@@ -11,5 +11,6 @@ jobs:
       directory-paths: ${{vars.WIDGETS_DIRECTORY_PATHS}}
       deploy-account-address: ${{ vars.DEV_SIGNER_ACCOUNT_ID }}
       signer-public-key: ${{ vars.DEV_SIGNER_PUBLIC_KEY }}
+      environment: dev
     secrets:
       SIGNER_PRIVATE_KEY: ${{ secrets.DEV_SIGNER_PRIVATE_KEY }}

.github/workflows/deploy-prod-widgets.yml

@@ -11,5 +11,6 @@ jobs:
       directory-paths: ${{vars.WIDGETS_DIRECTORY_PATHS}}
       deploy-account-address: ${{ vars.PROD_SIGNER_ACCOUNT_ID }}
       signer-public-key: ${{ vars.PROD_SIGNER_PUBLIC_KEY }}
+      environment: mainnet
     secrets:
       SIGNER_PRIVATE_KEY: ${{ secrets.PROD_SIGNER_PRIVATE_KEY }}

.github/workflows/deploy-widgets.yml

@@ -1,4 +1,4 @@
-name: Deploy Components to Mainnet
+name: Deploy Components
 on:
   workflow_call:
     inputs:
@@ -19,24 +19,43 @@ on:
         required: true
         description: "Comma-separated paths to the directories that contain the code to be deployed"
         type: string
+      environment:
+        required: true
+        description: "mainnet or dev"
+        type: string
     secrets:
       SIGNER_PRIVATE_KEY:
         description: "Private key in `ed25519:<private_key>` format for signing transaction"
         required: true
 jobs:
   deploy-widgets:
     runs-on: ubuntu-latest
-    name: Deploy widgets to social.near (mainnet)
+    name: Deploy widgets to social.near
     env:
       BOS_DEPLOY_ACCOUNT_ID: ${{ inputs.deploy-account-address }}
       BOS_SIGNER_PUBLIC_KEY: ${{ inputs.signer-public-key }}
-      BOS_SIGNER_PRIVATE_KEY: ${{ secrets.SIGNER_PRIVATE_KEY }}
       DIRECTORY_PATHS: ${{ inputs.directory-paths }}
+      ENVIRONMENT: ${{inputs.environment}}
+      BOS_SIGNER_PRIVATE_KEY: ${{ secrets.SIGNER_PRIVATE_KEY }}
 
     steps:
       - name: Checkout repository
         uses: actions/checkout@v2
 
+      - name: Set replacements
+        id: set_replacements
+        run: |
+          cd "frontend/widgets/"
+          echo "replacements=$(jq -r '[to_entries[] | .["find"] = "${" + .key + "}" | .["replace"] = .value | del(.key, .value)]' ../replacement.${ENVIRONMENT}.json | tr -d "\n\r")" >> $GITHUB_OUTPUT
+
+      - name: Replace placeholders
+        uses: flcdrg/replace-multiple-action@v1
+        with:
+          files: '**/*.jsx'
+          find: '${{ steps.set_replacements.outputs.replacements }}'
+          prefix: '(^|.*)'
+          suffix: '($|.*)'
+
       - name: Install near-social CLI
         run: |
           curl --proto '=https' --tlsv1.2 -LsSf https://github.com/FroVolod/bos-cli-rs/releases/download/v${{ inputs.cli-version }}/bos-cli-v${{ inputs.cli-version }}-installer.sh | sh

.github/workflows/runner-ci.yml

@@ -0,0 +1,63 @@
+name: Runner CI
+
+on:
+  pull_request:
+    paths:
+      - "runner/**"
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Setup Node
+        uses: actions/setup-node@v3
+        with:
+          node-version: '18.x'
+
+      - name: Install Node Dependencies
+        run: npm install
+        working-directory: ./runner
+
+      - name: Test
+        run: npm test
+        working-directory: ./runner
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Setup Node
+        uses: actions/setup-node@v3
+        with:
+          node-version: '18.x'
+
+      - name: Install Node Dependencies
+        run: npm install
+        working-directory: ./runner
+
+      - name: Lint
+        run: npm run lint
+        working-directory: ./runner
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Setup Node
+        uses: actions/setup-node@v3
+        with:
+          node-version: '18.x'
+
+      - name: Install Node Dependencies
+        run: npm install
+        working-directory: ./runner
+
+      - name: Build
+        run: npm run build
+        working-directory: ./runner

.gitignore

@@ -3,3 +3,4 @@
 redis/
 *.log
 /indexer/blocks/
+node_modules/

README.md

@@ -5,14 +5,16 @@ With QueryApi you can
 * Specify the schema for your own custom hosted database and write to it with your indexer function;
 * Retrieve that data through a GraphQL API.
 
-# Table of Contents / Applications
+## 🧩 Components
 1. [QueryApi Coordinator](./indexer)
 An Indexer that tracks changes to the QueryApi registry contract. It triggers the execution of those IndexerFunctions
 when they match new blocks by placing messages on an SQS queue. Spawns historical processing threads when needed.
    1.a.  Subfolders provide crates for the different components of the Indexer: indexer_rule_type (shared with registry contract), 
 indexer_rules_engine, storage.
 2. [Indexer Runner](.indexer-js-queue-handler)
    Retrieves messages from the SQS queue, fetches the matching block and executes the IndexerFunction.
+3. [Runner](.runner)
+   Retrieves messages from Redis Stream, fetching matching block and executes the IndexerFunction.
 3. [IndexerFunction Editor UI](./frontend)
    Serves the editor UI within the dashboard widget and mediates some communication with the GraphQL DB and block server.
 4. [Hasura Authentication Service](./hasura-authentication-service)
@@ -21,3 +23,55 @@ indexer_rules_engine, storage.
    Stores IndexerFunctions, their schemas and execution parameters like start block height.
 6. [Lake Block server](./block-server)
    Serves blocks from the S3 lake for in browser testing of IndexerFunctions.
+
+## 🚀 Getting Started
+
+The majority of the QueryApi components can be set up locally using Docker. For this purpose, a [Docker Compose file](./docker-compose.yml) has been provided. However, the local system still relies on the NEAR Mainnet, rather than running on a localnet.
+
+### Requirements
+- [Docker](https://docs.docker.com/engine/install/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
+- [Hasura CLI](https://hasura.io/docs/latest/hasura-cli/install-hasura-cli/)
+- AWS Access Keys
+
+### AWS Credentials Setup
+QueryApi requires AWS credentials to stream blocks from [NEAR Lake](https://github.com/near/near-lake-indexer). Credentials are exposed via the following environment variables, which can be found in the Docker Compose file:
+
+Runner:
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+
+Coordinator:
+- `LAKE_AWS_ACCESS_KEY`
+- `LAKE_AWS_SECRET_ACCESS_KEY`
+- `QUEUE_AWS_ACCESS_KEY`
+- `QUEUE_AWS_SECRET_ACCESS_KEY`
+
+These should be populated with your credentials. In most cases, the same key pair can be used for all 3 sets of credentials. Just ensure the keys have permissions to access S3 for handling [Requestor Pays](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) in Near Lake. 
+
+### Hasura Configuration
+Hasura contains shared tables for e.g. logging and setting arbitrary state. These tables must be configured prior to running the entire QueryApi application. Configuration is stored in the `hasura/` directory and deployed through the Hasura CLI.
+
+To configure Hasura, first start it with:
+```sh
+docker compose up hasura-graphql --detach
+```
+
+And apply the configuration with:
+```sh
+cd ./hasura && hasura deploy
+```
+
+### Running QueryApi
+With everything configured correctly, we can now start all components of QueryApi with:
+```sh
+docker compose up
+```
+
+### Local Configuration
+- Coordinator watches the dev registry contract by default (`dev-queryapi.dataplatform.near`). To use a different contract, you can update the `REGISTRY_CONTRACT_ID` environment variable.
+- Coodinator will log SQS messages rather than sending them. To use an actual Queue, you can update the `QUEUE_URL` and `START_FROM_BLOCK_QUEUE_URL` environment variables.
+
+### Known Issues
+
+It is expected to see some provisioning errors from `Runner` when starting QueryAPI for the first time. These occur when multiple indexers under the same account attempt to provision the same shared infrastructure. These should self resolve after a few seconds.