Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor CI to return better results faster #406

Merged
merged 17 commits into from
Nov 3, 2023
Merged

Refactor CI to return better results faster #406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
12e35fa
Refactor CI to return better results faster
bifurcation Oct 27, 2023
1770db5
Address review comments from @glhewett
bifurcation Nov 3, 2023
82e2728
Update the Makefile to match manifest location changes
bifurcation Nov 3, 2023
c989c15
Attempt to resolve CI YAML errors
bifurcation Nov 3, 2023
db4ab45
Attempt to resolve CI YAML errors
bifurcation Nov 3, 2023
927182a
Revert "Attempt to resolve CI YAML errors"
bifurcation Nov 3, 2023
181cdd4
Revert "Attempt to resolve CI YAML errors"
bifurcation Nov 3, 2023
180b834
Revert "Address review comments from @glhewett"
bifurcation Nov 3, 2023
e837e49
Remove <iostream> from test/bytes.cpp
bifurcation Nov 3, 2023
aee9d63
Manually expand test target
bifurcation Nov 3, 2023
75b1455
Remove 'if' lines (will break CI)
bifurcation Nov 3, 2023
44ce49a
Re-add if with single quotes
bifurcation Nov 3, 2023
0222b0d
Rename composite action
bifurcation Nov 3, 2023
af62b68
Pass more input to the composite action
bifurcation Nov 3, 2023
8f7d95f
Fix the filename of the vcpkg manifest
bifurcation Nov 3, 2023
73a3bf3
Key on library name rather than manifest hash
bifurcation Nov 3, 2023
5c6fc43
Add status line in CMakeLists.txt
bifurcation Nov 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .github/actions/prepare-build/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Install build prerequisites

inputs:
os:
description: The operating system on which the test is being run
required: true
crypto:
description: The crypto library being used
required: true
cache-dir:
description: Where to put vcpkg cache
required: true

runs:
using: "composite"
steps:
- name: Capture vcpkg revision for use in cache key
shell: bash
run: |
git -C vcpkg rev-parse HEAD > vcpkg_commit.txt

- name: Restore cache
uses: actions/cache@v3
with:
path: ${{ inputs.cache-dir }}
key: v01-vcpkg-${{ inputs.os }}-${{ inputs.crypto }}-${{ hashFiles('vcpkg_commit.txt', 'alternatives/*/vcpkg.json') }}
restore-keys: |
v01-vcpkg-${{ inputs.os }}

- name: Install dependencies (macOS)
if: ${{ runner.os == 'macOS' }}
shell: bash
run: |
brew install llvm pkg-config nasm
ln -s "/usr/local/opt/llvm/bin/clang-format" "/usr/local/bin/clang-format"
ln -s "/usr/local/opt/llvm/bin/clang-tidy" "/usr/local/bin/clang-tidy"

- name: Install dependencies (Ubuntu)
if: ${{ runner.os == 'Linux' }}
shell: bash
run: |
sudo apt-get install -y linux-headers-$(uname -r) nasm
257 changes: 73 additions & 184 deletions .github/workflows/main_ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,8 @@ on:
env:
CMAKE_BUILD_PARALLEL_LEVEL: 3
CTEST_OUTPUT_ON_FAILURE: 1
CMAKE_BUILD_DIR: ${{ github.workspace }}/build
CMAKE_BUILD_OPENSSL3_DIR: ${{ github.workspace }}/build_openssl3
CMAKE_BUILD_BORINGSSL_DIR: ${{ github.workspace }}/build_boringssl
VCPKG_BINARY_SOURCES: files,${{ github.workspace }}/build/cache,readwrite
VCPKG_TOOLCHAIN_FILE: ${{ github.workspace }}/vcpkg/scripts/buildsystems/vcpkg.cmake
VCPKG_REPO: ${{ github.workspace }}/vcpkg
CACHE_VERSION: v01
CACHE_NAME: vcpkg
VCPKG_BINARY_SOURCES: files,${{ github.workspace }}/vcpkg_cache,readwrite
CMAKE_TOOLCHAIN_FILE: ${{ github.workspace }}/vcpkg/scripts/buildsystems/vcpkg.cmake

jobs:
formatting-check:
Expand All @@ -34,59 +28,79 @@ jobs:
include-regex: '^\./(src|include|test|cmd)/.*\.(cpp|h)$'
fallback-style: 'Mozilla'

quick-linux-interop-check:
build-and-unit-test:
needs: formatting-check
name: Quick Linux Check and Interop
runs-on: ubuntu-latest
name: Build and test
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [windows-latest, ubuntu-latest, macos-latest]
crypto: [openssl_1.1, openssl_3, boringssl]
bifurcation marked this conversation as resolved.
Show resolved Hide resolved

env:
BUILD_DIR: "${RUNNER_TEMP}/build_${{ matrix.crypto }}"
CRYPTO_DIR: "./alternatives/${{ matrix.crypto }}"
bifurcation marked this conversation as resolved.
Show resolved Hide resolved

steps:
- name: Checkout repository and submodules
uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
submodules: recursive
fetch-depth: 0

# write the commit hash of vcpkg to a text file so we can use it in the
# hashFiles for cache
- run: |
git -C ${{ env.VCPKG_REPO }} rev-parse HEAD > vcpkg_commit.txt

# First, attempt to pull key key, if that is not present, pull one of the
# restore-keys so we do not need to build from scratch.
# CACHE_VERSION - provide a way to reset cache
# CACHE_NAME - name of the cache in order to manage it
# matrix.os - cache per OS and version
# hashFiles - Recache if the vcpkg files change
- name: Restore Cache
uses: actions/cache@v3
- uses: ./.github/actions/prepare-build
with:
path: ${{ github.workspace }}/build/cache
key: ${{ env.CACHE_VERSION }}-${{ env.CACHE_NAME }}-ubuntu-latest-${{ hashFiles('vcpkg_commit.txt', 'vcpkg.json', 'alternatives/openssl_3/vcpkg.json') }}
restore-keys: |
${{ env.CACHE_VERSION }}-${{ env.CACHE_NAME }}-ubuntu-latest
os: ${{ matrix.os }}
crypto: ${{ matrix.crypto }}
cache-dir: ${{ github.workspace }}/vcpkg_cache

- name: Dependencies
- name: Build
run: |
sudo apt-get install -y linux-headers-$(uname -r) nasm
# XXX(RLB): If we do not have SANITIZERS=ON here, the Windows CI builds
# hang in the middle of unit testing.
cmake -B "${{ env.BUILD_DIR }}" -DVCPKG_MANIFEST_DIR="${{ env.CRYPTO_DIR }}" -DTESTING=ON -DSANITIZERS=ON
cmake --build "${{ env.BUILD_DIR }}"

- name: Restore cache
uses: actions/cache@v3
with:
path: ${{ github.workspace }}/build/cache
key: VCPKG-BinaryCache-${{ runner.os }}
- name: Unit Test (non-Windows)
if: matrix.os != 'windows-latest'
run: |
cmake --build "${{ env.BUILD_DIR }}" --target test

- name: Build (OpenSSL 1.1)
- name: Unit Test (Windows)
if: matrix.os == 'windows-latest'
run: |
cmake -B "${{ env.CMAKE_BUILD_DIR }}" -DTESTING=ON -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake --build "${{ env.CMAKE_BUILD_DIR }}" --target all
cmake --build "${{ env.BUILD_DIR }}" --target RUN_TESTS

interop-test:
if: github.event.pull_request.draft == false
needs: build-and-unit-test
name: Interop test
runs-on: ubuntu-latest

env:
BUILD_DIR: "${RUNNER_TEMP}/build_openssl_1.1"
CRYPTO_DIR: "./alternatives/openssl_1.1"

- name: Unit Test (OpenSSL 1.1)
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
fetch-depth: 0

- uses: ./.github/actions/prepare-build
with:
os: ubuntu-latest
crypto-dir: openssl_1.1
cache-dir: ${{ github.workspace }}/vcpkg_cache

- name: Build
run: |
cmake --build "${{ env.CMAKE_BUILD_DIR }}" --target test
cmake -B "${{ env.BUILD_DIR }}" -DVCPKG_MANIFEST_DIR="${{ env.CRYPTO_DIR }}"
cmake --build "${{ env.BUILD_DIR }}"

- name: Build (Interop Harness)
run: |
cd cmd/interop
cmake -B build -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake -B build
cmake --build build

- name: Test self-interop
Expand All @@ -101,159 +115,34 @@ jobs:
run: |
cd cmd/interop
./grpc-self-test.sh

- name: Build (OpenSSL 3)
run: |
cmake -B "${{ env.CMAKE_BUILD_OPENSSL3_DIR }}" -DTESTING=ON -DVCPKG_MANIFEST_DIR="alternatives/openssl_3" -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake --build "${{ env.CMAKE_BUILD_OPENSSL3_DIR }}"

- name: Unit Test (OpenSSL 3)
run: |
cmake --build "${{ env.CMAKE_BUILD_OPENSSL3_DIR }}" --target test

- name: Build (BoringSSL)
run: |
cmake -B "${{ env.CMAKE_BUILD_BORINGSSL_DIR }}" -DTESTING=ON -DVCPKG_MANIFEST_DIR="alternatives/boringssl_1.1" -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake --build "${{ env.CMAKE_BUILD_BORINGSSL_DIR }}"

- name: Unit Test (BoringSSL)
run: |
cmake --build "${{ env.CMAKE_BUILD_BORINGSSL_DIR }}" --target test

platform-sanitizer-tests:
clang-tidy:
if: github.event.pull_request.draft == false
needs: quick-linux-interop-check
name: Build and test platforms using sanitizers and clang-tidy
runs-on: ${{ matrix.os }}
needs: build-and-unit-test
name: Build with clang-tidy
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
os: [windows-latest, ubuntu-latest, macos-latest]
include:
- os: windows-latest
ossl3-vcpkg-dir: "alternatives\\openssl_3"
boringssl-vcpkg-dir: "alternatives\\boringssl_1.1"
ctest-target: RUN_TESTS
- os: ubuntu-latest
ossl3-vcpkg-dir: "alternatives/openssl_3"
boringssl-vcpkg-dir: "alternatives/boringssl_1.1"
ctest-target: test
- os: macos-latest
ossl3-vcpkg-dir: "alternatives/openssl_3"
boringssl-vcpkg-dir: "alternatives/boringssl_1.1"
ctest-target: test

steps:
- name: Checkout repository and submodules
uses: actions/checkout@v4
with:
submodules: recursive
fetch-depth: 0

# write the commit hash of vcpkg to a text file so we can use it in the
# hashFiles for cache
- run: |
git -C ${{ env.VCPKG_REPO }} rev-parse HEAD > vcpkg_commit.txt

# First, attempt to pull key key, if that is not present, pull one of the
# restore-keys so we do not need to build from scratch.
# CACHE_VERSION - provide a way to reset cache
# CACHE_NAME - name of the cache in order to manage it
# matrix.os - cache per OS and version
# hashFiles - Recache if the vcpkg files change
- name: Restore Cache
uses: actions/cache@v3
with:
path: ${{ github.workspace }}/build/cache
key: ${{ env.CACHE_VERSION }}-${{ env.CACHE_NAME }}-${{ matrix.os }}-${{ hashFiles('vcpkg_commit.txt', 'vcpkg.json', 'alternatives/openssl_3/vcpkg.json') }}
restore-keys: |
${{ env.CACHE_VERSION }}-${{ env.CACHE_NAME }}-${{ matrix.os }}

- name: Dependencies (macOs)
if: ${{ matrix.os == 'macos-latest' }}
run: |
brew install llvm pkg-config nasm
ln -s "/usr/local/opt/llvm/bin/clang-format" "/usr/local/bin/clang-format"
ln -s "/usr/local/opt/llvm/bin/clang-tidy" "/usr/local/bin/clang-tidy"

- name: Dependencies (Ubuntu)
if: ${{ matrix.os == 'ubuntu-latest' }}
run: |
sudo apt-get install -y linux-headers-$(uname -r) nasm

- name: Build (OpenSSL 1.1)
run: |
cmake -B "${{ env.CMAKE_BUILD_DIR }}" -DTESTING=ON -DCLANG_TIDY=ON -DSANITIZERS=ON -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake --build "${{ env.CMAKE_BUILD_DIR }}"

- name: Unit Test (OpenSSL 1.1)
run: |
cmake --build "${{ env.CMAKE_BUILD_DIR }}" --target "${{ matrix.ctest-target}}"

- name: Build (OpenSSL 3)
run: |
cmake -B "${{ env.CMAKE_BUILD_OPENSSL3_DIR }}" -DTESTING=ON -DCLANG_TIDY=ON -DSANITIZERS=ON -DVCPKG_MANIFEST_DIR="${{ matrix.ossl3-vcpkg-dir }}" -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake --build "${{ env.CMAKE_BUILD_OPENSSL3_DIR }}"

- name: Unit Test (OpenSSL 3)
run: |
cmake --build "${{ env.CMAKE_BUILD_OPENSSL3_DIR }}" --target "${{ matrix.ctest-target}}"

- name: Build (BoringSSL)
run: |
cmake -B "${{ env.CMAKE_BUILD_BORINGSSL_DIR }}" -DTESTING=ON -DCLANG_TIDY=ON -DSANITIZERS=ON -DVCPKG_MANIFEST_DIR="${{ matrix.boringssl-vcpkg-dir }}" -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake --build "${{ env.CMAKE_BUILD_BORINGSSL_DIR }}"

- name: Unit Test (BoringSSL)
run: |
cmake --build "${{ env.CMAKE_BUILD_BORINGSSL_DIR }}" --target "${{ matrix.ctest-target}}"

old-macos-compatibility:
if: github.event.pull_request.draft == false
needs: quick-linux-interop-check
name: Build for older MacOS
runs-on: macos-latest
crypto: [openssl_1.1, openssl_3, boringssl]
bifurcation marked this conversation as resolved.
Show resolved Hide resolved

env:
CMAKE_BUILD_DIR: ${{ github.workspace }}/build
VCPKG_BINARY_SOURCES: files,${{ github.workspace }}/build/cache,readwrite
MACOSX_DEPLOYMENT_TARGET: 10.11
BUILD_DIR: "${RUNNER_TEMP}/build_${{ matrix.crypto }}"
CRYPTO_DIR: "./alternatives/${{ matrix.crypto }}"

steps:
- name: Checkout repository and submodules
uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
submodules: recursive
fetch-depth: 0

# write the commit hash of vcpkg to a text file so we can use it in the
# hashFiles for cache
- run: |
git -C ${{ env.VCPKG_REPO }} rev-parse HEAD > vcpkg_commit.txt

# First, attempt to pull key key, if that is not present, pull one of the
# restore-keys so we do not need to build from scratch.
# CACHE_VERSION - provide a way to reset cache
# CACHE_NAME - name of the cache in order to manage it
# matrix.os - cache per OS and version
# hashFiles - Recache if the vcpkg files change
- name: Restore Cache
uses: actions/cache@v3
- uses: ./.github/actions/prepare-build
with:
path: ${{ github.workspace }}/build/cache
key: ${{ env.CACHE_VERSION }}-${{ env.CACHE_NAME }}-macos-latest-legacy-${{ hashFiles('vcpkg_commit.txt', 'vcpkg.json', 'alternatives/openssl_3/vcpkg.json') }}
restore-keys: |
${{ env.CACHE_VERSION }}-${{ env.CACHE_NAME }}-macos-latest-legacy
${{ env.CACHE_VERSION }}-${{ env.CACHE_NAME }}-macos-latest
os: ubuntu-latest
crypto: matrix.crypto
cache-dir: ${{ github.workspace }}/vcpkg_cache

- name: Dependencies
- name: Build with clang-tidy
run: |
brew install llvm pkg-config
ln -s "/usr/local/opt/llvm/bin/clang-format" "/usr/local/bin/clang-format"
ln -s "/usr/local/opt/llvm/bin/clang-tidy" "/usr/local/bin/clang-tidy"

- name: Build
run: |
cmake -B "${{ env.CMAKE_BUILD_DIR }}" -DCMAKE_TOOLCHAIN_FILE="${{ env.VCPKG_TOOLCHAIN_FILE }}"
cmake --build "${{ env.CMAKE_BUILD_DIR }}" --target mlspp

cmake -B "${{ env.BUILD_DIR }}" -DVCPKG_MANIFEST_DIR="${{ env.CRYPTO_DIR }}" \
-DTESTING=ON -DCLANG_TIDY=ON -DSANITIZERS=ON
cmake --build "${{ env.BUILD_DIR }}"
Loading
Loading