[Ready for review] Fixed e2e tests #964
Conversation
cchen-vertica
requested review from
roypaulin,
fenic-fawkes,
HaoYang0000,
qindotguan and
LiboYu2
as code owners
cchen-vertica
force-pushed
the
cchen/fix-e2e
branch
2 times, most recently
from
7eaf23c
to
a9c2551
Compare
|
Should we fix those tests before merging this? Otherwise, regular e2e may always fail. |
|
We should also increase the timeout of istio, maybe to 12 min? As sometimes the istio job failed due to timeout |
cchen-vertica
force-pushed
the
cchen/fix-e2e
branch
from
a9c2551
to
1753338
Compare
cchen-vertica
force-pushed
the
cchen/fix-e2e
branch
from
d84fd5b
to
1306ca6
Compare
cchen-vertica
changed the title
Yes, I fixed the tests. The vdb-gen test is still somewhat flaky, but it’s much more stable now. |
istio should be handled by other PR. This one is for leg4 and udx. Also, before increasing the timeout, we should confirm that Istio is failing due to slowness. For instance, issues with vdb-gen and udx are caused by other factors, so increasing the timeout would only extend our wait time without resolving the root problems. |
| - name: Pick the name of the image | ||
| uses: spilchen/switch-case-action@v2 | ||
| id: legacy_vertica_image | ||
| with: | ||
| default: ghcr.io/${{ env.OWNER_LC }}/vertica-k8s:${{ github.sha }}-legacy | ||
| default: docker.io/opentext/vertica-k8s-private:latest-legacy-master |
There was a problem hiding this comment.
What's the difference between the sha-legacy and the latest-legacy-master image?
There was a problem hiding this comment.
sha-legacy image will be built by e2e tests, and latest-legacy-master is built by devops and already exists before e2e tests run.
| @@ -128,6 +128,9 @@ function scrutinizeForAdmintools() { | |||
| OP_FILE="$ns.$v.scrutinize.tar" | |||
| logInfo "Running scrutinize" | |||
| set -o xtrace | |||
| # scrutinize needs to be able to ssh to the pod | |||
| POD_IP=$(kubectl get pod -n $ns $pod -o jsonpath='{.status.podIP}') | |||
| kubectl exec -t -n $ns $pod -- bash -c "ssh-keyscan $POD_IP >> ~/.ssh/known_hosts" | |||
There was a problem hiding this comment.
Why do we need this now?
There was a problem hiding this comment.
Scrutinize still has a bug. See my comment under this Jira: https://jira.verticacorp.com/jira/browse/VER-96447. This is a workaround before client team fixed the scrutinize.
This PR addresses issues in the daily run for leg-4 and udx.
Leg-4: I improved stability by refactoring and isolating vdb-gen, reducing the likelihood of failures. However, the root cause of the reconcile requeue not functioning in GitHub CI remains unclear. After we use our own runner, we can revisit this issue again. Additionally, I resolved K8s scrutinizer issues for other failed revive tests.
UDX: I updated the DevOps Jenkins pipeline and fixed a script issue in the e2e test. Now udx is stable.