Updates/Fixes: merge in recent OTNS1 PRs (#161)

This commit includes all PRs merged into OTNS (v1) main branch, since the OTNS (v2) fork was created in 2022. Some PRs are applied as is, others are adapted to the new OTNS2 codebase. Also this includes version increases for Go 1.x and some Go and Python libraries used to newer versions. It also introduces some new features that were needed to optimally merge the PRs according to their intent.

* YAML based startup script configuration: some node types (ftd/mtd/br) can now have their own unique startup command sequence, encoded in the same script file. This also enables for the future a fully YAML based configuration of a simulation including all node positions, radio models, startup scripts, etc.

* [github-actions] fix for pushes do not run CI (openthread#395) - this was not done. It appears it is currently not needed, leads to many cancelled CI tasks. CI tasks will run already on every push to the PR branch. Keeping this note as informative message.

* update golangci-lint tool and fix pretty issue based on commit abf59e0

* [script] install python3-venv as an attempt to fix failing Docker build

* Bump wheel from 0.34.2 to 0.38.1 in /pylibs/unittests (openthread#439)
Bumps [wheel](https://github.com/pypa/wheel) from 0.34.2 to 0.38.1.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](pypa/wheel@0.34.2...0.38.1)
---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump golang.org/x/net from 0.0.0-20201021035429-f5854403a974 to 0.7.0 (openthread#478)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20201021035429-f5854403a974 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/commits/v0.7.0)
---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [.github][go.mod] update to go 1.22

* [.github] some version quoting issues / bump setup-go action

* [.github][GUIDE][go.mod] require Go 1.18 minimum; build for 1.20 and 1.22 also; clarified GUIDE

* docker file updates -> go 1.22 / removed unneeded items

* [script] fix hanging golangci-lint for go 1.20, 1.22 - bump version to 1.59.0

* [scripts] add wheel to install_python_libs() (openthread#485)
This should address this warning https://github.com/openthread/openthread/actions/runs/4321875033/jobs/7543625431#step:8:348

* [.github] quote go versions

* [dependabot] reduce frequency to monthly (openthread#494)

* [all] [tests] properly configure operational dataset (openthread#502)

* [pylibs] fix raw test_commissioning by giving more time for nodes to scan channels.

* [.github] change order of unit tests

* [openthread] bump

* Bump grpcio from 1.46.3 to 1.53.0 in /pylibs/unittests (openthread#506)
Bumps [grpcio](https://github.com/grpc/grpc) from 1.46.3 to 1.53.0.
- [Release notes](https://github.com/grpc/grpc/releases)
- [Changelog](https://github.com/grpc/grpc/blob/master/doc/grpc_release_schedule.md)
- [Commits](grpc/grpc@v1.46.3...v1.53.0)
---
updated-dependencies:
- dependency-name: grpcio
  dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pylibs] fixes requirements.txt bump protobuf, wheel

* [go.mod] lib versions update

* [otnstester][cli]  fix cli unit-test

* [pylibs] in tests don't set trace watch level by default / remove unnecessary 'watch default debug'

* [script] fix issue that historic commits not found in --depth 1 cloned repo.

* [pylibs] force_key_rotation test extension

* [pylibs] increase network-limits time limit for case BR is a Parent, to avoid spurious test fails.

* [pylibs] otns-performance stress test runs with fixed random seed (to get a more predictable baseline performance for number of events), to avoid spurious test fails on number of events.

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Simon Lin <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mason Tran <[email protected]>
Co-authored-by: Jonathan Hui <[email protected]>

.github/dependabot.yml

@@ -1,4 +1,4 @@
-# Copyright (c) 2021-2022, The OTNS Authors.
+# Copyright (c) 2021-2024, The OTNS Authors.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -29,12 +29,12 @@ updates:
   - package-ecosystem: "gitsubmodule"
     directory: "/"
     schedule:
-      interval: "weekly"
+      interval: "monthly"
     allow:
-      - dependency-name: "ot-rfsim"
+      - dependency-name: "openthread"
     commit-message:
       prefix: "submodule"
     rebase-strategy: "disabled"
     open-pull-requests-limit: 1
     assignees:
-      - "EskoDijk"
+      - "jwhui"

.github/workflows/build.yml

@@ -27,6 +27,7 @@
 name: Build
 
 on:
+  workflow_dispatch:
   pull_request:
     branches:
       - 'main'
@@ -46,10 +47,10 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: [ 1.18, 1.22 ]
+        go: [ '1.18', '1.20', '1.22' ]
         os: [ ubuntu-22.04, macos-13, macos-14 ]
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go }}
       - uses: actions/checkout@v3

.github/workflows/develop.yml

@@ -49,11 +49,11 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: [1.18]
+        go: [1.22]
         python-version: ['3.10']
         os: [ubuntu-22.04]
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go }}
       - uses: actions/setup-python@v5

.github/workflows/docker.yml

@@ -1,4 +1,4 @@
-# Copyright (c) 2020-2023, The OTNS Authors.
+# Copyright (c) 2020-2024, The OTNS Authors.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -56,7 +56,7 @@ jobs:
           submodules: recursive
           fetch-depth: 0
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v1
+        uses: docker/setup-buildx-action@v3
       - name: Prepare
         id: prepare
         run: |

.github/workflows/lint.yml

@@ -27,6 +27,7 @@
 name: Lint
 
 on:
+  workflow_dispatch:
   pull_request:
     branches:
       - 'main'
@@ -46,9 +47,9 @@ jobs:
     env:
       HOMEBREW_NO_AUTO_UPDATE: 1
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: 1.18
+          go-version: '1.22'
       - uses: actions/checkout@v3
       - name: Check pretty
         run: |

.github/workflows/stress.yml

@@ -27,6 +27,7 @@
 name: Stress
 
 on:
+  workflow_dispatch:
   pull_request:
     branches:
       - 'main'
@@ -46,14 +47,14 @@ jobs:
       fail-fast: false
       matrix:
         python-version: ['3.10']
-        go-version: [1.18]
+        go-version: ['1.22']
         suite: ["network-forming", "commissioning", "connectivity", "network-latency", "multicast-performance", "otns-performance", "network-limits"]
     runs-on: ubuntu-22.04
     timeout-minutes: 120
     env:
       HOMEBREW_NO_AUTO_UPDATE: 1
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go-version }}
       - name: Set up Python

.github/workflows/test.yml

@@ -27,6 +27,7 @@
 name: Test
 
 on:
+  workflow_dispatch:
   pull_request:
     branches:
       - 'main'
@@ -51,9 +52,9 @@ jobs:
     env:
       HOMEBREW_NO_AUTO_UPDATE: 1
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: 1.18
+          go-version: '1.22'
       - uses: actions/checkout@v3
       - name: Test
         run: |
@@ -73,20 +74,20 @@ jobs:
     env:
       HOMEBREW_NO_AUTO_UPDATE: 1
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: 1.18
+          go-version: '1.22'
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - uses: actions/checkout@v3
-      - name: Test multiple OT-versions
-        run: |
-          ./script/test py-ver-unittests
       - name: Test single OT-version
         run: |
           ./script/test py-unittests
+      - name: Test multiple OT-versions
+        run: |
+          ./script/test py-ver-unittests
 
   py-examples:
     name: Examples (${{ matrix.os }})
@@ -100,9 +101,9 @@ jobs:
     env:
       HOMEBREW_NO_AUTO_UPDATE: 1
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: 1.18
+          go-version: '1.22'
       - name: Set up Python
         uses: actions/setup-python@v5
         with:

.golangci.yml

@@ -0,0 +1,3 @@
+run:
+  skip-files:
+    - "web/site/bindata.go"

GUIDE.md

@@ -5,7 +5,7 @@ This guide covers the installation of Go, installation of OTNS, use of the OTNS
 
 OTNS requires [Go 1.18 or higher](https://golang.org/dl/) to build:
 
- - Install Go from https://golang.org/dl/
+ - Install Go from https://golang.org/dl/ or via a package manager (in this case check the Go version is high enough).
  - Add `$(go env GOPATH)/bin` (normally `$HOME/go/bin`) to `$PATH`.
 
 ## Get OTNS code
@@ -17,19 +17,21 @@ cd otns
 
 ## Automated installation
 
-An automated way to install dependencies, OTNS and all OT nodes is the following command:
+An automated way to install dependencies, OTNS and all OT nodes, and test the result, is the following command:
 
 ```bash
-./script/test build_openthread_versions
+./script/test go-tests
 ```
 
-Alternatively the extensive unit tests can be run also with the following command:
+Alternatively the more extensive Python unit tests can be run also with the following command:
 
 ```bash
 ./script/test py-unittests
 ```
 
-However, this can take a long time (5-10 minutes).
+However, this can take a long time (5-10 minutes). Running any Python (`py-*`) tests will set up a Python 3 virtual 
+environment locally in the `.venv-otns` directory. This virtual environment must also be active when manually running 
+any OTNS Python scripts.
 
 ## Manual step-by-step installation
 
@@ -171,7 +173,8 @@ See [OTNS CLI Reference](cli/README.md).
 ## OTNS Python Scripting
 
 [pyOTNS](pylibs/otns) library provides utilities to create and manage simulations through OTNS CLI. It is installed in a 
-Python 3 virtual environment `.venv-otns`. 
+Python 3 virtual environment `.venv-otns` by the `./script/install` script. The `./script/test` script also calls this 
+install script when needed as part of setup.
 
 ### Python Scripting Documentation
 

cli/ast.go

@@ -275,9 +275,9 @@ type ScanCmd struct {
 
 // noinspection GoVetStructTag
 type SpeedCmd struct {
-	Cmd   struct{}      `"speed"`               //nolint
-	Max   *MaxSpeedFlag `( @@`                  //nolint
-	Speed *float64      `| [ (@Int|@Float) ] )` //nolint
+	Cmd   struct{}      `"speed"`             //nolint
+	Max   *MaxSpeedFlag `[ ( @@`              //nolint
+	Speed *float64      `| (@Int|@Float) ) ]` //nolint
 }
 
 // noinspection GoVetStructTag

etc/cli-scripts/ot-script-example.yaml

@@ -0,0 +1,48 @@
+#
+# Example OT node CLI script, which can be passed to otns using the -ot-script parameter.
+# It's in YAML format. Script comment lines start with '#'.
+# TODO: enable using script config in same file with YAML network config.
+#
+
+script:
+    ftd: |
+        # Active Dataset parameters. This only works for an FTD (e.g. type 'router' or 'fed').
+        # Therefore it's in the 'ftd' script section.
+        dataset init new
+        dataset networkname Test\ Network
+        dataset panid 0x1234
+        dataset channel 15
+        dataset networkkey 998877665544332211ffeeddccbbaa00
+        dataset meshlocalprefix fd00:abba::
+        dataset commit active
+
+        # Some extra settings - differ from usual setup.
+        routerselectionjitter 10
+
+    mtd: |
+        # MTD not able to use 'dataset init new'. Therefore, other way of providing active dataset.
+        networkkey 998877665544332211ffeeddccbbaa00
+        panid 0x1234
+        channel 15
+
+    br: |
+        # BR-specific configuration goes here. There's some spaces at EOL to test script-parsing.
+        ############# 
+        routerselectionjitter 1       
+        routerdowngradethreshold 33                          
+        routerupgradethreshold 33                                                           
+        netdata publish route fc00::/7 s med                 
+        bbr enable
+        srp server enable
+        br init 1 1
+        br enable
+
+    all: |
+        # Some extra settings - differ from usual setup. These are applied to all node types,
+        # except for 'raw' added nodes.
+        ccathreshold -50
+        txpower 10
+
+        # Autostart the node
+        ifconfig up
+        thread start

etc/docker/environment/Dockerfile

@@ -1,4 +1,4 @@
-# Copyright (c) 2022-2023, The OTNS Authors.
+# Copyright (c) 2022-2024, The OTNS Authors.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -24,7 +24,7 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
-FROM golang:1.18
+FROM golang:1.22
 
 RUN apt-get update && apt-get install -y python3 python3-pip sudo unzip nodejs npm git && rm -rf /var/lib/apt/lists/*
 

etc/docker/playground/Dockerfile

@@ -1,4 +1,4 @@
-# Copyright (c) 2020, The OTNS Authors.
+# Copyright (c) 2020-2024, The OTNS Authors.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -25,21 +25,16 @@
 # POSSIBILITY OF SUCH DAMAGE.
 
 # Stage 0: build OTNS and dependencies
-FROM golang:1.18
+FROM golang:1.22
 
 RUN apt-get update
 RUN apt-get install -y python3 python3-pip unzip
 
 COPY . /otns
 WORKDIR /otns
 RUN ./script/install-deps
+RUN ./script/test build-openthread-versions
 RUN ./script/install
-WORKDIR /otns/ot-rfsim
-RUN ./script/bootstrap
-RUN ./script/build_latest
-RUN ./script/build_v11
-RUN ./script/build_v12
-RUN ./script/build_v13
 
 RUN strip /go/bin/grpcwebproxy /go/bin/otns
 

go.mod

@@ -1,4 +1,4 @@
-// Copyright (c) 2020-2023, The OTNS Authors.
+// Copyright (c) 2020-2024, The OTNS Authors.
 // All rights reserved.
 //
 // Redistribution and use in source and binary forms, with or without
@@ -29,28 +29,26 @@ module github.com/openthread/ot-ns
 go 1.18
 
 require (
-	github.com/alecthomas/participle v0.5.0
+	github.com/alecthomas/participle v0.7.1
 	github.com/chzyer/readline v1.5.1
 	github.com/mitchellh/go-wordwrap v1.0.1
 	github.com/pkg/errors v0.9.1
-	github.com/stretchr/testify v1.8.4
-	go.uber.org/zap v1.22.0
-	golang.org/x/term v0.4.0
-	google.golang.org/grpc v1.48.0
-	google.golang.org/protobuf v1.27.1
+	github.com/stretchr/testify v1.9.0
+	go.uber.org/zap v1.27.0
+	golang.org/x/term v0.21.0
+	google.golang.org/grpc v1.64.0
+	google.golang.org/protobuf v1.34.1
 	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	go.uber.org/atomic v1.7.0 // indirect
-	go.uber.org/multierr v1.6.0 // indirect
-	golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4 // indirect
-	golang.org/x/sys v0.4.0 // indirect
-	golang.org/x/text v0.3.3 // indirect
-	google.golang.org/genproto v0.0.0-20200608115520-7c474a2e3482 // indirect
+	go.uber.org/multierr v1.10.0 // indirect
+	golang.org/x/net v0.22.0 // indirect
+	golang.org/x/sys v0.21.0 // indirect
+	golang.org/x/text v0.14.0 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 // indirect
 	gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f // indirect
 )