Merge pull request #7736 from NvTimLiu/release-tmp

Merge branch 'branch-23.02' into main [skip ci]

.github/workflows/add-to-project.yml

@@ -1,4 +1,4 @@
-# Copyright (c) 2022, NVIDIA CORPORATION.
+# Copyright (c) 2022-2023, NVIDIA CORPORATION.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -28,8 +28,8 @@ jobs:
     name: Add new issues and pull requests to project
     runs-on: ubuntu-latest
     steps:
-      # TODO: update project version when new release supports node 16 instead of 12
-      - uses: actions/[email protected]
+      - uses: actions/[email protected]
         with:
           project-url: https://github.com/orgs/NVIDIA/projects/4
           github-token: ${{ secrets.PROJECT_TOKEN }}
+

.github/workflows/auto-merge.yml

@@ -1,4 +1,4 @@
-# Copyright (c) 2020-2022, NVIDIA CORPORATION.
+# Copyright (c) 2020-2023, NVIDIA CORPORATION.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,7 +18,7 @@ name: auto-merge HEAD to BASE
 on:
   pull_request_target:
     branches:
-    - branch-22.12
+    - branch-23.02
     types: [closed]
 
 jobs:
@@ -29,13 +29,13 @@ jobs:
     steps:
       - uses: actions/checkout@v3
         with:
-          ref: branch-22.12 # force to fetch from latest upstream instead of PR ref
+          ref: branch-23.02 # force to fetch from latest upstream instead of PR ref
 
       - name: auto-merge job
         uses: ./.github/workflows/auto-merge
         env:
           OWNER: NVIDIA
           REPO_NAME: spark-rapids
-          HEAD: branch-22.12
-          BASE: branch-23.02
+          HEAD: branch-23.02
+          BASE: branch-23.04
           AUTOMERGE_TOKEN: ${{ secrets.AUTOMERGE_TOKEN }} # use to merge PR

CONTRIBUTING.md

@@ -54,6 +54,20 @@ To get an uber jar with more than 1 version you have to `mvn package` each versi
 and then use one of the defined profiles in the dist module, or a comma-separated list of
 build versions. See the next section for more details.
 
+You might see a warning during scala-maven-plugin compile goal invocation.
+```
+[INFO] Compiling 94 Scala sources and 1 Java source to /home/user/gits/NVIDIA/spark-rapids/tests/target/spark3XY/test-classes ...
+OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
+OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
+CodeCache: size=245760Kb used=236139Kb max_used=243799Kb free=9620Kb
+ bounds [0x00007f9681000000, 0x00007f9690000000, 0x00007f9690000000]
+ total_blobs=60202 nmethods=59597 adapters=504
+ compilation: disabled (not enough contiguous free space left)
+```
+
+It can be mitigated by increasing [ReservedCodeCacheSize](https://spark.apache.org/docs/3.3.1/building-spark.html#setting-up-mavens-memory-usage)
+passed in the `MAVEN_OPTS` environment variable.
+
 ### Building a Distribution for Multiple Versions of Spark
 
 By default the distribution jar only includes code for a single version of Spark. If you want
@@ -138,7 +152,7 @@ To this end in a pre-production build you can set the Boolean property
 
 The time saved is more significant if you are merely changing
 the `aggregator` module, or the `dist` module, or just incorporating changes from
-[spark-rapids-jni](https://github.com/NVIDIA/spark-rapids-jni/blob/branch-22.12/CONTRIBUTING.md#local-testing-of-cross-repo-contributions-cudf-spark-rapids-jni-and-spark-rapids)
+[spark-rapids-jni](https://github.com/NVIDIA/spark-rapids-jni/blob/branch-23.02/CONTRIBUTING.md#local-testing-of-cross-repo-contributions-cudf-spark-rapids-jni-and-spark-rapids)
 
 For example, to quickly repackage `rapids-4-spark` after the
 initial `./build/buildall` you can iterate by invoking
@@ -200,27 +214,46 @@ This may require some modifications to IDEs' standard Maven import functionality
 
 #### IntelliJ IDEA
 
-_Last tested with 2021.2.1 Community Edition_
+Last tested with IntelliJ IDEA 2022.3.1 (Community Edition)
 
 To start working with the project in IDEA is as easy as
 [opening](https://blog.jetbrains.com/idea/2008/03/opening-maven-projects-is-easy-as-pie/) the top level (parent)
 [pom.xml](pom.xml).
 
+In the most recent versions of IDEA [unselect](https://www.jetbrains.com/help/idea/2022.3/maven-importing.html)
+"Import using the new IntelliJ Workspace Model API (experimental)".
 In order to make sure that IDEA handles profile-specific source code roots within a single Maven module correctly,
-[unselect](https://www.jetbrains.com/help/idea/2021.2/maven-importing.html) "Keep source and test folders on reimport".
+[unselect](https://www.jetbrains.com/help/idea/2022.3/maven-importing.html) "Keep source and test folders on reimport".
 
 If you develop a feature that has to interact with the Shim layer or simply need to test the Plugin with a different
-Spark version, open [Maven tool window](https://www.jetbrains.com/help/idea/2021.2/maven-projects-tool-window.html) and
-select one of the `release3xx` profiles (e.g, `release320`) for Apache Spark 3.2.0, and click "Reload"
-if not triggered automatically.
+Spark version, open [Maven tool window](https://www.jetbrains.com/help/idea/2022.3/maven-projects-tool-window.html) and
+select one of the `release3xx` profiles (e.g, `release320`) for Apache Spark 3.2.0.
+
+Go to `File | Settings | Build, Execution, Deployment | Build Tools | Maven | Importing` and make sure
+that `Generated sources folders` is set to `Detect automatically` and `Phase to be used for folders update`
+is changed to `process-test-resources`. In the Maven tool window hit `Reload all projects` and
+`Generate Sources and Update Folders For all Projects`.
 
-There is a known issue where, even after selecting a different Maven profile in the Maven submenu, the source folders from
-a previously selected profile may remain active. To get around this you have to manually reload the Maven project from
-the Maven side menu.
+Known Issues:
+
+* There is a known issue that the test sources added via the `build-helper-maven-plugin` are not handled
+[properly](https://youtrack.jetbrains.com/issue/IDEA-100532). The workaround is to `mark` the affected folders
+such as `tests/src/test/320+-noncdh-nondb` manually as `Test Sources Root`
+
+* There is a known issue where, even after selecting a different Maven profile in the Maven submenu,
+the source folders from a previously selected profile may remain active. As a workaround,
+when switching to a different profile, go to
+`File | Project Structure ... | Modules`, select the `rapids-4-spark-sql_2.12` module,
+click `Sources`, and delete all the shim source roots from the `Source Folders` list. Make sure
+the right test source folders are in `rapids-4-spark-sql_2.12` and `rapids-4-spark-tests_2.12`.
+Re-execute the steps above: reload, and `Generate Sources ...`.
 
 If you see Scala symbols unresolved (highlighted red) in IDEA please try the following steps to resolve it:
-- Make sure there are no relevant poms in "File->Settings->Build Tools->Maven->Ignored Files"
-- Restart IDEA and click "Reload All Maven Projects" again
+
+* Make sure there are no relevant poms in
+`File | Settings | Build, Execution, Deployment | Build Tools | Maven | Ignored Files`
+
+* Restart IDEA and click `Reload All Maven Projects` again
 
 #### Bloop Build Server
 

LICENSE

@@ -199,157 +199,3 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-
-
-MIT License
------------
-
-Boostrap V4.6.1
-    bootstrap.bundle.min.js
-    bootstrap.min.css
-
-    The MIT License (MIT)
-
-    Copyright (c) 2011-2019 Twitter, Inc.
-    Copyright (c) 2011-2019 The Bootstrap Authors
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in
-    all copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-    THE SOFTWARE.
-
-
-DataTables
-    DataTables-1.12.0/jquery.dataTables.min.js
-    DataTables-1.12.0/dataTables.bootstrap4.min.css
-    DataTables-1.12.0/dataTables.bootstrap4.min.js
-    searchpanes-2.0.1/searchPanes.bootstrap4.min.css
-    searchpanes-2.0.1/searchPanes.bootstrap4.min.js
-    searchpanes-2.0.1/dataTables.searchPanes.min.js
-    select-1.4.0/select.bootstrap4.min.css
-    select-1.4.0/dataTables.select.min.js
-    buttons-2.2.3/buttons.bootstrap4.min.css
-    buttons-2.2.3/buttons.bootstrap4.min.js
-    buttons-2.2.3/buttons.html5.min.js
-    buttons-2.2.3/dataTables.buttons.min.js
-    responsive-2.3.0/dataTables.responsive.min.js
-    responsive-2.3.0/responsive.bootstrap4.min.css
-    responsive-2.3.0/responsive.bootstrap4.min.js
-
-    The MIT License (MIT)
-
-    Copyright (C) 2008-2022, SpryMedia Ltd.
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy of this software
-    and associated documentation files (the "Software"), to deal in the Software without
-    restriction, including without limitation the rights to use, copy, modify, merge, publish,
-    distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the
-    Software is furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all copies or
-    substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
-    BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-    NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-    DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-jQuery V3.6.0
-    jquery.min.js
-
-    Copyright JS Foundation and other contributors, https://js.foundation/
-
-    This software consists of voluntary contributions made by many
-    individuals. For exact contribution history, see the revision history
-    available at https://github.com/jquery/sizzle
-
-    The following license applies to all parts of this software except as
-    documented below:
-
-    ====
-
-    Permission is hereby granted, free of charge, to any person obtaining
-    a copy of this software and associated documentation files (the
-    "Software"), to deal in the Software without restriction, including
-    without limitation the rights to use, copy, modify, merge, publish,
-    distribute, sublicense, and/or sell copies of the Software, and to
-    permit persons to whom the Software is furnished to do so, subject to
-    the following conditions:
-
-    The above copyright notice and this permission notice shall be
-    included in all copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-    EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-    MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-    NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-    LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-    WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-    ====
-
-    All files located in the node_modules and external directories are
-    externally maintained libraries used by this software which have their
-    own licenses; we recommend you read them, as their terms may differ from
-    the terms above.
-
-Mustache.js V4.10
-    mustache.min.js
-
-    The MIT License
-
-    Copyright (c) 2009 Chris Wanstrath (Ruby)
-    Copyright (c) 2010-2014 Jan Lehnardt (JavaScript)
-    Copyright (c) 2010-2015 The mustache.js community
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy of this software
-    and associated documentation files (the "Software"), to deal in the Software without
-    restriction, including without limitation the rights to use, copy, modify, merge, publish,
-    distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the
-    Software is furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all copies or
-    substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
-    BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-    NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-    DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-Spur Dashboard V1.1.0
-    spur.min.css
-
-    The MIT License (MIT)
-
-    Copyright 2018 Alexander Rechsteiner
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy of this software
-    and associated documentation files (the "Software"), to deal in the Software without
-    restriction, including without limitation the rights to use, copy, modify, merge, publish,
-    distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the
-    Software is furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all copies or
-    substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
-    BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-    NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-    DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

NOTICE

@@ -64,30 +64,3 @@ the following copyright notice:
 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 | See the License for the specific language governing permissions and
 | limitations under the License.
-
---------------------------------------------------------------------------------
-
-This product bundles various third-party components under other open source licenses.
-
-Boostrap V4.6.1 - The MIT License (MIT)
-    License Text ( https://github.com/twbs/bootstrap/blob/v4.6.1/LICENSE )
-    Copyright (c) 2011-2021 Twitter, Inc.
-    Copyright (c) 2011-2021 The Bootstrap Authors
-
-DataTablesSrc V1.12 - The MIT License (MIT)
-    License Text ( https://datatables.net/license/ )
-    Copyright (C) 2008-present, SpryMedia Ltd.
-
-jQuery V3.6.0 - The MIT License (MIT)
-    License Text ( https://jquery.org/license/ )
-    Copyright OpenJS Foundation and other contributors, https://openjsf.org/
-
-Mustache.js V4.10 - The MIT License (MIT)
-    License Text ( https://github.com/janl/mustache.js/blob/master/LICENSE )
-    Copyright (c) 2009 Chris Wanstrath (Ruby)
-    Copyright (c) 2010-2014 Jan Lehnardt (JavaScript)
-    Copyright (c) 2010-2015 The mustache.js community
-
-Spur Dashboard V1.1.0 - The MIT License (MIT)
-    License Text ( https://github.com/HackerThemes/spur-template/blob/master/LICENSE )
-    Copyright 2016 - 2019 Alexander Rechsteiner

NOTICE-binary

@@ -463,29 +463,3 @@ misrepresented as being the original software.
 3. This notice may not be removed or altered from any source distribution.
 
 Mark Adler    [email protected]
-
----------------------------------------------------------------------
-This product bundles various third-party components under other open source licenses.
-
-Boostrap V4.6.1 - The MIT License (MIT)
-    License Text ( https://github.com/twbs/bootstrap/blob/v4.6.1/LICENSE )
-    Copyright (c) 2011-2021 Twitter, Inc.
-    Copyright (c) 2011-2021 The Bootstrap Authors
-
-DataTablesSrc V1.12 - The MIT License (MIT)
-    License Text ( https://datatables.net/license/ )
-    Copyright (C) 2008-present, SpryMedia Ltd.
-
-jQuery V3.6.0 - The MIT License (MIT)
-    License Text ( https://jquery.org/license/ )
-    Copyright OpenJS Foundation and other contributors, https://openjsf.org/
-
-Mustache.js V4.10 - The MIT License (MIT)
-    License Text ( https://github.com/janl/mustache.js/blob/master/LICENSE )
-    Copyright (c) 2009 Chris Wanstrath (Ruby)
-    Copyright (c) 2010-2014 Jan Lehnardt (JavaScript)
-    Copyright (c) 2010-2015 The mustache.js community
-
-Spur Dashboard V1.1.0 - The MIT License (MIT)
-    License Text ( https://github.com/HackerThemes/spur-template/blob/master/LICENSE )
-    Copyright 2016 - 2019 Alexander Rechsteiner

README.md

@@ -1,9 +1,11 @@
 # RAPIDS Accelerator For Apache Spark
-NOTE: For the latest stable [README.md](https://github.com/nvidia/spark-rapids/blob/main/README.md) ensure you are on the main branch. The RAPIDS Accelerator for Apache Spark provides a set of plugins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries and UCX. Documentation on the current release can be found [here](https://nvidia.github.io/spark-rapids/).
+NOTE: For the latest stable [README.md](https://github.com/nvidia/spark-rapids/blob/main/README.md) ensure you are on the main branch.
 
 The RAPIDS Accelerator for Apache Spark provides a set of plugins for
-[Apache Spark](https://spark.apache.org) that leverage GPUs to accelerate processing
-via the [RAPIDS](https://rapids.ai) libraries and [UCX](https://www.openucx.org/).
+[Apache Spark](https://spark.apache.org) that leverage GPUs to accelerate processing 
+via the [RAPIDS](https://rapids.ai) libraries.
+
+Documentation on the current release can be found [here](https://nvidia.github.io/spark-rapids/).
 
 To get started and try the plugin out use the [getting started guide](./docs/get-started/getting-started.md).
 
@@ -54,12 +56,8 @@ access to any of the memory that RMM is holding.
 
 ## Qualification and Profiling tools
 
-The Qualification tool is used to look at a set of applications to determine if the RAPIDS Accelerator for Apache Spark
-might be a good fit for those applications.
-
-The Profiling tool generates information which can be used for debugging and profiling applications.
-Information such as Spark version, executor information, properties and so on. This runs on either CPU or
-GPU generated event logs.
+The Qualification and Profiling tools have been moved to
+[nvidia/spark-rapids-tools](https://github.com/NVIDIA/spark-rapids-tools) repo.
 
 Please refer to [Qualification tool documentation](docs/spark-qualification-tool.md)
 and [Profiling tool documentation](docs/spark-profiling-tool.md)
@@ -75,7 +73,7 @@ as a `provided` dependency.
 <dependency>
     <groupId>com.nvidia</groupId>
     <artifactId>rapids-4-spark_2.12</artifactId>
-    <version>22.12.0</version>
+    <version>23.02.0</version>
     <scope>provided</scope>
 </dependency>
 ```