Duplicate SBOM packages in a file scan #7795

goneall · 2024-10-25T03:55:04Z

goneall
Oct 25, 2024

Description

This is similar to #6204, but perhaps different.

In scanning a large repo as a file system with SPDX as the output format, multiple packages with the same SPDX-ID are present in the output.

From a cursory look, it appears that these are multiple entries for exactly the same package.

I suspect that, similar to #6204 the same package shows up in more than one file reference (e.g. as the same Python dependency reference in two separate requirements.txt files).

Desired Behavior

For SPDX SBOMs the package SPDX ID's must be unique within the same SPDX document - duplicates are not allowed.

If the duplicate SPDX ID represents the exact same package, the duplicate should just be removed.

If the duplicate SPDX ID represents a different package, a different SPDX ID should be generated.

Actual Behavior

Duplicate SPDX IDs are found.

For example, in the attached MaterialX-2024-08-21-trivy-spdx.json you will find:

{
      "name": "@jridgewell/gen-mapping",
      "SPDXID": "SPDXRef-Package-e5e18bf88fdfc531",
      "versionInfo": "0.3.5",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: MaterialX/javascript/MaterialXView/package-lock.json",
      "licenseConcluded": "NONE",
      "licenseDeclared": "NONE",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:npm/%40jridgewell/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: @jridgewell/[email protected]",
        "PkgType: npm"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },

as well as:

{
      "name": "@jridgewell/gen-mapping",
      "SPDXID": "SPDXRef-Package-e5e18bf88fdfc531",
      "versionInfo": "0.3.5",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: MaterialX/javascript/MaterialXView/package-lock.json",
      "licenseConcluded": "NONE",
      "licenseDeclared": "NONE",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:npm/%40jridgewell/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: @jridgewell/[email protected]",
        "PkgType: npm"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },

Reproduction Steps

1. Create a directory (root of the file system to scan)
2. Clone the repositories https://github.com/AcademySoftwareFoundation/MaterialX, https://github.com/AcademySoftwareFoundation/OpenPBR and https://github.com/AcademySoftwareFoundation/materialx-website creating 3 subdirectories
3. Execute the command `trivy fs --timeout 120m --scanners license,vuln --format spdx-json dir`
4. The generated file will contain duplicate SPDX IDs

Target

Filesystem

Scanner

License

Output Format

SPDX

Mode

Standalone

Debug Output

Please ping me if you need the debug output - it is quite large

Operating System

Ubuntu

Version

Created from Git, tag v0.55.2

Checklist

Run trivy clean --all
Read the troubleshooting

DmitriyLewen · 2024-10-28T11:58:11Z

DmitriyLewen
Oct 28, 2024
Maintainer

Hello @goneall
Thanks for your report!

I can't reproduce your case (spdx file deson't contain duplicates).
I even checked not the latest Trivy:

➜ docker run -it --rm --entrypoint /bin/sh aquasec/trivy:0.55.2
/ # mkdir dir && cd dir

/dir # git clone https://github.com/AcademySoftwareFoundation/MaterialX
Cloning into 'MaterialX'...
...
Resolving deltas: 100% (16626/16626), done.

/dir # git clone https://github.com/AcademySoftwareFoundation/OpenPBR
Cloning into 'OpenPBR'...
...
Resolving deltas: 100% (404/404), done.

/dir # git clone https://github.com/AcademySoftwareFoundation/materialx-website
Cloning into 'materialx-website'...
...
Updating files: 100% (1397/1397), done.

/dir # ls -hl
total 12K    
drwxr-xr-x   11 root     root        4.0K Oct 28 11:53 MaterialX
drwxr-xr-x    7 root     root        4.0K Oct 28 11:54 OpenPBR
drwxr-xr-x    8 root     root        4.0K Oct 28 11:54 materialx-website

/dir # trivy fs --timeout 120m --scanners license,vuln --format spdx-json . |  g
rep '"name": "@jridgewell/gen-mapping"' -A 21
...
      "name": "@jridgewell/gen-mapping",
      "SPDXID": "SPDXRef-Package-717dff8c43fff5ed",
      "versionInfo": "0.3.5",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: MaterialX/javascript/MaterialXView/package-lock.json",
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:npm/%40jridgewell/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: @jridgewell/[email protected]",
        "PkgType: npm"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },

What did I miss?

0 replies

goneall · 2024-10-28T14:59:45Z

goneall
Oct 28, 2024
Author

@DmitriyLewen - Thanks for the quick review. I'll get the latest code and see if it duplicates. If it duplicates, I'll see if I can do some debugging and come up with a smaller reproduceable example.

1 reply

DmitriyLewen Oct 29, 2024
Maintainer

Thanks a lot!

goneall · 2024-10-30T03:56:16Z

goneall
Oct 30, 2024
Author

@DmitriyLewen I wasn't able to duplicate with the "smaller" file system referenced above, but I was able to duplicate it with a much larger file system scan. It took a couple hours to scan and it produced a 29MB SBOM containing 10,549 duplicate SPDX ID's.

The files analyzed are from the following repos followed by the git hashes for the version scanned from the CNCF KeyCloak project:

                            "kc-sig-fapi": "a7722e7918585396ce6dd8f5489aaea7740eb9da",
                            "keycloak": "0e717f735e1b17d2ef10fa7b4764b1dc495d4746",
                            "keycloak-benchmark": "5f7d404e81fd262f3686829f5285115958e27319",
                            "keycloak-community": "140f549093e817e994811306ae5334be6c19af35",
                            "keycloak-extension-poc": "64e3a0e90d26f52bfef43465d9d608ea14f4cede",
                            "keycloak-github-bot": "4e825b2941d60a499e0f4e6f91f224b4abe43e49",
                            "keycloak-k8s-resources": "02a2913e2ad9e3454844bd8f6bb0ebfb03765c0e",
                            "keycloak-misc": "dee033f2d6d6b5c3a6ce8eb84e285f7e5626dbf6",
                            "keycloak-nodejs-connect": "cee3608b35273d9e37f70fa9b5f24d8465bb9870",
                            "keycloak-playground": "6a7de121d58d9760523b8d80edb7c2efa5da1d86",
                            "keycloak-quickstarts": "66fc8d915dbcbed5abd0acb9b353aa3bef02b26a",
                            "keycloak-realm-operator": "f0b2224aee10685b49c48febf38ddfbbe2e6ba0b",
                            "keycloak-rest-api-description": "83557d81ec027a786f7cc467ad954982644fa8bc",
                            "libunix-dbus-java": "06e3d1e5987f96eccad530b59cd133dcb895ae01"

I'll go through the duplicates and see if I can create a smaller set of files to duplicate the problem so you don't have to run such a large (time consuming) scan.

1 reply

DmitriyLewen Oct 30, 2024
Maintainer

I'll go through the duplicates and see if I can create a smaller set of files to duplicate the problem so you don't have to run such a large (time consuming) scan.

It will be great. Thanks!

goneall · 2024-10-30T04:16:07Z

goneall
Oct 30, 2024
Author

Here are 3 duplicates from the output - I'm scanning just the integration-arquillian directory from the keycloak repo to see if it duplicates with a smaller set of files. It's still pretty large (around 20MB) and taking a long time to scan (> 20 minutes and still running). I'll update the discussion in about 8 hours (getting late my time).

   {
      "name": "aopalliance:aopalliance",
      "SPDXID": "SPDXRef-Package-7e1772d4f32ebab8",
      "versionInfo": "1.0",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: keycloak/testsuite/integration-arquillian/pom.xml",
      "licenseConcluded": "Unlicense",
      "licenseDeclared": "Unlicense",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:maven/aopalliance/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: aopalliance:aopalliance:1.0",
        "PkgType: pom"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },
    {
      "name": "aopalliance:aopalliance",
      "SPDXID": "SPDXRef-Package-7e1772d4f32ebab8",
      "versionInfo": "1.0",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: keycloak/testsuite/integration-arquillian/pom.xml",
      "licenseConcluded": "Unlicense",
      "licenseDeclared": "Unlicense",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:maven/aopalliance/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: aopalliance:aopalliance:1.0",
        "PkgType: pom"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },
    {
      "name": "aopalliance:aopalliance",
      "SPDXID": "SPDXRef-Package-7e1772d4f32ebab8",
      "versionInfo": "1.0",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: keycloak/testsuite/integration-arquillian/pom.xml",
      "licenseConcluded": "Unlicense",
      "licenseDeclared": "Unlicense",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:maven/aopalliance/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: aopalliance:aopalliance:1.0",
        "PkgType: pom"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },

0 replies

goneall · 2024-10-30T04:55:04Z

goneall
Oct 30, 2024
Author

integration-arquillian.zip

@DmitriyLewen The above file will duplicate the problem. I ran this on version v0.55.2 git hash 928c7c0 (I wasn't able to get the latest development code to run on my local machine).

Unzip the file and run the command trivy fs --timeout 220m --scanners license,vuln --format spdx-json integration-arquillian

Here's the resultant JSON file:
smalltrivy.zip

The file has 317 duplicates.

One example is:

{
      "name": "aopalliance:aopalliance",
      "SPDXID": "SPDXRef-Package-c0c2a989259b4e3a",
      "versionInfo": "1.0",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: pom.xml",
      "licenseConcluded": "Unlicense",
      "licenseDeclared": "Unlicense",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:maven/aopalliance/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: aopalliance:aopalliance:1.0",
        "PkgType: pom"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },
    {
      "name": "aopalliance:aopalliance",
      "SPDXID": "SPDXRef-Package-c0c2a989259b4e3a",
      "versionInfo": "1.0",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: pom.xml",
      "licenseConcluded": "Unlicense",
      "licenseDeclared": "Unlicense",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:maven/aopalliance/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: aopalliance:aopalliance:1.0",
        "PkgType: pom"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },
    {
      "name": "aopalliance:aopalliance",
      "SPDXID": "SPDXRef-Package-c0c2a989259b4e3a",
      "versionInfo": "1.0",
      "supplier": "NOASSERTION",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "sourceInfo": "package found in: pom.xml",
      "licenseConcluded": "Unlicense",
      "licenseDeclared": "Unlicense",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:maven/aopalliance/[email protected]"
        }
      ],
      "attributionTexts": [
        "PkgID: aopalliance:aopalliance:1.0",
        "PkgType: pom"
      ],
      "primaryPackagePurpose": "LIBRARY"
    },

1 reply

DmitriyLewen Oct 30, 2024
Maintainer

Thanks for this report!
I found reason this issue.

Created #7824

DmitriyLewen · 2024-10-30T10:11:07Z

DmitriyLewen
Oct 30, 2024
Maintainer

Created #7824

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Duplicate SBOM packages in a file scan #7795

{{title}}

Replies: 6 comments 3 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Duplicate SBOM packages in a file scan #7795

goneall Oct 25, 2024

Description

Desired Behavior

Actual Behavior

Reproduction Steps

Target

Scanner

Output Format

Mode

Debug Output

Operating System

Version

Checklist

Replies: 6 comments · 3 replies

DmitriyLewen Oct 28, 2024 Maintainer

goneall Oct 28, 2024 Author

DmitriyLewen Oct 29, 2024 Maintainer

goneall Oct 30, 2024 Author

DmitriyLewen Oct 30, 2024 Maintainer

goneall Oct 30, 2024 Author

goneall Oct 30, 2024 Author

DmitriyLewen Oct 30, 2024 Maintainer

DmitriyLewen Oct 30, 2024 Maintainer

goneall
Oct 25, 2024

Replies: 6 comments 3 replies

DmitriyLewen
Oct 28, 2024
Maintainer

goneall
Oct 28, 2024
Author

DmitriyLewen Oct 29, 2024
Maintainer

goneall
Oct 30, 2024
Author

DmitriyLewen Oct 30, 2024
Maintainer

goneall
Oct 30, 2024
Author

goneall
Oct 30, 2024
Author

DmitriyLewen Oct 30, 2024
Maintainer

DmitriyLewen
Oct 30, 2024
Maintainer