help testing with yarn #16

naugtur · 2019-08-05T19:20:38Z

I personally don't use yarn for any real projects at this time so it'd be helpful if someone with real life examples tested the support.

How to?

install npm-audit-resolver@next
try out both check-audit and resolve-audit commands on your yarn project. use --yarn flag to enable support
report back with thoughts and results.

Anyone willing to help?
@clement-escolano @MerlinDMC @danielabel @rouanw

The text was updated successfully, but these errors were encountered:

danielabel · 2019-08-05T19:30:51Z

I've weeded out the yarn from our services. I don't think we have any projects left.

I'll check and get back to you if i find any.

MerlinDMC · 2019-08-06T08:16:21Z

As @danielabel said we don't have yarn anymore. But I know Kibana is using it so I gave that a go with somewhat amazing results.

$ node_modules/.bin/check-audit --yarn
WARNING: yarn support is experimental
Creating package-lock.json from yarn.lock
>>>> npm audit --json
npm ERR! code ELOCKVERIFY
npm ERR! Errors were found in your package-lock.json, run  npm install  to fix them.
npm ERR!     Missing: @elastic/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: lodash@npm:@elastic/[email protected]
npm ERR!     Missing: @elastic/[email protected]
npm ERR!     Missing: @elastic/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]
npm ERR!     Missing: @kbn/[email protected]

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/daniel/.npm/_logs/2019-08-06T08_11_55_002Z-debug.log
>>>> exit: 1
Removing package-lock.json
TypeError: Cannot read property 'length' of undefined
    at pkgFacade.getAudit.then.input (/Users/daniel/src/forked/kibana/node_modules/npm-audit-resolver/check.js:22:45)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Essentially I think whatever converts the yarn.lock to package-lock.json is missing bits.

I also don't think it is needed... why don't you just run yarn audit --json and parse that output to get the vulnerabilities?
The format is line based so would need a slightly different parsing approach or a transform before you feed it into the other bits of the resolver.

But I would suggest not using Kibana to try that as the audit JSON output is 59GB in size ... and it isn't easy to parse on a normal workstation 😄

mtt87 · 2019-08-06T08:47:01Z

Did a quick test

npm i -g npm-audit-resolver@next
npx create-react-app test-with-yarn
cd test-with-yarn
check-audit --yarn

output

WARNING: yarn support is experimental
Creating package-lock.json from yarn.lock
>>>> npm audit --json 
>>>> exit: 1
Removing package-lock.json
Total of 4 actions to process
.... long list of stuff
--------------------------------------------------
 😱   Unresolved issues found!
--------------------------------------------------

If I run resolve-audit --yarn then I ge that long list and then prompted about what to do

 f) fix automatically
 d) show more details and ask me again
 r) remind me in 24h
 i) ignore paths
 del) Remove all listed dependency paths
 s) Skip this
 q) Quit
What would you like to do?

After I say f a few times then I get to the end

>>>> exit: 0

That said what happened was the creation of a file audit-resolve.json and pretty much nothing happened to yarn.lock or package.json

I also tried this in a monorepo (yarn workspaces) and that fails immediately

naugtur · 2019-08-06T09:01:36Z

Thanks!

So the converter was not my idea but seemed a good one - I checked yarn audit output and it's very different. I assumed the output format of npm audit is pretty much what the registry returns and I'm surprised yarn returned something else.

Maybe something else confused me. Anyway, looking into mapping it to what the rest of the core logic expects could be a better way out in this case.

As for kibana audit, It's larger than most of my disks partitions :p
With enough RAM there's no reason it wouldn't work though ;)

Does yarn audit work with yarn workspaces?

mtt87 · 2019-08-06T09:05:04Z

Does yarn audit work with yarn workspaces?

Yes if I run yarn audit at the root of the monorepo workspace it works.

naugtur · 2019-08-06T09:26:09Z

Another reason to try using yarn audit output instead.
Thanks!

Please share the yarn and package files you used if they're not secret. I'd like to make more e2e tests out of those.

mtt87 · 2019-08-06T09:27:40Z

For the monorepo workspace I used this
https://github.com/mtt87/ks-serverless-now

For a generic yarn project you can do

npx create-react-app my-yarn-project

😄

clement-escolano · 2019-08-06T14:24:21Z

Hello,

On some project, I reproduced the same error as the one reported by MerlinDMC: ELOCKVERIFY, the package-lock.json generated by synp seems to be incorrect.

The reason I used synp instead of parsing yarn audit --json output in the original PR is the following:

npm-audit-resolver offers a fix option for every found vulnerability. This option comes from the NPM audit report that gives recommendation to fix vulnerabilities. However, the yarn report does not gives this recommandation. To keep this feature, the only way is to use NPM for the report (hence the conversion from yarn.lock to package-lock.json).

If synp is not working correctly, the only solution seems to parse yarn audit --json report and not to provide the fix option for yarn audit. I worked on this solution a while back. The (WIP) commit for the feature is here.

If you agree with the strategy of this commit, I can work on a PR for the refactoring branch.

Cheers

naugtur · 2019-08-06T20:41:57Z

check out yarn-support-2.0 branch - I think it'd make more sense to start there.
I saw you adapted the output from yarn and it seems to be pretty much ready.

If you can PR the adapter to the branch, I could pull it in and figure out what to do with the fixing part. I could make it depend on some information and later we'd work with yarn people to get it addressed.

naugtur · 2019-08-06T21:01:44Z

Also, I postponed work on the investigate feature and it's not available now.
I'm thinking of making it something separate or at least put some boundaries in to make adding features to it easier.

Undistraction · 2019-09-18T22:18:27Z

I get the following with npm-audit-resolver@next

running npx audit-resolver --yarn:

resolve-audit --yarn
WARNING: yarn support is experimental
/Users/me/Sites/frontend/node_modules/npm-audit-resolver/src/pkgmanagers/yarn.js:31
            .finally(removePackageLockIfNecessary)
                    ^

TypeError: Promise.resolve(...).then(...).then(...).finally is not a function
    at Object.getAudit (/Users/me/Sites/frontend/node_modules/npm-audit-resolver/src/pkgmanagers/yarn.js:31:21)
    at Object.getAudit (/Users/me/Sites/frontend/node_modules/audit-resolve-core/pkgFacade/index.js:37:35)
    at Object.<anonymous> (/Users/me/Sites/frontend/node_modules/npm-audit-resolver/resolve.js:15:11)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Function.Module.runMain (module.js:694:10)
    at startup (bootstrap_node.js:204:16)
The script called "audit" which runs "resolve-audit --yarn" failed with exit code 1 https://github.com/sezna/nps/blob/v5.9.8/other/ERRORS_AND_WARNINGS.md#failed-with-exit-code

Running npx check-audit --yarn:

npx check-audit --yarn
WARNING: yarn support is experimental
Creating package-lock.json from yarn.lock
>>>> npm audit --json 
Promise.resolve(...).then(...).then(...).finally is not a function
>>>> exit: 0

rouanw · 2019-09-19T05:06:56Z

Hi @Undistraction. Thanks for reporting. Which version of Node are you using? Promose.prototype.finally is only supported from Node 10.

I remember @naugtur was keen on supporting older node versions, so they might be open to changing this code.

naugtur · 2019-09-19T08:43:28Z

Uh... Yes... Let's.
I don't think there's anyone interested in maintaining security of their dependencies while using v4 anymore, but v8 is still good.
I'll publish another prerelease today.

Thanks!

Undistraction · 2019-09-19T09:14:13Z

@naugtur, @rouanw thanks both. I'm on 18.13.0.

naugtur · 2019-09-19T09:48:56Z

published a new version (2.0.0-4) with a total rewrite of yarn support. please check if it works for you.

@Undistraction @rouanw @danielabel @MerlinDMC @mtt87 @clement-escolano

Undistraction · 2019-09-19T13:12:06Z

@naugtur Unfortunately (2.0.0-4):

npx resolve-audit --yarn
WARNING: yarn support is experimental
>>>> npm audit --json 
npm ERR! code EAUDITNOLOCK
npm ERR! audit Neither npm-shrinkwrap.json nor package-lock.json found: Cannot audit a project without a lockfile
npm ERR! audit Try creating one first with: npm i --package-lock-only

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/me/.npm/_logs/2019-09-19T13_08_52_978Z-debug.log
>>>> exit: 1
{ SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at Parser._handleLines (/Users/me/Sites/frontend/node_modules/jsonlines/lib/parser.js:24:19)
    at Parser._transform (/Users/me/Sites/frontend/node_modules/jsonlines/lib/parser.js:48:8)
    at Parser.Transform._read (_stream_transform.js:186:10)
    at Parser.Transform._write (_stream_transform.js:174:12)
    at doWrite (_stream_writable.js:396:12)
    at writeOrBuffer (_stream_writable.js:382:5)
    at Parser.Writable.write (_stream_writable.js:290:11)
    at promiseCommand.then.output (/Users/me/Sites/frontend/node_modules/npm-audit-resolver/src/me/yarn.js:52:24)
    at <anonymous> source: '{' }

And

npx check-audit --yarn
WARNING: yarn support is experimental
>>>> npm audit --json 
npm ERR! code EAUDITNOLOCK
npm ERR! audit Neither npm-shrinkwrap.json nor package-lock.json found: Cannot audit a project without a lockfile
npm ERR! audit Try creating one first with: npm i --package-lock-only

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/me/.npm/_logs/2019-09-19T13_10_40_524Z-debug.log
>>>> exit: 1
{ SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at Parser._handleLines (/Users/me/Sites/frontend/node_modules/jsonlines/lib/parser.js:24:19)
    at Parser._transform (/Users/me/Sites/frontend/node_modules/jsonlines/lib/parser.js:48:8)
    at Parser.Transform._read (_stream_transform.js:186:10)
    at Parser.Transform._write (_stream_transform.js:174:12)
    at doWrite (_stream_writable.js:396:12)
    at writeOrBuffer (_stream_writable.js:382:5)
    at Parser.Writable.write (_stream_writable.js:290:11)
    at promiseCommand.then.output (/Users/me/Sites/frontend/node_modules/npm-audit-resolver/src/pkgmanagers/yarn.js:52:24)
    at <anonymous> source: '{' }

naugtur · 2019-09-19T15:56:33Z

Looks like it didn't even switch to yarn and the --yarn flag didn't work.
That's why testing in isolation is lame. I'll add an end to end test to fix this. Check back on Monday

naugtur · 2019-09-22T21:06:45Z

published 2.0.0-5 and it should work fine now

Undistraction · 2019-09-23T15:29:13Z

@naugtur That's working great for me. Thanks. Feel free to ping me if you need anything testing in yarn in the future.

naugtur · 2019-09-23T18:10:11Z

the fix option for yarn is not implemented correctly so I'll have more stuff to test.
Let me know if you notice anything wrong. Feel free to report regular bugs

naugtur · 2019-09-24T21:25:32Z

v2.0.0 is out

Undistraction · 2020-04-21T09:48:14Z

v2.2.0
[email protected]
[email protected]
OSX

@naugtur I'm now seeing this hanging. There is the following output, then nothing:

WARNING: yarn support is experimental
>>>> yarn audit --json

I've left this for 15 mins and there is still no output.

Running yarn audit works fine.

naugtur · 2022-02-26T21:22:24Z

Hi everyone.
A lot changed since. Anyone interested in testing npm-audit-resolver@next? Version 3 is getting closer to releasable.
Yarn 1 seems to work. Not sure about anything else at this point TBH.

snikoyo · 2022-03-11T14:39:40Z

Just a little feedback for yarn: For a middle-sized vue-based project this has worked smoothly with yarn version 1.22.5, even in the gitlab CI pipeline.

Only the global install does not work - yarn global add npm-audit-resolver will install without errors but the two commands resolve-audit and check-audit are not there afterwards.

So our gitlab CI looks like this:

    - yarn install --ignore-scripts
    - npm install -g npm-audit-resolver
    - check-audit --yarn

naugtur · 2022-03-13T16:56:21Z

you could install in a container and use that as the CI base.
Which version have you tested? v3 (3.0.0-7 precisely) is very close to release.

snikoyo · 2022-03-15T15:39:02Z

We didn't specify the version so it should be 2.3.1. It also didn't work locally on my computer to install via yarn add global.

naugtur added the help wanted Extra attention is needed label Aug 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

help testing with yarn #16

help testing with yarn #16

naugtur commented Aug 5, 2019

danielabel commented Aug 5, 2019

MerlinDMC commented Aug 6, 2019

mtt87 commented Aug 6, 2019

naugtur commented Aug 6, 2019

mtt87 commented Aug 6, 2019

naugtur commented Aug 6, 2019

mtt87 commented Aug 6, 2019

clement-escolano commented Aug 6, 2019

naugtur commented Aug 6, 2019

naugtur commented Aug 6, 2019

Undistraction commented Sep 18, 2019 •

edited

Loading

rouanw commented Sep 19, 2019

naugtur commented Sep 19, 2019

Undistraction commented Sep 19, 2019

naugtur commented Sep 19, 2019

Undistraction commented Sep 19, 2019 •

edited

Loading

naugtur commented Sep 19, 2019

naugtur commented Sep 22, 2019

Undistraction commented Sep 23, 2019

naugtur commented Sep 23, 2019

naugtur commented Sep 24, 2019

Undistraction commented Apr 21, 2020 •

edited

Loading

naugtur commented Feb 26, 2022

snikoyo commented Mar 11, 2022 •

edited

Loading

naugtur commented Mar 13, 2022

snikoyo commented Mar 15, 2022

help testing with yarn #16

help testing with yarn #16

Comments

naugtur commented Aug 5, 2019

danielabel commented Aug 5, 2019

MerlinDMC commented Aug 6, 2019

mtt87 commented Aug 6, 2019

naugtur commented Aug 6, 2019

mtt87 commented Aug 6, 2019

naugtur commented Aug 6, 2019

mtt87 commented Aug 6, 2019

clement-escolano commented Aug 6, 2019

naugtur commented Aug 6, 2019

naugtur commented Aug 6, 2019

Undistraction commented Sep 18, 2019 • edited Loading

rouanw commented Sep 19, 2019

naugtur commented Sep 19, 2019

Undistraction commented Sep 19, 2019

naugtur commented Sep 19, 2019

Undistraction commented Sep 19, 2019 • edited Loading

naugtur commented Sep 19, 2019

naugtur commented Sep 22, 2019

Undistraction commented Sep 23, 2019

naugtur commented Sep 23, 2019

naugtur commented Sep 24, 2019

Undistraction commented Apr 21, 2020 • edited Loading

naugtur commented Feb 26, 2022

snikoyo commented Mar 11, 2022 • edited Loading

naugtur commented Mar 13, 2022

snikoyo commented Mar 15, 2022

Undistraction commented Sep 18, 2019 •

edited

Loading

Undistraction commented Sep 19, 2019 •

edited

Loading

Undistraction commented Apr 21, 2020 •

edited

Loading

snikoyo commented Mar 11, 2022 •

edited

Loading