Use the bucket listing

[dennari]

Hi! I've been running into some performance issues that have to do with making a separate headObject request for every file. Why not use the GET Bucket (List Objects) command and get the ETags for 1000 files at a time?

[pgherveou]

Have you considered

• using the cache so you only have these perf issues he first time you run your command
• upload files in parallel https://github.com/pgherveou/gulp-awspublish#upload-file-in-parallel

That's sound a bit harder to develop, but if that does not work for you, you can give it a try

[dennari]

Well, the cache works really well in a local setup, but in a shared CI environment it's harder to set up. Parallelization would probably help to a degree, but I don't think its a solution that scales. Here's an outline to a solution that does scale: Getting Size and File Count of a 25 Million Object S3 Bucket. Is there some reason you wouldn't like to use the LIST API calls?

[pgherveou]

I think the current solution works well for most of the use cases

Concerning your use case what I think what would be best, would be:

• to add an additional method that let you populate the cache
• to add an an option to by pass the header requests

So to solve your problem you could then

• create a transform stream or call an async function that feed the cache
• call publish

it could look like this

  var publisher = awspublish.create({ bucket: '...' });
  return gulp
     .src('./public/*.js')
     .pipe(publisher.populateCache())
     .pipe(publisher.publish())
     .pipe(awspublish.reporter());

What do you think?

[alexgorbatchev]

@dennari curious, were you able to find a suitable solution?

[cedricve]

Is there any news about the feature request?