Using Gulp – packaging files by folder

GulpJS is a great Node-based build system following in the footsteps of Grunt but with (in my opinion) a much simpler and more intuitive syntax. Gulp takes advantage of the streaming feature of NodeJs which is incredibly powerful, but means in order for you to get the most out of Gulp, you certainly need some understanding of what is going on underneath the covers.

As I was getting started with Gulp, I had a set of folders, and wanted to minify some JS files grouped by folder. For instance:

/scripts
/scripts/jquery/*.js
/scripts/angularjs/*.js

and want to end up with

/scripts
/scripts/jquery.min.js
/scripts/angularjs.min.js

and so on. This wasn’t immediately obvious at the time (I’ve now contributed this example back to the recipes), as it requires some knowledge of working with underlying streams.

To start with, I had something like this:

var gulp = require('gulp');
var concat = require('gulp-concat');
var rename = require('gulp-rename');
var uglify = require('gulp-uglify');

var scriptsPath = './src/scripts/';

gulp.task('scripts', function() {
    return gulp.src(path.join(scriptsPath, 'jquery', '*.js'))
      .pipe(concat('jquery.all.js'))
      .pipe(gulp.dest(scriptsPath))
      .pipe(uglify())
      .pipe(rename('jquery.min.js'))
      .pipe(gulp.dest(scriptsPath));
});

Which gets all the JS files in the /scripts/jquery/ folder, concatenates them, saves them to a /scripts/jquery.all.js file, then minifies them, and saves it to a /scripts/jquery.min.js file.

Simple, but how can we do this for multiple folders without manually modifying our gulpfile.js each time? Firstly, we need a function to get the folders in a directory. Not pretty, but easy enough:

function getFolders(dir){
    return fs.readdirSync(dir)
      .filter(function(file){
        return fs.statSync(path.join(dir, file)).isDirectory();
      });
}

This is JavaScript after all, so we can use the map function to iterate over these.


   var tasks = folders.map(function(folder) {

The final part of the equation is creating the same streams as before. Gulp expects us to return the stream/promise from the task, so if we’re going to do this for each folder, then we need a way to combine these. The concat function in the event-stream package will combine streams for us, and end only once all it’s combined streams have completed:

var es = require('event-stream');
...
return es.concat(stream1, stream2, stream3);

The catch is it expects streams to be listed explicitly in it’s arguments list. If we’re using map then we’ll end up with an array, so we can use the JavaScript apply function :

return es.concat.apply(null, myStreamsInAnArray);

Putting this all together, and we get the following:

var fs = require('fs');
var path = require('path');
var es = require('event-stream');
var gulp = require('gulp');
var concat = require('gulp-concat');
var rename = require('gulp-rename');
var uglify = require('gulp-uglify');

var scriptsPath = './src/scripts/';

function getFolders(dir){
    return fs.readdirSync(dir)
      .filter(function(file){
        return fs.statSync(path.join(dir, file)).isDirectory();
      });
}

gulp.task('scripts', function() {
   var folders = getFolders(scriptsPath);

   var tasks = folders.map(function(folder) {
      return gulp.src(path.join(scriptsPath, folder, '/*.js'))
        .pipe(concat(folder + '.js'))
        .pipe(gulp.dest(scriptsPath))
        .pipe(uglify())
        .pipe(rename(folder + '.min.js'))
        .pipe(gulp.dest(scriptsPath));
   });

   return es.concat.apply(null, tasks);
});

Hope this helps someone!

8 thoughts on “Using Gulp – packaging files by folder”

  1. This was super helpful and exactly what I was looking for 🙂

    I’m building new forum software, and there are theme folders, and all CSS files in a theme folder are to be concatenated and placed in a /public/[themeName]/styles.css file. But I don’t know the theme names in advance. Being new to Node.js and its API, I found your code that iterates over folders and builds streams dynamically to be very useful.

    (Minor note: I would have renamed your `var tasks` to `var streams` since its contents is streams not tasks :-))

  2. if you want to know when all these operations are done, you can use something like this:


    var folders = getFolders('myFolder'),
    folderCount = folders.length;

    function doneStreams() {
    folderCount--;

    if (folderCount === 0) {
    done();
    }
    }

    // add dist structure and nocache-file to each theme folder
    var tasks = folders.map(function(folder) {
    gulp.src(path.join(path, folder, '*'))
    ...
    .pipe(gulp.dest('dest'))
    .on('end', doneStreams);
    ;
    ;
    });

    without this gulp didn’t recognize when the task was really finished.

  3. As KajMagnus mentioned, this has been very helpful, as I needed to segregate my asset files depending on the packages structure.

  4. Thanks so much for sharing this! Exactly what I was looking for.

    You should write your own gulp plugin with the code above – very useful.

    I made some minor changes due to my folder structure containing an extra level before the js file. I used it for concatenating my Angular JS views in each module separately after using ngHtml2Js.

    Cheers,
    Saz

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.