Building a site with Eleventy is pretty slick and relatively simple, but when you are ready to deploy your content you really need to do some cleanup, minfiication, etc., to ensure that you are using as little bandwidth and storage as possible. You could do that manually every time or build scripts to do that, but there is a better answer if you are somewhat familiar with Javascript: Gulp.
Gulp is a task runner that essentially is like a makefile utility. The tasks are defined in a file named gulpfile.js. Rather than bore the reader with a long, theoretical monologue, let me provide an example of a gulpfile that I just completed for the documentation site I built for Farmer Frog. This walkthrough assumes you are familiar with Javascript.
Installing Gulp
Installing Gulp is simple. Just use the following command to install the command line utility.
npm i --global gulp-cli
To add gulp to your project, use:
npm i --save-dev gulp
Okay, now let's walk through the gulpfile...
Module Declarations
The first part of the file are the includes. Gulp allows you to deconstruct its exported methods so that you are not burdened with having to type gulp.method.
As is typical, the first portion of gulpfile.js are the module declarations:
const {
src,
dest,
series,
parallel,
watch
} = require('gulp');
const cp = require("child_process");
const cssnano = require('gulp-cssnano');
const uglify = require('gulp-uglify');
const rename = require('gulp-rename');
const imagemin = require('gulp-imagemin');
const htmlmin = require('gulp-htmlmin');
const sitemap = require('gulp-sitemap');
const save = require('gulp-save');
const removeEmptyLines = require('gulp-remove-empty-lines');
const autoprefixer = require('gulp-autoprefixer');
const babel = require('gulp-babel');
const browserSync = require('browser-sync').create();
const glob = require ('glob');
const fs = require('fs');
const cheerio = require('cheerio');
const del = require('del');
Render Task
Following the module declarations, we come to the first gulp private task, render. The render task launches Eleventy in a child process and renders the site to the build folder[1] (the folder name is specified in the .eleventy.js file).
// Use Eleventy to generate the site in the 'build' folder
const render = () => {
return cp.spawn("npx", ["eleventy", "--quiet"], { shell: true,
stdio: "inherit"
});
};
HTML Minification Task
The next task minifies the HTML created by the render() task. This task provides an example of the use of Gulp's .pipe() function to insert processing steps. As this function is typical of a Gulp task, let's examine it more closely.
// process HTML files (minify)
const processHTML = () => {
return src('build/**/*.html')
.pipe(htmlmin({ collapseWhitespace: true }))
.pipe(dest('./dist'))
}
In the processHTML() task, we see that the locaiton of the input files are specified in the src arguments and a globstar argument ('**') is used to take in all files and directories under the source folder. Essentially the src method will loop over each file it finds and read the content into a stream that it passes on down the chain.
As each file is read, it is passed into the .pipe() function that calls the gulp plug-in for htmlmin and passes the collapseWhitespace argument to htmlmin.
The output of the .pipe() function is then passed to the dest function that specifies the top-level location to write the file. It is important to note that the full relative path of the file will be written. For example, if an HTML file is at ./build/page/index.html then the output file will be written to ./dist/page/index.html.
Create Site Map Task
The next function creates a sitemap for web crawlers.
// create SEO sitemap
const siteMap = () => {
return src('build/**/*.html', { read: false })
.pipe(save('before-sitemap'))
.pipe(sitemap({ siteUrl: 'http://farmerfrog.org'}))
.pipe(removeEmptyLines())
.pipe(dest('./dist'))
.pipe(save.restore('before-sitemap'))
}
the siteMap() task starts by setting up the src function, but in this case we do not want to read the contents and only want the file names so we set the argument read to false.
As each file is read, the state of the file is saved using the save() function so that it can be restored after all processing is complete. This may be an unnecessary step, to be honest, but it doesn't appear to impact performance so better safe than sorry. It and the restore() function may be removed in the future, but for now it's there.
After saving the state the stream essentially passed the filename to the gulp-sitemap plug-in and specifies the site URL to be prepended to the file name.
This output is then sent to the removeEmptyLines plug-in to remove any empty lines, a sort of minification process. A further optimization might be to remove line feeds but I am not sure at this time that it would bring much value due to the relatively small size of the site.
From there, the output is routed to the location specified in the dest function.
CSS Autoprefixer / Minification Task
The processCSS() task uses the gulp-cssnano and gulp-autoprefixer plugins to apply autoprefixer to the CSS before minifying it.
// process CSS files (autoprefix for cross-browser compatibility, minify)
const processCSS = () => {
return src('./src/**/*.css')
.pipe(autoprefixer())
.pipe(cssnano())
.pipe(dest('./dist'))
}
Javascript Babel / Minification Task
Like the processCSS() task, the processJavascript() task uses the same pattern src -> pipe(s) -> dest to run the Javascript files through the gulp-babel and gulp-uglify plug-ins to ensure compatibility with earlier versions of Javascript and to minify the output.
// process Javascript files (babel for cross-browser compatiblity, minify)
const processJavascript = () => {
return (src(['./src/js/**/*.js', '!./src/utilities/indexer.js']))
.pipe(babel({ presets: ["@babel/env"]}))
.pipe(uglify())
.pipe(dest('./dist/js'))
}
Image Minification Task
The optimizeIMages() task uses the gulp-imagemin plug-in to reduce the size of the images on the site. The imagemin library has several plug-ins of its own to address different image formats.
// optimize images (reduce image sizes)
const optimizeImages = () => {
return (src('./src/img/**/*'))
.pipe(imagemin([
imagemin.gifsicle({ interlaced: true }),
imagemin.mozjpeg({ quality: 50, progressive: true }),
imagemin.optipng({ optimizationLevel: 5 }),
imagemin.svgo( {
plugins: [
{ removeViewBox: true },
{ cleanupIDs: false}
]
})
]))
.pipe(dest('./dist/img'))
}
Building the Site Search Index
The Farmer Frog site uses the minisearch module to provide a full-text site search capability. The minisearch module uses a remarkably small search index that it builds when first invoked. However, that process relies on a pre-built array of JSON objects. There is not a gulp plug-in for minisearch at this time, but it is relatively simple to create a custom function and call it from the gulp task.
The task itself is straightforward:
// build the site search index
const buildSiteIndex = async () => {
await buildIndex();
}
Note the use of the asynch/await directives. These are required because all Gulp tasks are asynchronous. There is no need to minify the output from buildIndex() because the file is already "minifed" by the JSON stringify method. The buildIndex() function code is included in the gulpfile.js and appears as follows:
// Build the site index from the HTML files
const buildIndex = () => {
const jsonDocs = [];
const EXCLUDES = ['**/node_modules/**',
'**/categories/**',
'**/tags/**',
'**/docs/**',
'**/articles/**',
'**/authors/**'];
const OUTPUT_DIR = 'src/_data';
const INCLUDE_PATTERN = '**/*.html';
const myList = glob.sync(INCLUDE_PATTERN, {ignore: EXCLUDES});
// convert HTML documents into JSON documents for array
for (let i = 0; i < myList.length; i++) {
let indexObj = {};
// load the file into a string
let htmldoc = fs.readFileSync(myList[i], 'utf8');
let $ = cheerio.load(htmldoc);
// build the JSON document
indexObj.id = i;
indexObj.ref = myList[i].replace('dist', '');
let title = $('title');
indexObj.title = $('title').text();
indexObj.text = ' ';
// Concatenate the text from the paragraph tags
$('p').each(function(i, e) {
let str = $(this).text().trim().replace(/\s+/g, ' ');
if (str.length > 0) {
indexObj.text += str + ' ';
}
});
indexObj.text = indexObj.text.trim();
// add the object to the array of JSON docs
jsonDocs.push(indexObj);
}
// save the index to disk
fs.writeFile(OUTPUT_DIR + '/searchIndex.idx', JSON.stringify(jsonDocs), function(err) {
if (err) throw err;
console.log('Index saved.');
});
}
Copy Search Index
After the index file has been built, the copyIndexFile() task copies it from the build/data directory to the dist/data directory. This is one of the simplest tasks you can perform with Gulp.
// copy the search index
const copyIndexFile = () => {
return src(['./src/_data/**/*'])
.pipe(dest('./dist/_data'))
}
Copy Robot.txt files
Like the copyIndexFile() task, this task simply copies files.
// Move the robots.txt files
const copyRobotsText = () => {
return src(['./src/robots*.txt'])
.pipe(dest('./dist'))
}
Clean Tasks
It is generally a good practice to build from scratch. Why? Builds need to be reproducibly consistent. When one performs a "partial-build" there is the chance that something won't build correctly but the relics from a previous build will still be hanging around thereby complicating debugging. Your mileage may vary on this, but from my experience building from scratch ensures that I see the actual errors rather than ghost errors that occur from pre-existing artifacts.
The tasks are straightforward and easy to implement:
// clean the dist folder
const cleanDist = () => {
return(del('./dist/**/*'))
}
// clean the build folder
const cleanBuild = () => {
return(del('./build/**/*'))
}
Monitor Task
Gulp also provides the capability to spawn a web server to view the site and uses the browserSync module to update the page when a change occurs. However, this is generally less useful because the same functionality is provide via Eleventy. For the Farmer Frog project, Gulp is used to create the production distribution, not the ongoing development so the watch task is primarily used to preview the production build although changes made to the files in the build directory should also trickle up.
// watch for changes to files
const monitor = () => {
browserSync.init({
server: 'dist',
browser: 'Google Chrome'
})
watch([ './build/**/*' ])
}
Exported Tasks
All of the tasks discussed so far are private. Like all Javascript modules, anything that must be externally visible must be exported. Gulp supports the notion of a default task (named default) as well as individually named tasks.
An exported task may be assigned to a single function, as in the case of the monitor, clean_build, clean_dist, and clean_all tasks. Gulp also provides two functions, serial() and parallel(), that provide the capability to specify that tasks should be performed serially or in parallel despite all tasks in Gulp being asynchronous.
// define Gulp External Tasks
// build the dist folder contents
exports.default = series( cleanBuild,
cleanDist,
render,
buildSiteIndex,
copyIndexFile,
processHTML,
siteMap,
processCSS,
processJavascript,
optimizeImages,
copyRobotsText);
// Monitor the site in the dist folder
exports.monitor = monitor
// clear the contents of the build folder
exports.clean_build = cleanBuild;
// clear the contents of the dist folder
exports.clean_dist = cleanDist;
// clear the contents of the build and dist folders
exports.clean_all = parallel( cleanBuild, cleanDist)
As you can see, the default task is reponsible for creating the production build. To do so, it first cleans out the build and dist directories, then calls the render internal task to generate the site to the build directory before executing the remaind of the tasks. It is also important to note that default task conducts its tasks in series while the clean_all task executes its tasks in parallel.
Wrapping Up
I do not claim to be a master of Gulp nor Javascript and there are likely more improvements that can and will be made going forward. However, this is, in my opinion, a reasonably good start on using Gulp for build automation.
[1] The name build is used so that the developers are able to test their changes. Gulp will be used to generate the production deployment and creates the dist folder for that output. ↩