A Kue tutorial using MEAN.io - Part 2

Posted by Cayle Sharrock on April 3, 2015

In Part 1, I set up the mean.io app and tweaked a few things. Now, we can dive into some coding hopefully.

The imageProcessor package

The idea is to create a little image processing app that takes a batch of files in the front end and then processes them (using ImageMagick) on the back-end to resize them, or whatever.

The main thrust here is to

  • Figure out how to use Mean.io
  • Create a package in mean.io
  • Use kue to do the job processing (using redis for job queuing)
  • Use MongoDB to handle the large data handling

Create the package

Mean.io makes this pretty simple

mean package imageProcessor

Voila! A bunch of files are already set up for you in packages/custom/imageProcessor

Defining the API

URL Method Description
/jobs GET List all jobs assigned to logged in user, along with their status, and if they’re complete, with a link to be able to download the results
/jobs/new POST Submit a new job with attached data
/job/:id GET DELETE Get a detailed summary & status of indicated job; delete job and data
/job/:id/results GET Retrieve the results for job :id
/job/:id/cancel DELETE Cancel a (presumably) running job. Has no effect otherwise

Database design

Based on the above API, I need to define schemas for the job queue and imageProcessor Models.

The Redis job queue will carry only the following information:

Field Type Description
user user name  
timeJobStarted timeStamp  
jobComplete boolean True if job is complete and results are ready
jobFailed int The number of times the job has failed
data object if no large data is required, additional data can be stored here
mongoId string Link to the MongoDoc for extra infos

The JobId is a timestamp String followed by a random 8 digit code.

The job type will always be ‘imageprocessor’ for this demo.

Each job type will have its own Schema associated with it. For this demo, the MongoDB entries for the ImageProcessor job type will have the following schema:

Field Type Description
_id   Autogenerated
user The user  
command string the type of imageMagick command (resize, etc)
command_parms string the parameters to pass to imageMagick
image blob the actual image data
results blob the results of the command

This is implemented as a Mongoose schema in the models folder of the custom package.

I also added some validations on the parameter value that depends on the actual command issued. This requires us to add a hook after basic validation (so that the values are available) but before the document is saved.

ImageProcessorSchema.pre('save', function(next, doc) {
    var err = null;
    var command = doc.command;
    var parms = doc.command_parms;
    switch (command) {
        case 'resize':
            //Param must be of form nnnxnnn
            err = /^\d+x\d+$/.test(parms)? null : new Error("Invalid image size specifications: " + parms);
        case 'convert':
            err = /^(png|jpg|gif)$/.test("png")? null : new Error("Invalid image format specification: " + parms);
            err = new Error("Invalid command: " + command);

Note the pretty neat error handling pattern. Whether or not an error condition exists, you call next with err (it’s null, or contains an Error object) and if there is an error, it will get handled in an appropriate place.