Poor Man’s Parallelization for Batch Processing Jobs

One common problem that I’ve seen time and time again with batch processing jobs (generally cronjobs) is that when they’re written they run quickly. Over time their workload grows until eventually it’s unacceptably slow. Most of simple jobs are single threaded and follow the pattern:

  • Query a database to get a list of objects to work on (users, pages, customers, etc)
  • Process or derive some data for each user

As an example let’s use a simple script that just iterates over a list of users.


$userIds = range(1,10);

foreach ($userIds as $userId) {
    echo "working on userId=$userId". PHP_EOL;

If you run this script it’ll just output:

working on userId=1
working on userId=2
working on userId=3
working on userId=10

To create a simple of way of separating the jobs in a consistent way we can use the modulus operator. It just calculates the remainder of two numbers. It’s a common arithmetic operator in almost all languages so this technique is pretty portable.

Let’s allow two arguments to be passed into the code. We want to define the number of different jobs to run in parallel and also which of the two this one is. Here’s a simple way of doing it:


$job = (int)$argv[1];
$jobs = (int)$argv[2];

echo "running job $job of $jobs". PHP_EOL;

$userIds = range(1,10);

foreach ($userIds as $userId) {

    if ($userId % $jobs != $job - 1) {
    echo "working on userId=$userId". PHP_EOL;

Running this on the command-line like php myScript.php 1 2 will output:

running job 1 of 2
working on userId=2
working on userId=4
working on userId=6
working on userId=8
working on userId=10

…and running with php myScript.php 2 2

running job 2 of 2
working on userId=1
working on userId=3
working on userId=5
working on userId=7
working on userId=9

Nice! Now we can run the even and odd jobs separately and at the same time.

Have a play with the script and try to run it split three ways with the options: "1 3", "2 3" and "3 3"

You can use the modulus operator in SQL queries too which could be useful if you’re pulling your list of Ids from the database.

SELECT userId FROM users WHERE userId % 2 = 1;

It’s worth noting that this hack won’t always work. You need to be able to identify each iteration of your loop with some unique integer and have sufficient free resources on your machine(s) to run the script in parallel. In some cases the proper way to go is a full on async setup using something like Gearman but if you’re in a hurry or the code is trivial this is a great little five minute fix.


Node.js 20 line proxy server ported to 15 lines of CoffeeScript

I only recently discovered CoffeeScript which seems like it could be a nicer way of writing simple JavaScript code.

After seeing, and being impressed by A HTTP Proxy Server in 20 Lines of node.js Code around a year ago I thought it might be a suitable piece of code to try porting to CoffeeScript. More complicated than the “Hello World” on the Node.js homepage but still fairly simple.

http = require 'http';

http.createServer( (request, response) ->

  proxy = http.createClient 80, request.headers['host']
  proxy_request = proxy.request request.method, request.url, request.headers;

  proxy_request.addListener 'response', (proxy_response) ->

    proxy_response.addListener 'data', (chunk) ->
      response.write chunk, 'binary'

    proxy_response.addListener 'end', -> 

    response.writeHead proxy_response.statusCode, proxy_response.headers

  request.addListener 'data', (chunk) ->
    proxy_request.write chunk, 'binary'

  request.addListener 'end', ->


The port was incredibly simple, just some search/replace type modification and the five line saving is only on lines closing functions with });. I’m not entirely convinced that it’s nicer to read for the moment but maybe it’s because I’m not very used to it yet. This code doesn’t exactly use many of the nice tricks that it can do.

I’ll definitely give CoffeeScript a go for with a small project in the future to evaluate it but for the moment the jury’s out.

Working with Date and Time in PHP

A lot of people ask questions relating to date and time in PHP. Here are some answers to the most commonly asked questions and common mistakes.

Procedural vs OO

Historically PHP provided a set of commands that allowed easy parsing of dates/times into UNIX timestamps (mktime(), strtotime(), etc) and then some tools to manipulate these timestamps and return them as formatted strings (date(), gmdate(), etc).

PHP5 introduced the DateTime class but lots of users still seem to be using the old procedural classes. I’ll try to demonstrate both approaches for each example and hope to win you over that DateTime is far easier to use and more powerful.


strtotime() is probably the simplest way takes a date as a string and attempts to convert it to a unix timestamp. Here are some examples of using strtotime()

A lot of people seem to try to use mktime() and gmmktime() to create dates from data they’ve fetched from a user or from the database. In most cases you can use strtotime() which very powerful and far simpler to use.

using mktime()

$dateString = '2011-05-01 09:22:34';
if (preg_match('/^(\d{4})\-(\d{2})\-(\d{2}) (\d{2}):(\d{2}):(\d{2})$/', $dateString, $a)) {
    $t = mktime($a[4], $a[5], $a[6], $a[2], $a[3], $a[1]);
    echo date('r', $t) . PHP_EOL; // returns: Sun, 01 May 2011 09:22:34 +0100

using strtotime()

$dateString = '2011-05-01 09:22:34';
$t = strtotime($dateString);
echo date('r', $t) . PHP_EOL; // returns: Sun, 01 May 2011 09:22:34 +0100

As you can see it’s far easier to use strtotime() to do this. The DateTime class wraps this functionality too:

using DateTime

$dateString = '2011-05-01 09:22:34';
$dt = new DateTime($dateString);
echo $dt->format('r') . PHP_EOL; // returns: Sun, 01 May 2011 09:22:34 +0100


If you tried out the examples above and got a warning that a default timezone isn’t set then you should set one using either date_default_timezone_set() at runtime or by defining date.timezone in your php.ini file. There’s a full list of timezones here

Setting the timezone at runtime


Formatting/returning dates

As a UNIX timestamp isn’t human readable you’ll need to convert it back out to something readable, you can format it into just about any style you want using this list of format strings. If you ever need to find it in a hurry just search on Google for “php date“. The procedural function date() is wrapped by the format() method in the DateTime class.

$dateString = '2011-05-01 09:22:34';
$t = strtotime($dateString);
echo date('l jS \of F, Y', $t) . PHP_EOL; // returns: Sunday 1st of May, 2011
$dateString = '2011-05-01 09:22:34';
$dt = new DateTime($dateString);
echo $dt->format('l jS \of F, Y') . PHP_EOL; // returns: Sunday 1st of May, 2011

Modifying dates

strtotime() is really powerful and allows you to modify/transform dates easily with it’s relative expressions too:


$dateString = '2011-05-01 09:22:34';
$t = strtotime($dateString);
$t2 = strtotime('-3 days', $t);
echo date('r', $t2) . PHP_EOL; // returns: Thu, 28 Apr 2011 09:22:34 +0100


$dateString = '2011-05-01 09:22:34';
$dt = new DateTime($dateString);
$dt->modify('-3 days');
echo $dt->format('r') . PHP_EOL; // returns: Thu, 28 Apr 2011 09:22:34 +0100

The stuff you can throw at strtotime() is quite surprising and very human readable. Have a look at this example looking for Tuesday next week.


$t = strtotime("Tuesday next week");
echo date('r', $t) . PHP_EOL; // returns: Tue, 10 May 2011 00:00:00 +0100


$dt = new DateTime("Tuesday next week");
echo $dt->format('r') . PHP_EOL; // returns: Tue, 10 May 2011 00:00:00 +0100

Note that these examples above are being returned relative to the time now.

The full list of time formats that strtotime() and the DateTime constructor takes are listed on the PHP Supported Date and Time Formats page.

Hopefully by now I’ve won you over that using the DateTime object is easier/cleaner than using the procedural functions. I’m going to follow by solving some common problems with the DateTime object and some other friends of its.

Converting time between timezones

Displaying a time in multiple time zones

$dt = new DateTime('1 May 2011 14:15', new DateTimeZone('America/Los_Angeles'));
// returns: LA:	Sun, 01 May 2011 14:15:00 -0700
echo "LA:\t". $dt->format('r') . PHP_EOL;

$dt->setTimeZone(new DateTimeZone('Europe/London'));
// returns: London:	Sun, 01 May 2011 22:15:00 +0100
echo "London:\t". $dt->format('r') . PHP_EOL;

Here we use the DateTimeZone class to convert a known time in a known timezone to a different timezone. This could allow users to select their timezone in their profile and then display all times from your application in that time zone. You’d need to be storing all of the dates in the database in the same timezone (e.g. GMT) though.

Displaying the difference between two times

This could be handy for stuff like showing a countdown to an event on your website (e.g. an offer expiring). PHP’s built-in DateInterval class makes this really simple.

// returns 25th December of this year
$offerEnds = new DateTime('25th December');
$now = new DateTime();

$diff = $offerEnds->diff($now);

// using the public properties
echo "It's only {$diff->days} days until Christmas!" . PHP_EOL;

// using the format string
echo $diff->format("%m months, %d days, %h hours and %i minutes till Christmas") . PHP_EOL;

The first echo uses the public properties of the DateDiff class and the second one uses the format strings and format() method.

Simple handling of date/time from databases

Converting DateTime objects into MySQL date format

class MySQLDateTime extends DateTime
    public function __toString()
        return $this->format('Y-m-d H:i:s');

$now = new MySQLDateTime();

$sql = "SELECT foo, bar FROM myTable WHERE someDate<'$now';";
echo $sql . PHP_EOL;

Converting dates from MySQL into DateTime objects

while (list($id, $date) = mysql_fetch_row($r)) {
    $dateObj = new DateTime($date);


These were just a few simple examples/tricks with date/time handling which demonstrate just how easy it is to work with dates/times using these classes. Hopefully this means an end to the days of working in UNIX timestamps by subtracting seconds, crudely manipulating time differences, etc.

If you have any other specific, common use cases that you’d like included in the article just add a comment to the post and I’ll try to add them.