Testing high concurrency http server
Apr 17, 2015

I was making some research for one of upcoming project at my job. I needed to test http server if it can handle 500-1500 concurrent requests.

It's was more hard to make right testing rather then right server. Web server don't have any record in error log and could process requests that I sent with curl while load test is running.

Setup

First I setup Nginx with HttpEchoModule, I used openresty to easily configure and compile with extra modules.

Next I create config to wait 3 seconds and response with some text.

I choose nginx as example to compare with real application.

location / {
    add_header Content-Disposition "inline";
    add_header Content-Type "text/plain";

    echo_sleep 3;
    echo "response text";
}

Testing

Then I start to test how many requests it can process at same time:

ab -n 100 -c 100 http://127.0.0.1:7080/
# => ok
ab -n 200 -c 200 http://127.0.0.1:7080/
# => ok
ab -n 500 -c 500 http://127.0.0.1:7080/
# => socket: Too many open files (24)

Probably we hit system limitation on max open files per process.

Read more

Synchronous Javascript with fibers
Mar 2, 2015

I was suprised how nice to get rid of endless nested callbacks in javascript.

I was playing with node-fibers, trying to make it easier to write tests.. One day I came up with idea: what if one function can run asynchronously when we call inside fiber and synchronous when passing a callback function. Here's how I can make it:


var Fiber = require('fibers');

// can be called to convert multiple methods
Fiber.makeSync = function (receiver) {
  for (var n = 1; n < arguments.length; n++) {
    Fiber.makeSyncFn(receiver, arguments[n]);
  }
};

Fiber.makeSyncFn = function(receiver, methodName, errorArgNum) {
  var origFn = receiver[methodName];

  if (origFn == undefined) {
    throw "Object don't have property '" + methodName + "'";
  }

  receiver[methodName] = function () {
    var lastArg = arguments[arguments.length - 1];

    // check if it called inside Fiber
    if (Fiber.current && typeof lastArg != 'function') {
      var fiber = Fiber.current;
      var newValue;
      var args = Array.prototype.slice.call(arguments);
      if (typeof errorArgNum == 'undefined') errorArgNum = 1;
      args.push(function(data) {
        // retrieve error from arguments (optional)
        var error = arguments[errorArgNum];
        if (error) {
          throw error;
        }
        // assign result and resume fiber
        newValue = errorArgNum == 0 ? arguments[1] : data;
        fiber.run();
      });

      // call original function with fiber-aware callback
      origFn.apply(this, args);
      // pause and wait till resume
      Fiber.yield();
      return newValue;
    } else {
      origFn.apply(this, arguments);
    }
  };
};

module.exports = Fiber;

Read more

Convert mysql database to postgresql
Jan 29, 2015

Today I find tool pgloader http://pgloader.io/ to convert database from Mysql to Postgres.

Precompiled package can be installed from downloads page or you can compile by yourself from lisp source on github.

Then run it as:

psql -c "CREATE DATABASE db_name"
pgloader mysql://user:pass@localhost/db_name postgres:///db_name

It shows warning that can't find libsybdb.dylib, press 0 to continue and it working well.

At the end you will see statistics table:

                    table name       read   imported     errors            time
------------------------------  ---------  ---------  ---------  --------------
               fetch meta data         18         18          0         18.096s
                  create, drop          0         14          0          0.285s
------------------------------  ---------  ---------  ---------  --------------
                   attachments          0          0          0          0.154s
                        emails       1782       1782          0          2.425s
                     email_bcc         20         20          0          0.044s
                      email_cc          0          0          0          0.045s
                      email_to       1847       1847          0          0.242s
            http_notifications       1097       1097          0          0.317s
                     templates          6          6          0          0.171s
        Index Build Completion          0          0          0          0.000s
------------------------------  ---------  ---------  ---------  --------------
                Create Indexes         11         11          0          0.237s
               Reset Sequences          0          0          0          0.361s
                  Primary Keys          7          7          0          0.049s
                  Foreign Keys          4          4          0          0.071s
                      Comments          0          0          0          0.000s
------------------------------  ---------  ---------  ---------  --------------
             Total import time       4752       4752          0         22.260s

Done!

Validating hash params in ruby
Jan 27, 2015

If compare many arguments vs “options” hash, I usually choose options hash. And I often face problems when I pass wrong key name or what not supported.

ActiveSupport already have Hash#assertvalidkeys, I added one more method assert_required_keys:

class Hash
  def assert_required_keys(*keys)
    keys.flatten.each do |key|
      raise ArgumentError.new("Required key: #{key.inspect}") unless has_key?(key)
    end
  end
end

Here is my solution to validate hash keys:

module ValidateOptions
  def validate_options!(params, options)
    options[:optional] ||= []
    options[:required] ||= []

    params = params.deep_symbolize_keys
    params.assert_required_keys(options[:required])
    params.assert_valid_keys(options[:required] + options[:optional])
  end
end

Usage example:

include ValidateOptions

def method_with_options(options = {})
  validate_options!(options,
    required: [:order_id, :subject, :from, :to, :text_body, :html_body],
    optional: [:cc, :bcc, :template_name]
  )
end

Handle exceptions in D
Jan 14, 2015

Here's how to handle exceptions in D.

try {
    throw new Exception("error message");
} catch (Exception error) {
    writefln("Error catched: %s", error.msg);
} finally {
    writefln("in finaly block");
}

catch (Exception error) will catch all exception of type Exception and child (inhereted) types.

Read more

Anonymous functions in D
Jan 13, 2015

We can use anonymous functions in D.

delegate keyword

Example with argument:

import std.stdio;

void evenNumbers(int[] numbers, void delegate(int) callback) {
    foreach (int number; numbers) {
        if (number % 2 == 0) callback(number);
    }
}

void main() {
    auto numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
    evenNumbers(numbers, (num) => writeln(num));
    // or numbers.evenNumbers((num) => writeln(num));
    // or numbers.evenNumbers((num) { writeln(num); });
}

In this case (num) => writeln(num) is an argument, callable function.

Output:

% rdmd delegate.d
2
4
6
8
10

Read more

iTunes 12 blue icon
Oct 28, 2014

I'm not big fan of new icon for iTunes 12. Here is how to change it:

Picture

iTunes keep it's icon at

/Applications/iTunes.app/Contents/Resources/iTunes.icns

For iTunes 11 you can download here: https://dl.dropboxusercontent.com/u/586048/iTunes.icns

I checked this manual and app liteicon it worked for me.

But it also works if you just replace icon.

# download file
wget https://dl.dropboxusercontent.com/u/586048/iTunes.icns -O /private/tmp/iTunes.icns
cd /Applications/iTunes.app/Contents/Resources
# replace icons
sudo mv iTunes.icns iTunes_red.icns
sudo cp /private/tmp/iTunes.icns .
# drop cache
sudo touch /Applications/iTunes.app
sudo touch /Applications/iTunes.app/Contents/Info.plist
sudo find /private/var/folders/ -name com.apple.dock.iconcache -exec rm {} \;
# restart dock (may take few seconds)
killall Dock

I tested on os x 10.9.5, blue iTunes icon is from macmini having not updated iTunes

Convert ruby time to server timezone
Sep 29, 2014

In web application have to handle about 4 different timezones:

  • application timezone
  • user timezone
  • server time timezone
  • database timezone

Application timezone

Usually same with timezone of your business location, also usually a default timezone.

Rails.configuration.time_zone # => "Jakarta"
# or
MyApp::Application.config.time_zone

User timezone

Timezone of current user, can be saved in user profile, or defected from javascript, eg http://pellepim.bitbucket.org/jstz/

Usually set as: Time.zone

Server timezone

It's a system time of current server, mostly UTC but sometimes can be a time of datacenter location. In development it's developer's machine timezone. You may need it if you want to make a cron task. Cron works in server's timezone.

To calculate it I have handy function what I use with gem whenever:

# Your business timezone 
Time.zone = "Asia/Jakarta"

def to_server_time(time_str)
  time = Time.zone.parse(time_str)
  server_offset = Time.now.getlocal.utc_offset.to_f / 1.hour
  time.in_time_zone(server_offset)
end

to_server_time('00:30') # => Sun, 28 Sep 2014 17:30:00 WET +00:00
to_server_time('00:30').strftime("%H:%M") # => "17:30"

Database timezone

Mysql don't store timezone in datetime format, but Postgresql does. Anyway it better if all dates in database have same timezone. Usually it's UTC, or can be custom:

Rails.configuration.active_record.default_timezone
# or
config.active_record.default_timezone

Node.js standart libs auto-require
Aug 26, 2014

I like node.js because it's very fast. But I also like to make a global objects in my application. I think nothing wrong with it. For example I have global.App object to keep object-wide things.

It was really annoying for me to write require('fs') or require('util') in every file, I also think it's not as fast as access global object.

So I come up with this solution:

var node = {};

var modules = ['child_process', 'fs', 'http', 'https', 'cluster',
  'crypto', 'dns', 'domain', 'net', 'url', 'util', 'vm', 'path'
];

var loadedModules = {};

modules.forEach(function(moduleName) {
  Object.defineProperty(node, moduleName, {
    get: function() {
      if (!loadedModules[moduleName]) {
        loadedModules[moduleName] = require(moduleName);
      }
      return loadedModules[moduleName];
    }
  });
});

module.exports = node;
global.node = node;

It make lazy-loading for all standart libraries as getters of global object node Later in my code:

node.fs.readFile();
node.util._extend({}, params);
node.url.parse(node.url.resolve(endpoint, filepath));
node.url.format(urlObj);

Continuous testing for D
Jul 16, 2014

I'm not sure if Travis CI can install packages, but with drone.io it's possible

Here is how I make it:

  • Add project
  • Choose C/C++ as language
  • Add build command which will install D compiler and run tests.

Command:

# prepare apt repository
sudo wget http://netcologne.dl.sourceforge.net/project/d-apt/files/d-apt.list -O /etc/apt/sources.list.d/d-apt.list
sudo apt-get update && sudo apt-get -y --allow-unauthenticated install --reinstall d-apt-keyring && sudo apt-get update
# install dmd and dub
sudo apt-get install dmd-bin dub
# build and run tests
dub test

Next page