Lua-TTyrant: a TokyoTyrant binding for Lua.

Filed under: Dynamic scripting, Projects

Some time ago I started a small project to implement a C-to-Lua binding of the TokyoTyrant library, called “lua-ttyrant“. A few days ago, while working with a script that was actually using it, I decided to polish it up, finish it and eventually submit it to be included in the LuaRocks repositories. Given the fact that the “rock” was just accepted by the LuaRocks team, I thought a new post was in order.

Files

The lua-ttyrant source code can be found here (among my github repos).
The lua-ttyrant rockspec can be found here (among all the public Lua rocks).

Prerequisites

• TokyoTyant library and headers.
• LuaRocks.

Prerequisites on Debian/Ubuntu distros:

sudo apt-get install libtokyotyrant-dev

Installation

sudo luarocks install lua-ttyrant

Status

The binding is in *beta* for now. Please contribute a ticket if you find anything wrong with it.

For anything else please see the doc/DETAILS file.

Enjoy!

A simple(r) approach for class-based OOP in JavaScript

Filed under: Dynamic scripting, Research, Techniques and algorithms

Like others before me, I found JavaScript’s prototype-based OOP support – while very powerful – quite cumbersome in some situations where I needed a certain structure or hierarchy of objects, situations in which one quickly comes to the conclusion that the best answer would be the concept of a Class.

However, while I am well aware that the web is filled with various implementations of classical (i.e. class-based) object orientation in JavaScript, I can not help the feeling that they are over-engineered and over-complicated without need, quite often attempting to provide more functionality than is actually necessary. In short, I don’t feel they’re “classical” at all.

This is why I started working on my own implementation of classical OOP  in JavaScript, which is shown below. Currently this is only a proposal, although I already started using it in one of my projects. Anyway, let’s look at it first then I’ll try and explain how it works.

The Code

/** Class factory. */
function Class(members) {

    // proxy constructor
    var Proxy = function() {
        for (var item in members) {
            this[item] = members[item];
        }
    };

    // proxy inheritance
    Proxy.prototype = (members.base || Class).prototype;

    // class constructor
    var Build = members.init || function() {};

    // class inheritance
    Build.prototype = new Proxy();
    Build.prototype.base = Proxy.prototype;
    Build.prototype.constructor = Build;

    // ready
    return Build;
}

That is all, and this little function can have a whole world of nifty consequences like deep inheritance or access to the super-class via this.base (and others). But here is what happens: the basic principle at work here is the Proxy object, which essentially, contains all the members of our new class (the ones that are passed to the Class factory function as the members parameter). Read the rest of this entry »

Command-line option parser in JavaScript

Filed under: Command line tools, Dynamic scripting

Here’s a small but very effective command-line option parser written in JavaScript (uses NodeJs). You can get the up-to-date source code at http://gist.github.com/982499.

I wrote this because I needed it for a project and I did not think it was worth it to install some npm package just for this. The comment should explain most things you will need but ask me if you don’t understand something. Note that this code is not well tested (yet!) since I just wrote it this morning so post a comment when you find a bug.

Supports:

• Short and long options (i.e. ‘-t|--test‘).
• Option parameters (i.e. ‘-f /etc/resolv.conf‘).
• Mandatory options.
• Implicit help option.
• Option callback functions.
• Cumulated short options (i.e. ‘-ab‘).
• Repeatable options (i.e. ‘--add value1 --add value2‘).
• Non-option arguments.

Notes (extracted from the comments):

• Parser is case-sensitive.
• The ‘-h|–help’ option is provided implicitly.
• Successfully parsed options will be available as fields in the “options” object.
• Non-option arguments found will be available in order in the “arguments” array.
• Options and their parameters must be separated by space.
• Either one of «short» or «long» must always be provided.
• The «callback» function is optional and is invoked each time the option is encountered.
• Cumulated short options are supported (i.e. ‘-tv‘).
• If an error occurs, the process is halted and the help message is shown.
• Options with the “multiple” attribute will be cumulated into arrays (even if found only once).
• The parser does *not* test for duplicate definitions in the schema array.

See the code for everything else you need to use it (i.e. a sample schema definition). I will not waste any more time talking about it; if you like it, paste it at the top of your NodeJS script and start playing with it, e.g.:

$ node options.js -tf /some/file.txt foo bar

Also, please tell me about any improvements or fixes you produce. Thanks and… enjoy!

Anastasia

Filed under: Events

My blog is about programming, that’s a fact and while I never broke this tradition so far, today I feel like making an exception.

Why, you ask? For her: mademoiselle… Anastasia. She was born on the 5th of January, 2011 and she really wanted to say “Hi!” on daddy’s blog. Who am I to say no to such a gorgeous girl.

Anastasia giving a speech

Anastasia resting between speeches

For more pictures go here. Enjoy! (I know I do…)

“Loomiere 2.0 teaser” reloaded

Filed under: Networking, Projects

«Update… Today this machine reached 6500+ active concurrent connections for a total traffic of 2841.19 Mbps (2778.1 output + 63.09 input) while the SSD’s were still below 20% load. Now that’s what I call serious FUN! Here are some pictures…

6.5k+ active concurrent streams

2841.19 Mbps effective bandwidth

…end!»

Since my last post about describing the performance of Loomiere in production, quite a few things have happened. I tested a lot and found many things I would like to improve or add, I tried many hardware configurations to see what goes well with what. In short, I have a whole list of things I want to implement as soon as possible and the most important item on this list is to actually launch Loomiere to the public as soon as possible!

However, here is a quick update on the performance of Loomiere, but this time with a nice twist: it’s running on a SSD-based machine! Here are some specs:

• Intel Core I7 3GHz
• 12 GB RAM
• 6 x 256GB SSD Kingston
• 3 x 1 Gigabit Ethernet cards
• Ubuntu Lucid Lynx with 2.6.37-12-server kernel

And here are the results after only a few hours from launch (graphs follow):

• 4500+ active oncurrent connections (streams)
• 300+ MB/s of throughput (aprox. 2500 Gbps)

Concurrent connections

Network traffic: eth0

Network traffic: eth1

Network traffic: eth2

However, at this point we hit a wall: the mainboard or the kernel. I have to dig into this problem further, but from what I could gather so far, it seems that the mainboard or the Linux kernel can’t handle more punishment. Following are the CPU and memory graphs; it is clear that while the memory has no worries, the CPU is hogging! The problem lies somewhere between the I/O operations and the actual sending of data to the sockets (interrupts maybe?). More testing will follow regarding this issue.

CPU activity

Memory usage

Hmm… this is a deep one.

So, that’s it for now, I’m going to sleep. Good night .

Loomiere 2.0 teaser

Filed under: Networking, Projects

Concurrent streams

Bandwidth utilization

This article is a quick peek into how Loomiere 2.0 is (currently) serving close to 3k active streams from a single server in production which is restricted to 1.6 Gbps of bandwidth (we are currently negotiating with our ISP to raise this limitation). All these statistics represent a single machine, in real time, over a period of 12 hours, equipped as follows:

• 2 x Xeon Dual Core (with HyperThreading) at 3.2GHz
• 16 GB RAM
• 15 x 1TB HDD (7200rpm)
• 4 x 1GB NIC’s
• This system runs on Ubuntu 10.04.1 LTS.

OK so here are the graphs:

• Cpu load.
• Network traffic (i.e. all interfaces cumulated).
• Active connections (i.e. concurrent streams).
• Memory usage.

The graphs may not yet be as stable as I would wish at the moment since I am still testing this setup as well as Loomiere itself, but I think soon I will have a more stable configuration and also some time to comment on these graphs.

Enjoy!

Case study: fixed number of iterations with LPeg

Filed under: Dynamic scripting, Research

Words seem to fail to describe just how awesome LPeg is. Designed as a Lua implementation of the PEG concept, it is a true programming gem! Please, if you dont’t know what it is, take some time to familiarize yourself with it! It’s not the easiest thing to grasp, but you will not regret it! It is certainly one of the most worthwhile learning efforts you can make in generic parsing.

One great feature of LPeg is that it’s binary-safe, meaning that (unlike regular expressions) it can be safely used to parse binary data! This makes it an excellent tool for parsing binary protocols, especially network communication protocols, such as the Action Message Format (used by Adobe Flash for making remote calls and even in FLV movie files). I’ll leave it to you to explore the possibilities…

Beware that from here on, I assume that you know your way around Lua, LPeg and how they work.

The problem

That being said, this article is actually about an unusual roadblock I hit while using LPeg to build a Lua-based AMF parser, and the various solutions I found and/or came up with to overcome it (you didn’t think that I mentioned AMF before by accident, did you?).

The issue is LPeg’s implementation of repetitive patterns: in particular, its inability to match (or capture) a fixed number of occurrences of a certain pattern, although it can match a minimum or a maximum number of such occurrences, which is perfect for stream-oriented parsing (such as parsing programming languages) but insufficient for binary data.

Just to clarify, here’s a small list of LPeg patterns which correspond to the typical PCRE repetitive constructs (in each case we’re trying to match the string ‘cloth’):

For the last two cases (i.e. A and B), LPeg does not offer any simple constructs so we have to find a complex one. Let’s put aside the case B for now, and try to tackle A… Read the rest of this entry »

Lua2C, an updated version

Filed under: Command line tools, Dynamic scripting

I know I have been “missing in action” lately but I am working furiously, and I seem to have too little time for my blog (very sad face). But, just for a breath of fresh air, I thought I’d share something with the world.

Entering lua2c.lua

Lately I became quite interested in Lua (a lot actually). It has phenomenal speed, exceptional interfacing with C and some features and libraries that just make my day (i.e. coroutines, lpeg, lua-ev and others), and since I needed to embed some Lua scripts (entirely) in a C project I’m currently working on, I ended up adapting Mike Edgar’s “bin2c.lua” script (which takes a Lua script and turns it into a C header file) to suit my needs.

Basic functionality

Specifically, this adaptation generates a function that takes a Lua state as the only argument and then runs the embedded Lua code in the given state after which it returns the status (as opposed to putting the code straight in the top-level scope of the generated file). This makes it easier to embed code in C and then invoke it, and also to apply the same code onto multiple Lua states (e.g. multiple threads).

Check further down for a short usage sample. Read the rest of this entry »

Loomiere/Stream: Revived!

Filed under: Networking, Projects, Web-development

OK folks, things have settled down and we are good to go. After some considerations the legal concerns for Loomiere/Stream are now cleared and gone. The source is now released and will be available indefinitely. Again, this streamer (minimally customized) has already been serving all the video content at http://peteava.ro for 3 full months (for those seeking a demo).

Source code: loomiere-0.2.1-tar.gz

Warning:
Any software downloaded from this website is *never* to be associated in any form with pornographic or erotic content! I.E. “Loomiere/Stream” must *never* be used to stream pornographic or erotic videos! There are plenty alternatives if you can’t help it.

Just a teaser:
Fighting the video-streaming problem has taught me very many things which, in turn, led me to realize that I might be able to use a substantially different streaming approach to achieve a massive amount of optimization in this field. So quite soon, I think I’ll have ready a brand new (and far more powerful) streamer that aims at making it possible for a single server to serve many thousands (I am still testing this) of streams simultaneously using commodity hardware. I am not yet settled on whether this project will be commercial, open-sourced or both but I hope to clear this aspect soon as well. Real-world streaming (i.e. before and after) statistics will be made available on release.

Stay tuned!

Seven essential WordPress plugins

Filed under: Security, Uncategorized

Here are the plugins I recommend for use on a WordPress blog.

Most (not all) of these plugins are generic, meaning that they are not bound to a specific type of blog and may (or should) be used regardless of what you write.

All of these plugins are top quality (IMHO) and provide an invaluable service for any blogger. I wholeheartedly recommend that you use and support them. I think that all of these should be apart of a standard WordPress installation.
Read the rest of this entry »