July 26, 2014 · erlang chicago boss riak docker.io development nosql chéri cart

Creating a Web App in Erlang with Chicago Boss and Riak from Scratch - I

"Many years ago I used to think that programming was easy, as the years have passed I have have realized that programming is not easy. This is due to a slow perceptual shift in what I think programming is and what it is that a programmer does." - Joe Armstrong

Summary : During this article series, I will try to cover developing a basic eCommerce app in Erlang which is named as Chéri Cart. Through the development, we will use Chicago Boss Framework, Riak, an Erlang based open source distributed NoSQL key/value database and some third-party libraries. For development environment, we'll use the docker.io technology.


Disclaimer : I can't guarantee the validity of the information found here. The content of any given article may recently have changed, vandalized or altered by someone. I can't hold any responsibility for any damage done to your computer systems or data. The disclaimer implies also that using documentation or software in source codes. A published article doesn't means obliged to test, confirm, correct, approve the methods. The article is provided at no charge. Thus, so it is completely unsupported and it is publishing with no warranty of any kind.

Chéri Cart Logo

Why Chéri?

"The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail." - Joe Armstrong

Chéri? DAFUQ?

Chéri is an acronym of Chicago Boss, Erlang and Riak. I am prolly the first person who ever used it. I think the meaning of the French word chéri is fits well to the abilities of this development environment.

ERLANG :

Erlang logo by Erlang Solutions

You may remember my first blog post. I've shared a short introduction to the Erlang programming language and it's concurrency philosophy in the article. There are some highlights of that Erlang already include:

High Availability and Reliability

Scalability and Heterogeneity

Erlang multi-core support presentation Figure : An Erlang based Web application runing on 16-core CPU.
Source: http://smyck.net/2012/04/22/why-erlang/

Less Effort

So, what is the "dark side" of Erlang for most developers?

Wrote a

Correct answer is :
- The Erlang syntax!

In my first week with Erlang, I had no idea what I've been doing while coding something. Believe me, if you fall into the Erlang world from any imperative language, you'd feel like me. The problem could be good to discuss, but that's beside the point.

Thanks to the GNU, - our Lord - there are some heroes working to figure out the problem. Elixir is the one of these efforts. Most people code with Elixir nowadays. I didn't try to code with Elixir because to learn pure Erlang syntax. BTW, I've read some good reviews from Joe Armstrong's article about the language.

Chicago Boss : Start small, dream big

Chicago Boss Logo

In software development, using a framework is almost a rule for fast, clean, easy readable and standardized coding. Chicago Boss is a framework that is heavily inspired by Rails. Set up and use Chicago Boss is easy as falling off a log. Chicago Boss allows you to code with the aforementioned standards in Erlang. Plus, offers conveniences of modern web development, including WebSocket and Comet. Basic features of Chicago Boss listed below :

We'll cover almost all of features of Chicago Boss during developing the Chéri Cart.

So, let's write about Riak, our database layer.

Riak : Open source, distributed, rock-solid NoSQL database

Riak Logo

NoSQL / Not Only SQL concept is increasing catching the developers' attention since 2009. The historic moment for the concept was no:sql(east) conference (October 2009, Atlanta).

select fun, profit from real_world where relational=false;

NoSQL is a generic term used to refer to any data store that does not follow the
traditional RDBMS model. In NoSQL approach, the data are non-relational and it does not use SQL as the query language.

The main problem of RDBMS systems scalability and availability. It doesn't mean RDBMS systems can't scale at all, but it is more expensive and harder to manage comparing to any NoSQL database.
Nosql Comic According to Brewer's CAP theorem, a distributed system can't assure all of the following features at the same time:

RDBMS systems focused to the consistency instead of availability. Because RDMS systems are often classified as providing ACID semantics properties *
Unlike RDMS systems, NoSQL systems are often targeted for availability. This, resulted the concept in computer science which known as BASE.

For more information about the ACID and BASE concepts please refer to the article.

There are several data modeling techniques in NoSQL. These techniques aim to different problems. We can easily group the NoSQL systems under 5 titles : Key-Value stores, Document databases, Graph databases, BigTable-style databases and Full-Text search engines.

Riak

Riak is an Erlang based decentralized key-value data store which offers extremely high availability, fault-tolerance, operational simplicity and scalability as default. Riak is developed to be the primary data storage mechanism in production applications by Basho Technologies.

What is Key-Value Store?

What is Key-Value Store?

Key-value stores allow to store schema-less data. The data are usually consisting of a string which represents the key and the actual data which is considered to be the value in the key -> value relationship. Key/Value databases use the associative array (dictionary / map) as their base data model. Different subtitles can cover under the title. such as "ordered"," eventually consistent","immediately consistent", etc... I think this is the subject of another and more technical article. Riak, DynamoDB, Voldemort, Kyoto Tycoon, HamsterDB and Redis some well-known examples of the key-value stores.

What is schema-less data?

The relational schema defines what columns appear in the table, their names, and their data types. To understand schema-less data structures, let's begin with a relational schema:

sql schema diagram

The schema-less databases allow to store any data in the database, that structured with selfstanding fields and structures.
In the schema-less you don't have to define schemas and you can store all sorts of data without definition such as int, varchar, decimal, text etc... Let's look the diagram how it's look like :

schema-less data store

I'll give detailed information in the further articles.

Riak : The Art of Keeping Your Ass Alive

A traditional small *SQL Cluster architecture looks like the following :

Traditional Sql Cluster Diagram

Almost all growing SQL-based projects tries to use master/slave cluster based solutions to manage easily its growing data or improve the performance and availability of the project. It is traditional and there are numerous way to manage, implement or extend the cluster. But, the master/slave approach on clustering has some key problems. Please look to the following scenario:

Traditional Sql Cluster Master Node Failed Diagram

In the above situation, your application not able to store new data to your cluster.

So, what is the solution of Riak for the problem?

The Riak Ring is a 160-bit integer space. This space is equally divided into partitions, each of which is claimed by a vnode, which themselves reside on actual physical server nodes, says Basho. Let's see what they want to explain :

Riak Ring

Media source : http://docs.basho.com/

In the consistent hashing method, for any key, its owner node is defined as the first encountered virtual node if walking clockwise from that key. When the owner node crashes, all the keys it owns will be owned by its clockwise next neighbour. As a result, key redistribution happens only within the neighbour of the crashed node, all other nodes retains the same set of keys. Following diagram is more expositive for master-less clustering architecture:

consistent hashing diagram

Let's see what happens when a node crashed :

Consistent Hashing node Crashed

I think it's enough for this article. I highly recommend to you the Riak Handbook for more information about data replication, membership changes, vector-clocks etc...

Also, following slide is good startpoint :


#### Docker.io : Build -> Ship -> Run! Docker.io Logo Docker is a portable, lightweight runtime and packaging tool. Docker offers to developers a new way to use containers for developing, testing, delivering and managing the applications.

As a developer, docker.io technology offers me more than that. I have several Docker Files that includes applications, my dot files, vim and mutt plugins, default /etc and application config files blah, blah...

That means, I can bring easily anywhere with me my all development needs as just some RAW files or docker images. Since Docker.io, I do not interest the installed OS or Linux distribution to start working out of my home office or without my old fashioned laptop (yes, I need and wish have an Apple more than dating with Emma Starr!). Even there is Windows OS installed PC! Thanks to GNU -our Lord-, I always keep with me a USB-stick that ready to give hardwares the kiss of life : Exterminate Windows!

We will start to set-up our development environment. It is a good time to take a coffee break and bring some cookies to the desktop.

Setting up the Development Environment

"Chimpanzees, gorillas, orangutans have been living for hundreds of thousands of years in their forest, living fantastic lives, never overpopulating, never destroying the forest. I would say that they have been in a way more successful than us as far as being in harmony with the environment." - Jane Goodall

For our development environment we need to have in order:

macbook air on the desktop

I perefer to use Ubuntu based distribution which called as elementary OS (yes, I need and wish have an Apple more than dating with Emma Starr!). So, I have to use apt.

Warning : elementary ships with old linux 3.2 kernel. You need to install the backported kernel at first. If you are not using this you should check your kernel version. You need to install > 3.8 kernel with AUFS and ZFS support.

# install the backported kernel
$ sudo apt-get update
$ sudo apt-get install linux-image-generic-lts-raring linux-headers-generic-lts-raring
# reboot the system for booting with new kernel
$ sudo reboot

First we need to install lxc-docker package:

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

If you receive a warning that the package isn't trusted. You should answer yes to continue.

$ sudo sh -c "echo deb https://get.docker.io/ubuntu docker main\ > /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker

After installation you can check the docker under simply

$ sudo docker version

If the output looks like something the following, that means the installation is successful :

Client version: 1.1.1
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): bd609d2
Server version: 1.1.1
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): bd609d2

If you are not familiar with docker.io let's visit the interactive demonstration : https://www.docker.com/tryit/

Installing Chicago Boss Container

You can pull my chicago boss docker image via following command:

$ sudo docker pull drlinux/chicagoboss

You may give following command to run the Chicago Boss container:

$ sudo docker run -i -p 8001 -v ~/development/chericart:/source/chericart -t drlinux/chicagoboss /bin/bash

Also, I've created a Dockerfile for creating a Chicago Boss docker container.

You can easily clone the repo giving following command under console:

$ git clone https://github.com/drlinux/docker-chicagoboss.git

The repo already includes the bash scripts for building and run the container. Now you can build the image and run your Chicago Boss container :

cd docker-chicagoboss/

# build the image, this will take some time
$ sudo ./build.sh

# now run the container
$ sudo ./run.sh

The Dockerfile seems like following :

I do not explain the Dockerfile, it is out of the article's target. Please refer to Dockerfile reference.

Let's build and run the container :

$ sudo docker build -t drlinux/chicagoboss .
$ sudo docker run -i -d -p 8001:8001 -t drlinux/chicagoboss

You can check your container status giving following command under console :

$ sudo docker ps

If the output like the following, that means our Chicago Boss container up and running :

docker ps command output

Let's check out the Chicago Boss. Open your cherished Web-browser and type the address : http://127.0.0.1:8001

If you see the following page, that means the Chicago Boss is running :

Chéri Cart's First Screen Shot

Installing Riak Nodes on Docker

Hector Castro, the guy from Basho has recently done an amazing thing : Riak cluster on Docker!

Some months ago I've tried to install a cluster that has 15 nodes on a XEN virtual machine.I've just installed and checked to load on my machine, never used in production or development. Here you can find the htop screen shot from the installation.

Riak on Docker

Wait! Don't judge me easily. The VM already had a monster inside : numerous PHP + MySQL based applications and YAWS webserver behind an nginx proxy as http server. The screenshot taken while 15 nodes up and running at the same time with all of the configuration.

Let's clone Hector's repository and build the image from the Dockerfile:

$ export DOCKER_HOST="tcp://127.0.0.1:2375"
$ git clone https://github.com/hectcastro/docker-riak.git
$ cd docker-riak
$ make build

After successful build, we can start a 5 node Riak cluster via following command :

$ DOCKER_RIAK_AUTOMATIC_CLUSTERING=1 DOCKER_RIAK_CLUSTER_SIZE=5 DOCKER_RIAK_BACKEND=riak_kv_eleveldb_backend make start-cluster ./bin/start-cluster.sh

The output should be like the following:

  Bringing up cluster nodes:

    Successfully brought up [riak01]
    Successfully brought up [riak02]
    Successfully brought up [riak03]
    Successfully brought up [riak04]
    Successfully brought up [riak05]

  Please wait approximately 30 seconds for the cluster to stabilize.

What's next?

So, as you may strongly understood, I need and wish have an Apple more than dating with Emma Starr!

In the next chapter of the series I'm planing to cover :

Please do not hesitate to comment, fork and contribute.

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket
Comments powered by Disqus