Creating a Web App in Erlang with Chicago Boss and Riak from Scratch - I
"Many years ago I used to think that programming was easy, as the years have passed I have have realized that programming is not easy. This is due to a slow perceptual shift in what I think programming is and what it is that a programmer does." - Joe Armstrong
Summary : During this article series, I will try to cover developing a basic eCommerce app in Erlang which is named as Chéri Cart. Through the development, we will use Chicago Boss Framework, Riak, an Erlang based open source distributed NoSQL key/value database and some third-party libraries. For development environment, we'll use the docker.io technology.
Disclaimer : I can't guarantee the validity of the information found here. The content of any given article may recently have changed, vandalized or altered by someone. I can't hold any responsibility for any damage done to your computer systems or data. The disclaimer implies also that using documentation or software in source codes. A published article doesn't means obliged to test, confirm, correct, approve the methods. The article is provided at no charge. Thus, so it is completely unsupported and it is publishing with no warranty of any kind.
"The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail." - Joe Armstrong
Chéri is an acronym of Chicago Boss, Erlang and Riak. I am prolly the first person who ever used it. I think the meaning of the French word chéri is fits well to the abilities of this development environment.
You may remember my first blog post. I've shared a short introduction to the Erlang programming language and it's concurrency philosophy in the article. There are some highlights of that Erlang already include:
High Availability and Reliability
- Simple and consistent error recovery and supervision hierarchiess.
- Built-in fault tolerance (Let it fail/crash! - Do not program defensively).
- Hot code loading during runtime (software upgrades with zero downtime).
Scalability and Heterogeneity
- Run on multiple platforms, HetNet support (Heterogeneous network).
- Network aware runtime, out-of-the-box distributed architectures.
- Very light-weight processes, highly scalable transparent or explicit concurrency.
- Awesome transparent multi-core support :
Figure : An Erlang based Web application runing on 16-core CPU.
- Functional programming language, high abstraction level, concise readable programs.
- When compared with any imperative language, 4–20 times less code written for same application.
- Suitable for rapid prototyping.
- Impressive and powerful libraries and middleware (Open Telecom Platform - OTP).
So, what is the "dark side" of Erlang for most developers?
Correct answer is :
- The Erlang syntax!
In my first week with Erlang, I had no idea what I've been doing while coding something. Believe me, if you fall into the Erlang world from any imperative language, you'd feel like me. The problem could be good to discuss, but that's beside the point.
Thanks to the GNU, - our Lord - there are some heroes working to figure out the problem. Elixir is the one of these efforts. Most people code with Elixir nowadays. I didn't try to code with Elixir because to learn pure Erlang syntax. BTW, I've read some good reviews from Joe Armstrong's article about the language.
Chicago Boss : Start small, dream big
In software development, using a framework is almost a rule for fast, clean, easy readable and standardized coding. Chicago Boss is a framework that is heavily inspired by Rails. Set up and use Chicago Boss is easy as falling off a log. Chicago Boss allows you to code with the aforementioned standards in Erlang. Plus, offers conveniences of modern web development, including WebSocket and Comet. Basic features of Chicago Boss listed below :
- 100% asynchronous I/O
- Support for Erlang and Elixir code
- BossDB : Database connection layer with an advanced ORM which with built-in support for Mnesia, MongoDB, MySQL, PostgreSQL, Riak and Tokyo Tyrant.
- BossCache : Database caching layer
- BossMQ : Cluster–wide, channel–based message queue
- BossRouter : URL router
- BossSession : Session storage layer
- BossNews : Event listener, model event system
- BossMail : Built-in email server
- Django and Jade template support
- Very clean controllers as result of pattern matching
- Auto document generation for models
- An useful admin interface
- Automatic code reloading
We'll cover almost all of features of Chicago Boss during developing the Chéri Cart.
So, let's write about Riak, our database layer.
Riak : Open source, distributed, rock-solid NoSQL database
NoSQL / Not Only SQL concept is increasing catching the developers' attention since 2009. The historic moment for the concept was no:sql(east) conference (October 2009, Atlanta).
select fun, profit from real_world where relational=false;
NoSQL is a generic term used to refer to any data store that does not follow the
traditional RDBMS model. In NoSQL approach, the data are non-relational and it does not use SQL as the query language.
The main problem of RDBMS systems scalability and availability. It doesn't mean RDBMS systems can't scale at all, but it is more expensive and harder to manage comparing to any NoSQL database.
According to Brewer's CAP theorem, a distributed system can't assure all of the following features at the same time:
- Consistency (All members / nodes has same time same data)
- Availability (Keep working even there is problem -i.e. crash- with node/s)
- Partition-tolerance (Tolerance to a network partition. An example of a network partition is when two nodes can't talk to each other, but there are clients able to talk to either one or both of those nodes.)
RDBMS systems focused to the consistency instead of availability. Because RDMS systems are often classified as providing ACID semantics properties *
Unlike RDMS systems, NoSQL systems are often targeted for availability. This, resulted the concept in computer science which known as BASE.
For more information about the ACID and BASE concepts please refer to the article.
There are several data modeling techniques in NoSQL. These techniques aim to different problems. We can easily group the NoSQL systems under 5 titles : Key-Value stores, Document databases, Graph databases, BigTable-style databases and Full-Text search engines.
Riak is an Erlang based decentralized key-value data store which offers extremely high availability, fault-tolerance, operational simplicity and scalability as default. Riak is developed to be the primary data storage mechanism in production applications by Basho Technologies.
What is Key-Value Store?
Key-value stores allow to store schema-less data. The data are usually consisting of a string which represents the key and the actual data which is considered to be the value in the key -> value relationship. Key/Value databases use the associative array (dictionary / map) as their base data model. Different subtitles can cover under the title. such as "ordered"," eventually consistent","immediately consistent", etc... I think this is the subject of another and more technical article. Riak, DynamoDB, Voldemort, Kyoto Tycoon, HamsterDB and Redis some well-known examples of the key-value stores.
What is schema-less data?
The relational schema defines what columns appear in the table, their names, and their data types. To understand schema-less data structures, let's begin with a relational schema:
The schema-less databases allow to store any data in the database, that structured with selfstanding fields and structures.
In the schema-less you don't have to define schemas and you can store all sorts of data without definition such as int, varchar, decimal, text etc... Let's look the diagram how it's look like :
I'll give detailed information in the further articles.
Riak : The Art of Keeping Your Ass Alive
A traditional small *SQL Cluster architecture looks like the following :
Almost all growing SQL-based projects tries to use master/slave cluster based solutions to manage easily its growing data or improve the performance and availability of the project. It is traditional and there are numerous way to manage, implement or extend the cluster. But, the master/slave approach on clustering has some key problems. Please look to the following scenario:
In the above situation, your application not able to store new data to your cluster.
So, what is the solution of Riak for the problem?
The Riak Ring is a 160-bit integer space. This space is equally divided into partitions, each of which is claimed by a vnode, which themselves reside on actual physical server nodes, says Basho. Let's see what they want to explain :
Media source : http://docs.basho.com/
In the consistent hashing method, for any key, its owner node is defined as the first encountered virtual node if walking clockwise from that key. When the owner node crashes, all the keys it owns will be owned by its clockwise next neighbour. As a result, key redistribution happens only within the neighbour of the crashed node, all other nodes retains the same set of keys. Following diagram is more expositive for master-less clustering architecture:
Let's see what happens when a node crashed :
I think it's enough for this article. I highly recommend to you the Riak Handbook for more information about data replication, membership changes, vector-clocks etc...
Also, following slide is good startpoint :
#### Docker.io : Build -> Ship -> Run! Docker is a portable, lightweight runtime and packaging tool. Docker offers to developers a new way to use containers for developing, testing, delivering and managing the applications.
As a developer, docker.io technology offers me more than that. I have several Docker Files that includes applications, my dot files, vim and mutt plugins, default /etc and application config files blah, blah...
That means, I can bring easily anywhere with me my all development needs as just some RAW files or docker images. Since Docker.io, I do not interest the installed OS or Linux distribution to start working out of my home office or without my old fashioned laptop (yes, I need and wish have an Apple more than dating with Emma Starr!). Even there is Windows OS installed PC! Thanks to GNU -our Lord-, I always keep with me a USB-stick that ready to give hardwares the kiss of life : Exterminate Windows!
We will start to set-up our development environment. It is a good time to take a coffee break and bring some cookies to the desktop.
Setting up the Development Environment
"Chimpanzees, gorillas, orangutans have been living for hundreds of thousands of years in their forest, living fantastic lives, never overpopulating, never destroying the forest. I would say that they have been in a way more successful than us as far as being in harmony with the environment." - Jane Goodall
For our development environment we need to have in order:
- Docker and Docker containers
- Erlang and Chicago Boss
- Riak and some Riak clusters
I perefer to use Ubuntu based distribution which called as elementary OS (yes, I need and wish have an Apple more than dating with Emma Starr!). So, I have to use apt.
Warning : elementary ships with old linux 3.2 kernel. You need to install the backported kernel at first. If you are not using this you should check your kernel version. You need to install > 3.8 kernel with AUFS and ZFS support.
# install the backported kernel $ sudo apt-get update $ sudo apt-get install linux-image-generic-lts-raring linux-headers-generic-lts-raring # reboot the system for booting with new kernel $ sudo reboot
First we need to install lxc-docker package:
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
If you receive a warning that the package isn't trusted. You should answer yes to continue.
$ sudo sh -c "echo deb https://get.docker.io/ubuntu docker main\ > /etc/apt/sources.list.d/docker.list" $ sudo apt-get update $ sudo apt-get install lxc-docker
After installation you can check the docker under simply
$ sudo docker version
If the output looks like something the following, that means the installation is successful :
Client version: 1.1.1 Client API version: 1.13 Go version (client): go1.2.1 Git commit (client): bd609d2 Server version: 1.1.1 Server API version: 1.13 Go version (server): go1.2.1 Git commit (server): bd609d2
If you are not familiar with docker.io let's visit the interactive demonstration : https://www.docker.com/tryit/
Installing Chicago Boss Container
You can pull my chicago boss docker image via following command:
$ sudo docker pull drlinux/chicagoboss
You may give following command to run the Chicago Boss container:
$ sudo docker run -i -p 8001 -v ~/development/chericart:/source/chericart -t drlinux/chicagoboss /bin/bash
Also, I've created a Dockerfile for creating a Chicago Boss docker container.
You can easily clone the repo giving following command under console:
$ git clone https://github.com/drlinux/docker-chicagoboss.git
The repo already includes the bash scripts for building and run the container. Now you can build the image and run your Chicago Boss container :
cd docker-chicagoboss/ # build the image, this will take some time $ sudo ./build.sh # now run the container $ sudo ./run.sh
The Dockerfile seems like following :
I do not explain the Dockerfile, it is out of the article's target. Please refer to Dockerfile reference.
Let's build and run the container :
$ sudo docker build -t drlinux/chicagoboss . $ sudo docker run -i -d -p 8001:8001 -t drlinux/chicagoboss
You can check your container status giving following command under console :
$ sudo docker ps
If the output like the following, that means our Chicago Boss container up and running :
Let's check out the Chicago Boss. Open your cherished Web-browser and type the address : http://127.0.0.1:8001
If you see the following page, that means the Chicago Boss is running :
Installing Riak Nodes on Docker
Hector Castro, the guy from Basho has recently done an amazing thing : Riak cluster on Docker!
Some months ago I've tried to install a cluster that has 15 nodes on a XEN virtual machine.I've just installed and checked to load on my machine, never used in production or development. Here you can find the htop screen shot from the installation.
Wait! Don't judge me easily. The VM already had a monster inside : numerous PHP + MySQL based applications and YAWS webserver behind an nginx proxy as http server. The screenshot taken while 15 nodes up and running at the same time with all of the configuration.
Let's clone Hector's repository and build the image from the Dockerfile:
$ export DOCKER_HOST="tcp://127.0.0.1:2375" $ git clone https://github.com/hectcastro/docker-riak.git $ cd docker-riak $ make build
After successful build, we can start a 5 node Riak cluster via following command :
$ DOCKER_RIAK_AUTOMATIC_CLUSTERING=1 DOCKER_RIAK_CLUSTER_SIZE=5 DOCKER_RIAK_BACKEND=riak_kv_eleveldb_backend make start-cluster ./bin/start-cluster.sh
The output should be like the following:
Bringing up cluster nodes: Successfully brought up [riak01] Successfully brought up [riak02] Successfully brought up [riak03] Successfully brought up [riak04] Successfully brought up [riak05] Please wait approximately 30 seconds for the cluster to stabilize.
So, as you may strongly understood, I need and wish have an Apple more than dating with Emma Starr!
In the next chapter of the series I'm planing to cover :
- Configuring Chicago Boss
- Configuring the database connection and connect to the Riak cluster
- Working with Chicago Boss
- Controller, Model and Template structure of the framework
- Working with Riak
- Discussing the Chéri Cart's database structure and use Riak's advantages.
Please do not hesitate to comment, fork and contribute.