• Cache scopes
• Caches Policies
• Caching storage strategies and implications on performance

Web Application Caching

Caching within web application can be done at different scope and granularity.

Some well know (common) ways of caching in web application is based on scope of cached values

1. Session Cache :– Cached value is stored per key per user and cached values are used for a single user. For example :– User home country.
2. Application Cache :– Cached value is stored per key and cached values are used for multiple users. For example :– List of countries.

Session cache’s Time To Live (TTL) is usually in minutes while Application cache’s TTL varies widely and usually cached forever until changed with application events or business triggers.

Application cache is memory efficient compared to Session cache, as same cached values is used by multiple users in application cache. But if the cached value is personalized data for a specific user, it should stay in Session Cache.

While Session Cache has been used a lot (sometime abused), Application cache is not widely used.

Cache Policies

• Write-through : Cache is updated along with backing datastore synchronously. Since both datastore and cache is always kept in sync, Write-through provides high data integrity and consistency at the cost of performance. Write-through caching makes sense with read heavy applications with very few writes.

• Write-back : Cache is updated synchronously and backing datastore is updated asynchronously. Since only cache is updated synchronously Write-back provides better performance but at the cost of inconsistency or data loss in an event of crash. Write-back caching policy makes sense when there is large number of writes and lossing latest data does not effect application.

As you can see detecting and handling cached value modification is very important. In fact, it is good practice to make all cached values immutable.

Cache strategies

Below are different cache storage options available

• In-memory cache : cached values are stored in RAM memory.
  • (a) in-process : caching with-in application process
  • (b) out-of-process : caching in another process
• Persistent cache : caching in persistent systems like files or database.

To choose caching option, we have to understand performance characteristics of different storage option. Lets start with time taken to perform typical operations on computer. Below numbers are from Jeff Dean presentation at Google.

First thing to notice from above numbers is, L1 and L2 cache. Caching is not only used in applications, network software, database systems, operating systems but also in Computer design.

• Back of the Envelope Calculations to determine performance of different caching strategies

  • In-memory and in-process caching :– Cache reads are fetched directly from current process memory.

    In-memory and in-process cache fetch time = Fetch from main memory = 100 nanosec

  • In-memory and out-of-process caching :– Cache reads are fetched from another process usually over network. Cached values are stored in another processes memory.

    In-memory and out-of-process cache fetch time = Round trip within same datacenter + Fetch from main memory = 500,000 nanosec + 100 nanosec

  • Persistent caching :– Cache reads are read from disk usually over network.

    In-memory and out-of-process cache fetch time = Fetch from new disk location (seek) + Round trip within same datacenter = 10,000,000 nanosec + 500,000 nanosec

    Persistent caching does not always mean reading from disk, databases usually cache working set data ( often used data ) in its main memory so when there is cache hit it performs similar to In-memory and out-of-process.

Summary

• In-memory and in-process caching is 500 times faster than In-memory and out-of-process cache fetch time
• In-memory and out-of-process cache fetch time is 20 times faster than Persistent caching

Clearly in-memory in-process caching is big winner and is widely used in web applications. Does that mean we should always use in-memory in-process caching ? To answer this question we have to understand characteristics of large scale web application.

Caching in Web Applications Cluster

As discussed in Architecture Issues Scaling Web Applications, large scale web application should be able to

• Horizontal scale out We should be able to add identical nodes in each layer to scale web applications.

• No single point of failure Is large cluster of nodes, failure of single node can happen and it should not bring down application.

Due to above two characteristics we end-up with cluster of nodes in any large scale applications.

Getting back to caching, if we go with in-memory in-process caching, each node will have to cache required data. Below are few issues with in-process caching

• Redundant caching consumes lot of memory. In cluster with N web application processes same cached value has to be stored N times.
• Each node will have to face cache miss and perform resource intensive operation before caching. i.e. In N Node cluster resource intensive operation is performed N times. This problem become noticeable if TTL of cached value is low. For Example :– A web application cluster with 100 nodes, and TTL of 1 min will perform 100 resource intensive operations every minute.
• Cache Refresh is another major problem with in-memory in process in cluster of web application process. Refreshing cache across the cluster is not easy and if cache is refreshed based on time. Due to machine time synchronization issue stale cached value will be used in some nodes, depending on application it could cause application in-consistent results based on which node handles request.

Out-of-process caching gets around the above issues by storing cached values in distributed caching systems like memcache. Due to central cache handling in out-of-process caching

• If one of the web application node caches a value, it is available to all other nodes reducing cache misses.
• Cache can be refreshed or invalided easily by updating central cache system.

Persistent caches is used when it is important to recover from crash with cached values intact. It is achieved by re-loading cached data from disk.

Caching is the most common way to improve performance or scalability of an web application. In fact, effectiveness of caching is arguably main reason for developer confidence to ignore premature optimisation or to postpone fixing probable performance issue until detecting an issue. The underlying assumption for this confidence is (a) the performance optimisation is may not be required (b) if required, performance issue could be fixed by caching.

Effectiveness of caching in improving performance is due to the fact that most applications are read heavy and perform lot of repetitive operations. So application’s performance can be greatly improved by avoiding repeatedly performing resource intensive operation several times. An operation could be resource intensive due to IO activity such as accessing database/service or due to computation/calculations which are CPU intensive.

What is caching?

Caching is storing result of an operation which can be used later instead of repeating the operation again.

Cache hit ratio

Before executing an operation, a check is performed to determine if the operation result (requested data) is already available in cache. If requested data is available, it is called cache hit and if requested data is not available in cache, it is called cache miss. When cache miss occurs, the requested operation is executed and result is cached for future use (lazy cache population).

Cache hit ratio is the number of cache hits divided by total number of requests for an operation. For efficient utilisation of memory, cache hit ration should be high.

Cache hit ratio closer to 0 means most of the requests miss cache and the requested operation is executed every time. Low cache hit ratio could also lead to large cache memory usage as every cache miss could mean new addition to cache storage. Cache hit ratio closer to 1 means most of the requests have cache hits and requested operation is almost never executed.

Cache population

Cache could be populated lazily after the executing the operation very first time or by pre-populating cache at the start of application or by another process/background job. Lazy cache population is most common usage pattern but cache population by another background job can be effective when possible.

Caching Layers in Web Architecture

There are several layers in typical web application architecture where caching can be performed.

As shown in the above diagram, caching can be done right from browser to database layer of architecture. Let us walk through each layer of caching touching how and what could be cached at that layer and understand pros and cons of caching at each layer.

Few general points to note in the above diagram

• Caching at left most layer is better for latency.
• Caching at right most layers gives better control over granularity of caching and ways to clear or refreshing cache.

Browser Cache

Browser loads a web page by makes several requests to server for both dynamic resource and static resources like images, style sheets, java scripts etc. Since web application is used by user several times, most of the resources are repeatedly requested from server. Browser can store some of the resources in browser cache and subsequent requests load locally cached resource instead of making server request, reducing the load on web server.

Cache granularity

• Static file caching like images , style sheets, java scripts etc
• Browsers also provide local storage (HTML5) and cookies which can be used for caching dynamic data.

How to populate cache

• Browser caching works by setting HTTP header parameters like cache control headers of resource to be cached with time to live (TTL).

Regular Cache Refresh

• Browser makes server requests after TTL time period and refreshes browser cache.

Forced Cache Refresh

• Cache refresh can be forced by changing cache invalidation parameter in the URL of the resource. Cache invalidation can be
• a query parameter Eg. http://server.com/js/library.js?timestamp
• a version number in resource path URL Eg. http://server.com/v2/js/library.js

In Single Page Applications (SPA), the initial page is always loaded from server but all other dependents like images, stylesheets, java scripts etc. are loaded based on (query parameter or version number in URL). When a new resource or application is deployed to production, cache invalidation parameter is changed to refresh cache.

Pros

• User will experience best response time with no latency.
• Can cache both static files and dynamic data.

Cons

• Load on server is marginally reduced since repeated requests for the same resource from other machine still hit the server.
• Dynamic data stored in HTML5 local storage could be security risk depending on application even through the html6 local storage is accessible to application domain only.

Content Delivery Network(CDN)

Content Delivery Network is a large distributed system of servers deployed in multiple data centres across the Internet. The goal of a CDN is to serve content to end-users with high availability and high performance. CDNs serve a large fraction of the Internet content today, including web objects (text, graphics and scripts), downloadable objects (media files, software, documents), applications (e-commerce, portals), live streaming media, on-demand streaming media, and social networks. —from Wikipedia

Content Delivery Network (CDN) providers usually work with Internet Service Providers (ISP) and Telecom companies to add data centres at the last mile of internet connection. Browser request latency is reduced since servers handling requests to cached resource are close to browser client. Browsers usually hold limited number of connection to each host, if the number of requests to a host is more than browser connection limit, requests are queued. CDN caching provides another benefit of distributing resources over several named domains, hence handling more browser requests parallely provides better performance or response time to user. Only static resources are cached in CDN since TTL of resources is usually in the order of days and not seconds.

Cache granularity

• Static File caching like images , style sheets, java scripts etc

How to populate cache

• Static resources such as images, stylesheets, java scripts can be stored in CDN using CDN provided tool or API. CDN provides an url for each resource which can be used in web application.

Regular Cache Refresh

• CDN tool or API can be used to configure time to live and when to replace static resource on CDN network servers, but replication of CDN resource over distributed network takes time, hence can not be used for very short TTL dynamic resource.

Forced Cache Refresh

• Similar to browser cache refresh above, forced cache refresh can be achieved by introducing versioning in url to invalidate cache.

Pros

• User will experience good response time with very little latency since resource is returned from nearest server.
• Load on server is greatly reduced since all requests for the resource will be handle by CDN.

Cons

• Cost of using CDN.
• Cache refresh time takes time as it has to refresh on highly distributed network of servers all over the world.
• Cache static resources only, since cache refresh is not easy.

Reverse Proxy Cache

A reverse proxy server is similar to normal proxy server, both act as intermediary between browser client and web server. The main difference is normal proxy server is placed closer to client and reverse proxy server is placed closer to web server. Since requests and responses go through reverse proxy, reverse proxy can cache response to a url and use it to respond to subsequent requests without hitting the web application server. Dynamic resource caching is significant benefit of caching at reverse proxy level with very low TTLs (few seconds). Reverse proxy can cache data in memory or in external cache management tool like memcache, will discuss more on it later.

There are several reverse proxy servers to choose from. Varnish, Squid and Ngnix are some of the options.

Cache granularity

• Static file caching like images, style sheets, java scripts etc
• Dynamic page response of http requests.

How to populate cache

• Other then reverse proxy server configurations to cache certain resources, proxy servers caching works using HTTP header parameters like cache control, expires headers of resource to be cached with time to leave (TTL).

Regular Cache Refresh

• Reverse proxy servers provide ways to invalidate and refresh cache and they make server requests after TTL time period and to refresh cache.

Forced Cache Refresh

• Reverse proxy servers can invalidate existing cache on demand.

Pros

• Web server load is reduced significantly as multi machine requests can use cache compared to browser cache where only requests for single machine are cached
• Reverse proxy servers can cache both static and dynamic resources. Eg. Home page of a site which is same for all clients can be cached for few seconds at reverse proxy.
• Most reverse proxy servers act as static file servers as-well, so web application never gets request for static resources.

Cons

• User will experience latency since request has to hit distant reverse proxy to get response.
• Reverse proxy processing and caching is required, which means added hardware costs compared to CDN costs. But most reverse proxies can be used for multiple purposes -as load balancer, as static file servers as-well as caching layer, added cost of additional box should not be a problem.

Next part of this post will cover different ways of caching within web application.

Thread Pool

In web applications thread pool size determines the number of concurrent requests that can be handled at any given time. If a web application gets more requests than thread pool size, excess requests are either queued or rejected.

Please note concurrent is not same as parallel. Concurrent requests are number of requests being processed while only few of them could be running on CPUs at any point of time. Parallel requests are number of requests being processed while all of them are running on CPUs at any point of time.

In Non-blocking IO applications such as NodeJS, a single thread (process) can handles multiple requests concurrently. In multi-core CPUs boxes, parallel requests can be handled by increasing number of threads or processes.

In blocking IO applications such as Java SpringMVC, a single thread can handle only one request concurrently. To handle more than one request concurrently we have to increase the number of threads.

CPU Bound Applications

In CPU bound applications thread Pool size should be equal number of cpus on the box. Adding more threads would interrupt request processing due to thread context switching and also increases response time.

Non-blocking IO applications will be CPU bound as there are no thread wait time while requests get processed.

IO Bound Applications

Determining thread pool size of IO bound application is lot more complicated and depends on response time of down stream systems, since a thread is blocked until other system responds. We would have to increase the number of threads to better utilise CPU as discussed in Reactor Pattern Part 1 : Applications with Blocking I/O.

Little’s law

Little’s law was used in non technology fields like banks to figure out number of bank teller counters required to handle incoming bank customers.

Little’s law

The long-term average number of customers in a stable system L is equal to the long-term 
average effective arrival rate, λ, multiplied by the average time a customer spends 
in the system, W; or expressed algebraically: 
L = λW.

Little’s law applied to web applications

The average number of threads in a system (Threads) is equal average web request 
arrival rate (WebRequests per sec), multiplied by the average response time (ResponseTime)

Threads = Number of Threads
WebRequests per sec = Number of Web Requests that can be processed in one second
ResponseTime = Time taken to process one web request

Threads = (WebRequests/sec) X ResponseTime

While the above equation provides the number of threads required to handle incoming requests, it does not provide information on the threads to cpu ratio i.e. how many threads should be allocated on a given box with x CPUs.

Testing to determining Thread Pool size

To find right thread pool size is to balance between throughput and response time. Starting with minimum a thread per cpu (Threads Pool Size = No of CPUs) , application thread pool size is directly proportional to the average response time of down stream systems until CPU usage is maxed out or response time is not degraded.

Below diagrams illustrate how number of requests, CPU and Response Time metrics are connected.

CPU Vs Number of Requests graph shows how CPU usage while increasing load on the web applications.

Response Time Vs Number of Requests graph shows response time impact due to increasing load on the web applications.

Green dot indicates point of optimal throughput and response time.

Thread pool size = Number of CPUs

Above diagram depicts blocking IO bound applications when number of threads is equal to number cpus. Application threads get blocked waiting for down stream systems to respond. Response time increases as requests get queued since threads are blocked. Application starts rejecting requests as all threads are blocked even though CPU usage is very low.

Large Thread pool size

Above diagram depicts blocking IO bound applications when large number of threads are created in web application. Due to large number of threads, thread context switching will be very frequent. Application CPU usage gets maxed out even though throughput has not increased due to unnecessary thread context switching. Response Time degrades since requests are interrupted with context switching.

Optimal Thread pool size

Above diagram depicts blocking IO bound applications when optimal number of threads are created in web application. CPU gets efficiently used with good throughput and fewer thread context switching. We notice good response time due to efficient request processing with fewer interrupts (context switching).

Thread Pool isolation

In most web applications, few types of web request take much longer to process than other web request types.The slower web requests might hog all threads and bring down entire application.

Couple of ways to handle this issue is

• to have separate box to handle slow web requests.
• to allocate a separate thread pool for slow web requests within the same application.

Determining optimal thread pool size of a blocking IO web application is difficult task. Usually done by performing several performance runs. Having several thread pools in a web application further complicates the process of determining optimal thread pool size.

Lets start by defining few terms to create common understanding and vocabulary. Later on I will go through different issues that pop-up while scaling web application like

• Architecture bottlenecks
• Scaling Database
• CPU Bound Application
• IO Bound Application

Determining optimal thread pool size of an web application will be covered in next blog.

Performance

Term performance of web application is used to mean several things. Most developers are primarily concerned with response time and scalability.

• Response Time

  Is the time taken by web application to process request and return response. Applications should respond to requests (response time) within acceptable duration. If application is taking beyond the acceptable time, it is said to be non-performing or degraded.

• Scalability

  Web application is said to be scalable if by adding more hardware, application can linearly take more requests than before. Two ways of adding more hardware are

  • Scaling Up (vertical scaling) :– increasing the number CPUs or adding faster CPUs on a single box.
  • Scaling Out (horizontal scaling) :– increasing the number of boxes.

Scaling Up Vs Scaling Out

Scaling out is considered more important as commodity hardware is cheaper compared to cost of special configuration hardware (super computer). But increasing the number of requests that an application can handle on a single commodity hardware box is also important. An application is said to be performing well if it can handle more requests with-out degrading response time by just adding more resources.

Response time Vs Scalability

Response time and Scalability don’t always go together i.e. application might have acceptable response times but may not handle more than certain number of requests or application can handle increasing number of requests but has poor or long response times. We have to strike a balance between scalability and response time to get good performance of the application.

Capacity Planning

Capacity planning is an exercise of figuring out the required hardware to handle expected load in production. Usually it involves figuring out performance of application with fewer boxes and based on performance per box projecting it. Finally verifying it with load/performance tests.

Scalable Architecture

Application architecture is scalable if each layer in multi layered architecture is scalable (scale out). For example :– As shown in diagram below we should be able linearly scale by add additional box in Application Layer or Database Layer.

Scaling Load Balancer

Load balancers can be scaled out by point DNS to multiple IP addresses and using DNS Round Robin for IP address lookup. Other option is to front another load balancer which distributes load to next level load balancers.

Adding multiple Load balancers is rare as a single box running nginx or HAProxy can handle more than 20K concurrent connections per box compared to web application boxes which can handle few thousand concurrent requests. So a single load balancer box can handle several web application boxes.

Scaling Database

Scaling database is one of the most common issues faced. Adding business logic (stored procedure, functions) in database layer brings in additional overhead and complexity.

RDBMS

RDBMS database can be scaled by having master-slave mode with read/writes on master database and only reads on slave databases. Master-Slave provides limited scaling of reads beyond which developers has to split the database into multiple databases.

NoSQL

CAP theorem has shown that is not possible to get Consistency, Availability and Partition tolerance simultaneously. NoSql databases usually compromise on consistency to get high availability and partition.

Splitting database

Database can be split vertically (Partitioning) or horizontally (Sharding).

• Vertically splitting (Partitioning) :– Database can be split into multiple loosely coupled sub-databases based of domain concepts. Eg:– Customer database, Product Database etc. Another way to split database is by moving few columns of an entity to one database and few other columns to another database. Eg:– Customer database , Customer contact Info database, Customer Orders database etc.

• Horizontally splitting (Sharding) :– Database can be horizontally split into multiple database based on some discrete attribute. Eg:– American Customers database, European Customers database.

Transiting from single database to multiple database using partitioning or sharding is a challenging task.

Architecture bottlenecks

Scaling bottlenecks are formed due to two issues

• Centralised component A component in application architecture which can not be scaled out adds an upper limit on number of requests that entire architecture or request pipeline can handle.

• High latency component A slow component in request pipeline puts lower limit on the response time of the application. Usual solution to fix this issue is to make high latency components into background jobs or executing them asynchronously with queuing.

CPU Bound Application

An application is said to be CPU bound if application throughput is limited by its CPU. By increasing CPU speed application response time can be reduced.

Few scenarios where applications could be CPU Bound

• Applications which are computing or processing data with out performing IO operations. (Finance or Trading Applications)
• Applications which use cache heavily and don’t perform any IO operations
• Applications which are asynchronous (i.e. Non Blocking), don’t wait on external resources. (Reactive Pattern Applications, NodeJS application)

In the above scenarios application is already working in efficiently but in few instances applications with badly written or inefficient code which perform unnecessary heavy calculations or looping on every request tend to show high CPU usage. By profiling application it is easy to figure out the inefficiencies and fix them.

These issues can be fixed by

• Caching precomputed values
• Performing the computation in separate background job.

Different ways of caching, how caching can reduce load and improve performance and scalability of web applications is covered in post Caching To Scale Web Applications

IO Bound Application

An application is said to be IO bound if application throughput is limited by its IO or network operations and increasing CPU speed does not bring down application response times. Most applications are IO bound due to the CRUD operation in most applications Performance tuning or scaling IO bound applications is a difficult job due to its dependency on other systems downstream.

Few scenarios where applications could be IO Bound

• Applications which are depended on database and perform CRUD operations
• Applications which consume drown stream web services for performing its operations

Next blog will cover how to determining optimal thread pool size of an web application.

Errors should never pass silently.
Unless explicitly silenced.

Silencing an error is hiding a symptom. Catching an exception and ignoring it is hiding error from being discovered and providing false feedback to user. Application should make errors transparent and visible. i.e. End user should know that the current task has failed and he had to take appropriate action and not proceed with assumption that the task has succeeded.

That said, in few occasions when a low priority task is performed along with high priority task, it is better not to fail the entire request if low priority task fails.

For Example:– User registration might have two steps

• Update user information in datastore
• Send out an confirmation email

If user information is updated in datastore and sending confirmation email failed it is ok to ignore send email error by queuing it to be processed later. But this also means taking appropriate steps to intimate support person and to make sure developer has enough information to figure out the reason for failure.

In the face of ambiguity, refuse the temptation to guess.
Although that way may not be obvious at first unless you’re Dutch.

In large systems it is difficult to figure what went wrong, without testing and inspecting several scenarios. I have seen developers jump into code to fix an issue without understanding the root caused for error and later to realise they were barking on wrong tree. It is better to simulate the error and than find a solution to fix the error, instead of working on a solution directly. After simulating error we could test out different hypothesis to confirm the assumptions.

Now is better than never.
Although never is often better than right now.

When faced with a need for large scale code refactoring, we tend to procrastinate and postpone the refactoring. Even thought we know the urgency or necessity for refactoring and how to fix the problem, we keep broken windows in codebase causing more issues.

There are cases where we don’t have a solution for the problem or we don’t have enough time to make the change right now. It is better to hold off changes, instead of doing half baked (partial) solution right now. In many cases it is better to hold off doing the changes until the right opportune moment, instead of jumping into action right away. (Eg :– Premature optimisation)

If the implementation is hard to explain, it’s a bad idea.
If the implementation is easy to explain, it may be a good idea.

If the business problem is not modelled correctly or there are several exceptions each core application flows. Software design tends to be hard to explain. In such cases, hard to explain, should be treated as a design-smell.

Difficulty to explain means bad design but easy to explain does not imply good design. Simple design might not have taken all different scenarios into consideration or could have made plain wrong assumption.

Design should be as-complex or as-simple as the requirement need, not more and not less. Over designing is adding unnecessary complexity to the system and under designing is not handling all scenarios.

Namespaces are one honking great idea — let’s do more of those!

There are several tools and conventions which make life easier, we should just follow them instead of reinventing new solutions. Following conventions and standards is very important means of managing code quality.

$ python

>>> import this

The Zen of Python, by Tim Peters
1 Beautiful is better than ugly.
2 Explicit is better than implicit.
3 Simple is better than complex.
4 Complex is better than complicated.
5 Flat is better than nested.
6 Sparse is better than dense.
7 Readability counts.
8 Special cases aren't special enough to break the rules.
9 Although practicality beats purity.
10 Errors should never pass silently.
11 Unless explicitly silenced.
12 In the face of ambiguity, refuse the temptation to guess.
13 There should be one-- and preferably only one --obvious way to do it.
14 Although that way may not be obvious at first unless you're Dutch.
15 Now is better than never.
16 Although never is often better than *right* now.
17 If the implementation is hard to explain, it's a bad idea.
18 If the implementation is easy to explain, it may be a good idea.
19 Namespaces are one honking great idea -- let's do more of those! 

Recap – Part 3 Conclusion

Async or Non-blocking IO introduces new challenges on how applications should to be structured and how async call backs can be abstracted away using promises like library. We need Non-blocking IO application since, sequential blocking IO applications are not scalable. So Non-Blocking IO or Asynchronous code is not a desired feature but a necessary evil to achieve scalability.

Finally, we should question assumption that Non-blocking IO and Asynchronous code are clubbed together and one comes with other. Is it possible to get the best of both the worlds, i.e. Sequential code and Non-Blocking IO scalability. In fact, I think there is an option based on Fibers which can provide best of both the worlds.

An interesting article on Why Events Are A Bad Idea makes sequential control flow arguments in more depth.

I will cover Fibers and how they achieve both Non-Blocking IO and Sequential codebase and demonstrate it with an expressjs resful service in the this blog.

Pre-emptive and co-operative multitasking is a good place to begin understanding Fibers.

Pre-emptive Multitasking and Co-Operative Multitasking

Pre-emptive Multitasking :–

In computing, preemption is the act of temporarily interrupting a task being carried out by a computer system, without requiring its cooperation, and with the intention of resuming the task at a later time. Such a change is known as a context switch. It is normally carried out by a privileged task or part of the system known as a preemptive scheduler, which has the power to preempt, or interrupt, and later resume, other tasks in the system. – from Wikipedia

Linux Scheduler (privileged task) pre-empts process tasks without its co-operation. The disadvantage of preemptive multitasking is that the OS may make a context switch at an inappropriate time.

Co-Operative Multitasking :–

Early multitasking systems used applications that voluntarily ceded time to one another. This approach, which was eventually supported by many computer operating systems, is known today as cooperative multitasking. – from Wikipedia

Co-Operative Multitasking relies on the threads relinquishing control once they are at a stopping point. The disadvantage of co-operative multitasking is that a poorly written application can blocking the entire system. Real-time embedded systems are often implemented using Co-Operative Multitasking paradigm to get real time performance.

What are Fibers?

Fibers are lightweight threads (also called green threads) which are process or application level concepts and don’t correspond to OS threads. They provide thread like execution flow. While OS threads are pre-emtively scheduled, programmer can use fibers to co-opratively multitask. Fibers are conceptually similar to coroutines .i.e. execution can be suspended and resumed programmatically.

Lets see an example of how fibers work

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

var Fiber = require('fibers');

var log_sequence_counter = 1;

function sleep(task, milliseconds) {
    var fiber = Fiber.current;
    setTimeout(function() {
      console.log(log_sequence_counter++ + task + ' callback');
        fiber.run();
    }, milliseconds);
    console.log(log_sequence_counter++ + task + ' thread/fiber suspended');
    Fiber.yield();
    console.log(log_sequence_counter++ + task + ' thread/fiber resumed');
}

var task1 = function() {
    console.log(log_sequence_counter++ + ' task 1 waiting for sleep to end ');
    sleep(" task 1",1000);
    console.log(log_sequence_counter++ + ' task 1 got back from sleep');
}

var task2 = function() {
    console.log(log_sequence_counter++ + ' task 2 waiting for sleep to end ');
    sleep(" task 2", 1000);
    console.log(log_sequence_counter++ + ' task 2 got back from sleep');
}

Fiber(task1).run();
Fiber(task2).run();
console.log( log_sequence_counter++ + ' main execution flow');

In the above example, you can notice that

• Fibers are created using Fiber() function and the task function is executed using run method.
• Pattern used to suspend and resume a Fiber. This pattern will be reused while making Non-Blocking IO calls.
• Within a fiber thread Fiber.current returns current executing fiber.
• Fiber.yield suspends execution of current thread i.e. voluntarily relinquish control. In other words it allows another fiber thread execute co-operatively.
• task1 and task2 functions don’t have callbacks or promises and is sequential code.
• task1 and task2 functions don’t know about fibers and developers can read/write these functions as sequential code.
• Fibers provide co-operative multi-tasking capability

1
2
3
4
5
6
7
8
9
10
11
12

$ node fibersExample.js
1 task 1 waiting for sleep to end
2 task 1 thread/fiber suspended
3 task 2 waiting for sleep to end
4 task 2 thread/fiber suspended
5 main execution flow
6 task 1 callback
7 task 1 thread/fiber resumed
8 task 1 got back from sleep
9 task 2 callback
10 task 2 thread/fiber resumed
11 task 2 got back from sleep

The output of fibersExample.js shows the other of execution with sequence numbers. Even though NodeJS is single threaded application, the above code demonstrates – how multiple tasks are run with-out blocking each other.

Fibers – In ExpressJS Restful Service

Lets look at an ExpressJS restful service example, to keep code simple, I have not included exception handling which can be done using try – catch blocks as we do other languages. It provides three service methods

1. /google :– Makes a http get call to google and returns html response from google to client.
2. /user/:fb_id :– Return User JSON for given facebook id.
3. /user/:fb_id/events :– Returns User and User’s Events for a given facebook id.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

var express = require('express');
var fibersMiddleWare = require('./fib-middleware');
var request = require("./fib-request");
var fib_redis = require("./fib-redis");
var redis_client = fib_redis.init(require("redis-url").connect());

/*------------------------------------Models----------------------------------*/

var User = {};
User.get = function(id){
              var user_json = redis_client.get("user:" + id);
              return JSON.parse(user_json);
          };

var Event = {};
Event.getUserEvents = function(user_id){
                        var user_json = redis_client.mget("user:" + id + ":events");
                        return JSON.parse(user_json);
                    };

/*----------------------------------------------------------------------------*/

var app = express();
app.use(fibersMiddleWare.runInFiber);

app.get('/google', function(req, res){
  var google_response_body = request.get('http://google.com')
  res.send(google_response_body);
});

app.get('/users/:fb_id',function(req, res){
                  var user = User.get(req.params.fb_id)
                  res.setHeader('Content-Type', 'application/json');
                  res.send(200,JSON.stringify(user));
                });

app.get('/users/:fb_id/events',function(req, res){
                  var user = User.get(req.params.fb_id)
                  var events = Event.getUserEvents(user.id);
                  var response = {'user' : user, 'events' : events};

                  res.setHeader('Content-Type', 'application/json');
                  res.send(200,JSON.stringify(response));
                });

var server = app.listen(3000, function() {
    console.log('Listening on port %d', server.address().port);
});

In the above example, we have used few custom/wrapper libraries ‘fib-middleware’, ‘fib-request’ and ‘fib-redis’. They are simple extensions of simple fibers example above.

You can notice that

• Fibers are setup as middleware using app.use(fibersMiddleWare.runInFiber)
• Controller and Model methods are Synchronous and Sequential Code.
• Http get Request and Datastore (redis_client) operations are also Synchronous.
• Other than sequential code there is nothing special happening with-in Server.js code above.
• Code demonstrates that Non-Blocking IO code can be synchronous and sequential

Lets looks at the libraries that server.js depends on

1
2
3
4
5
6
7
8
9

var Fiber = require("fibers");

function fiberMiddleWare(req,resp,next){
  Fiber(function(){
      next();
  }).run();
}

exports.runInFiber = fiberMiddleWare

ExpressJS middleware is an implementation intercepting filters pattern which can be used to perform any processing before and after passing the request to controller method.

fib-middleware.js is a simple mechanism to process all http requests within a fiber context. It is similar to Fiber(task1).run() in the fibersExample.js above.

Custom Wrapper Libraries to provide Fibers Support

fib-request.js and fib-redis.js are wrappers which provide Fibers support and are not required if original library (redis.js or request.js) support Fibers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

var request = require('request');
var Fiber = require("fibers");

function get(url){
  var error,response,body;
  var fiber = Fiber.current;
  request.get(url,
                    function(err, resp, b){
                      error = err;
                      response = resp;
                      body = b;
                      fiber.run()
                    });
  Fiber.yield();
  return body;
}

exports.get = get;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

var Fiber = require('fibers');
var conn;

function get(key){
  var error, value;
  var fiber = Fiber.current;
  conn.get(key, function(err,val){
      error = err;
      value = val;
      fiber.run();
    });
    Fiber.yield();
    return value;
}
function init(connection){
   conn = connection;
   return { 'get':get};
}

exports.init = init;

As you can see, fib-request.js and fib-redis.js follow code pattern similar to the fibers example of sleep where thread is suspended and resumed to make the function synchronous.

The above sample ExpressJS application shows that it is possible to write sequential Non-Blocking I/O using Fibers. Though I have used NodeJS in the sample codebase. It is not restricted to NodeJS, the same applies to other languages which support Fibers (light-weight threads) like Ruby, Python etc.

Libraries Supporting Fibers

Most application codebase can be divided into two major parts

1. Business or functional part :– All business and function logic of application is written in this part of codebase by project team developers. Most of developers time is spend working on this part of application.
2. Framework or Library part :– In all projects we end-up using several libraries like mvc framework, database drivers etc. which are third part libraries developed by developers outside team. These usually are used across projects and developers usually spend every little time modifying or changing this part.

In the express sample above

1. Server.js belongs to Business application part.
2. ExpressJS, Fibers.js, redis.js, fib-middleware.js, fib-request and fib-redis.js belong to framework or library part. i.e. once implemented by library developers or project team, it can be re-used across projects.

As shown in the above code, creating fiber based wrapper libraries for NodeJS libraries is pretty simple.

Recently, library developers started supporting Promise Library, which was not the case before. As fibers gain wider developer acceptance, library developers would also support fibers.

Asynchronous Vs Parallel

Most of the examples we have seen till now (this and previous blogs on reactor pattern) has been an instance of synchronous code converted to asynchronous code to make it non-blocking. While this is a common scenario and in normal projects it covers 90 to 95% of scenarios in NodeJS applications. But there are occasional requirement which require multiple parallel requests to be made and wait for all the responses to get back. This is a genuine case where parallel or async processing is required and should be allowed to work asynchronously in usual NodeJS Async pattern with help Promise Library (Q.all()).

Performance/Scalability

Siege based Performance testing of async call-back based code and sequential fibers code (shown above) showed similar results on my MacBook Pro.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ siege -r 10 -c 100 http://localhost:3000/users/11265765672

Transactions:             1000 hits
Availability:           100.00 %
Elapsed time:            10.06 secs
Data transferred:         0.20 MB
Response time:                0.01 secs
Transaction rate:        99.40 trans/sec
Throughput:               0.02 MB/sec
Concurrency:              0.66
Successful transactions:        1000
Failed transactions:             0
Longest transaction:          0.05
Shortest transaction:         0.00

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ siege -r 10 -c 100 http://localhost:3000/users/11265765672

Transactions:             1000 hits
Availability:           100.00 %
Elapsed time:             9.09 secs
Data transferred:         0.20 MB
Response time:                0.01 secs
Transaction rate:       110.01 trans/sec
Throughput:               0.02 MB/sec
Concurrency:              0.92
Successful transactions:        1000
Failed transactions:             0
Longest transaction:          0.06
Shortest transaction:         0.00

Conclusion

With Lightweight thread (Fibers) we can do Co-Operative multi-tasking which voluntarily relinquish control and resume processing. If application framework and libraries support Fibers or library wrappers created as shown in above example, functional and application logic can be written in synchronous style. Presence of Fibers, async code can be abstracted into Framework and libraries from functional code. As functional code tend to be larger than wrapper code that might be required, this will substantially reduce application complexity.

Lets get little more detail on what the callback issues are and introduce few options or libraries used to partially solve the issue, in this blog.

Example scenario

Consider a simple scenario where Restful service has to return a list of events for a given user. User is identified by Facebook Id. The service datastore has two tables or collections, (a) User table and (b) Event table which stores.

• user information

  • id
  • fb_id
  • name
  • ….

• events information.

  • id
  • user_id
  • event_name
  • event_desc
  • start_date
  • ….

Assuming, you are not allowed to join user and event table or datastore is mongodb. The service controller will use below Pseudocode steps to get events.

• Given facebook id, get user id from User collection
• Given user id, get events for the user from events collection.

Sequential Code with Blocking I/O

In normal scenario the above steps directly translate to below code. If performance or scalability is not something playing on developers mind, Sequential code is the simplest and normal thing to do.

1
2
3
4
5
6
7
8
9
10

function getUserEvents(request,response){
  var facebook_id = request.param('facebook_id');
  try{
      var user = db.users.findOne({fb_id:facebook_id});
      var events = db.events.find({user_id:user.id});
      response.write(events);
  }catch(err){
      response.status(500).send(err);
  }
}

There are religious wars between Ruby on Rails and NodeJS developers on Sequential Vs Evented style code.

• Key Points
  • Majority of web applications (Ruby on Rails, Java Spring, Django etc) are written in sequential style.
  • Sequential style is simple and readable.
  • Most people think in sequential style i.e. Developers tend to break application logic into sequential steps like Pseudocode provided above.
  • Boundaries of Pseudocode Step does not usually end at network call or IO call.
  • Non-blocking I/O is considered when we need better scalabilty or performance.

Unfortunately Sequential code is linked to blocking I/O calls, becuase threads follow pre-emptive multitasking and not co-operative multitaking. More on this later when talking about Fibers.

Callback based solution

To solve Blocking I/O problem, code is split to three parts

1. Processing done before making a network or IO call
2. Network or IO call
3. Processing done after getting back data from network or IO call.

Execution flow for each of above steps is seperated and such that each of them can be executed from the event loop.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

function getUserEvents(request,response){

  var returnEvents = function(err,events){
          if (err) respone.status(500).send(err);;
          response.write(events);
      });  

  var givenUserFindAndReturnEvents = function(err,user){
      if (err) respone.status(500).send(err);;
      db.events.find({user_id:user.id},returnEvents);    
  };

  var findUserAndReturnEvents = function(){
      var facebook_id = request.param('facebook_id');
      db.users.findOne({fb_id:facebook_id}, givenUserFindAndReturnEvents);
  }

  findUserAndReturnEvents();
}

Notice that request and response objects are not passed to sub-functions. The sub-functions get access request and response since sub-functions are javascript closures. In fact if we move the sub-functions out-side getUserEvents method, it would not work. Which lead to chooice of keeping givenUserFindAndReturnEvents and returnEvents as sub-functions. Curring can be used to fix this problem, more on that in another blog.

Each of the sub-functions (findUserAndReturnEvents, givenUserFindAndReturnEvents, returnEvents) are executed asynchronously. functions givenUserFindAndReturnEvents and returnEvents are called call-back functions since they are triggered after getting back user object and event objects respectively from datastore.

The sub-functions could have been left as in-line or nested lamda functions. Nesting several such functions is another issue with call-backs.

• Key Points
  • Code is separated based on pre-network call and post network call.
  • The caller of the sub-function has to pass a callback function to execute after finishing sub-function task.
  • Sequential logic is expressed asynchronously.
  • Asynchronous code above is more scalable but may not be more performant (response time).
  • Call-back causes readability issues – callback hell.
  • Following execution flow is difficult, so called spaghetti-code.
  • Non-Blocking API’s impose major constrain on how you structure your code.
  • Functions are hierarchy, i.e. calling function is responsible for functionality it provides as-well as the sub-function it calls. For example:– givenUserFindAndReturnEvents includes functionality of finding and returning Events to http response.

Promise based solution

To solve Call-back issues like spaghetti-code, we could use code structuring library like q promise. Promise library provides some code style standards and structuring, making it more readable compared to call-back based code shown above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

var loadEventsForUser = function(err,user){
  return db.events.find({user_id:user.id});  
};

var findUser = function(){
  var facebook_id = request.param('facebook_id');
  return db.users.findOne({fb_id:facebook_id});
}

function getUserEvents(request,response){
  var success = function(events){
          response.write(events);
  };

  var error = function(err){
      response.status(500).send(err);
  };

  findUser()
  .then(loadEventsForUser)
  .then(success)
  .fail(error);
}

Notice how code is split into smaller independent functions and how they are chained together using .then and .fail functions. Another important feature of promise library is how exception flow is handled. Compare promise based code with the call back based one above. Observe that errors are handled in each of the call back functions and when using promise, errors are isolated and handle seperately.

• Key Points
  • Functions are flat, i.e. calling function is responsible for only its own functionality and can be used independently. For example :– findUser can be used independent of loadEventsForUser.
  • Spliting sequential code into indenpendent functions which are reusable in multiple scenarios is not always easy. Many times functions are created just to work around Non-blocking reactive pattern.
  • Functions can be used in other flows and could form reusable components.
  • Better readability compared to call back option but not as simpile as sequential option
  • Better exception handling compared to call back option but not as simple as sequential option.
  • When libraries don’t support promises, we end up writing boiler plat code to create promise and to handle async flows.

Conclusion

Async or Non-blocking IO introduces new challenges on how applications should to be structured and how async call backs can be abstracted away using promises like library. We need Non-blocking IO application since, sequencial blocking IO applications are not scalable. So Non-Blocking IO or Asynchronous code is not a desired feature but a nessesary evil to achive scalability.

Finally, assumption that Non-blocking IO and Asynchronous code are clubed together and one comes with other. Is it possible to get the best of both the worlds, i.e. Sequential code and Non-Blocking IO scalability. In fact, I think there is an option based on Fibers which can provide best of both the worlds. I will cover Fibers and how they achive both Non-Blocking IO and Sequential codebase in the Reactor Pattern Part 4 Write Sequential Non Blocking IO Code With Fibers In NodeJS.

Lines 1 to 9 of Zen of Python is covered in this blog, Lines 10 to 19 will be covered in Zen of Software Design : Part 2.

Interpretation

The way to read “better than” statements in poem, is to understand that one does not always win over other and both have their place under sun. But the poem advocates one over another when all things are equal and there is a choice to be made. i.e. leaning or favouring one over another.

Beautiful is better than ugly

While it is easy to understand that beautiful is better than ugly, it is very subjective to know beautiful from ugly. If you look at source code as an art, like a painting, bad code/design will stink (code-smell) and make you uncomfortable instinctively. I don’t have standard rules by which you can differentiate beautiful from ugly design (in same sense as you can not set standard rules for beautiful paintings). I guess it comes from looking at different codebase and experience.

Explicit is better than implicit.

It should be obvious to figure-out what’s happening in code, even for someone not familiar with the codebase. Usually implicit design makes it look like black magic without any clue how it is working or where to look for related code. There are two kind of implicit we face in day-to-day work.

• Well known implicit :– Things which are well documented and have become standard way of doing things. These implicit usually go across project team boundaries. Example : SpringMVC annotations, Rails conventions

• Project implicit :– Things which project teams builds within the project which is not something a new developer would be acquainted with in other projects.

I have seen developers going overboard with Annotations or conventions in projects, where it is difficult to figure-out how things work. A rule of thumb I use to figure out if we have gone overboard is by asking myself the question “Can I explain, to a new developer joining team, all the implicit things happening in the project within 1 hour session?”.

Simple is better than complex.

Again this seems very obvious but still elusive. No one wants to create a complex system but creating simple design/software is a complex job.

Few questions which help me be on track of simplicity

• How many execution flows/paths exists?

  if there are few execution paths than its easy to keep them in mind and figure out a specific scenario falls in which one of the paths.

• How easy is it to figure-out a given scenario falls under which execution path?

  It should be easy to figure that out, if the application is well designed and simple to remember.

• How many exceptions flows exist as against conventional flows?

  In every application there are few exceptional scenarios which don’t fall under normal execution paths and need exceptional flows for them. These should be minimal and explicitly identified.

• How easy it is to explain high level design to a new developer?

  I found this rule of thumb to be very helpful. It becomes obvious when you explain (or imagining to explain) how application works to a new developer. If you can explain it without flinching several times when the developer says “ah in scenario x app will do this” and you go hmmm its mostly correct but there are some other issues etc.

Complex is better than complicated.

Design can be complex because domain is complex but sometime domain is simple but software is designed or implemented in complicated way. Design should be as-complex or as-simple as the requirement need, not more and not less.

The questions I ask to figure out if design is complicated is

• What part of requirement is causing the design to be complex?
• Is the technology choice introducing additional complexity?
• Is there any negotiable requirement which can be removed to make it more simpler?
• Are we designing the system for future proofing which might not be required?

Flat is better than nested.

When we look at code there are few obvious cases where this is true. For example it is better to have several flat functions than having one long nested function. For example :– Using Strategy Design Pattern or Command Design pattern to separate code into independent units. Where it is not very obvious and still useful to think is

• How many layers are we introducing in software architecture?
• How many jumps do we need to make to get to the final value (may be cached value or database lookup) ?
• Is the path of execution intermingled and can not be modified independently?

Sparse is better than dense. Readability counts.

Few times I have questioned myself if single responsibility principle, unit testing, dependency injection and design patterns are good. The reason for this doubt is that after following single responsibility principle, unit testing with dependency injection and following design pattern, a small codebase grows to considerable code size. I call this new increased codebase sparse compared with earlier code which was dense.

The advantages of sparse code base compared to dense code is

• It is easier to understand and modify.
• It is easier to extend to add more scenarios.
• It is easier to isolate an issue and fix it.

Special cases aren’t special enough to break the rules. Although practicality beats purity.

Good software Architecture/Design usually settle with a set of rules like

• Execution flows/paths
• Logical Architecture Layers and how each layer communicated with another layer.
• Inter-process communication protocols

Every now-and-then a new requirement which breaks design rules under which application has operating comes in. Usual tendency is to treat this as special case which works differently to existing architecture.

While it might be the right approach in few cases, we should

• strive to see if this special case can be modelled as one of the existing flows
• check if existing rules can be extended to incorporate the new requirement as first class design decision instead of treating it as special case which breaks the rules.
• check if we have been adding a lot of special cases and do a course correction if required.

On the other hand if they truly are special cases and trying to extend or modelling it has one of the existing flows make design complicated, we should not hesitate to incorporate the new requirement as special case as a pragmatic/practical architect or developer should do.

Lines 10 to 19 will be covered in Zen of Software Design : Part 2 of this blog

$ python

>>> import this
The Zen of Python, by Tim Peters

1 Beautiful is better than ugly.
2 Explicit is better than implicit.
3 Simple is better than complex.
4 Complex is better than complicated.
5 Flat is better than nested.
6 Sparse is better than dense.
7 Readability counts.
8 Special cases aren't special enough to break the rules.
9 Although practicality beats purity.
10 Errors should never pass silently.
11 Unless explicitly silenced.
12 In the face of ambiguity, refuse the temptation to guess.
13 There should be one-- and preferably only one --obvious way to do it.
14 Although that way may not be obvious at first unless you're Dutch.
15 Now is better than never.
16 Although never is often better than *right* now.
17 If the implementation is hard to explain, it's a bad idea.
18 If the implementation is easy to explain, it may be a good idea.
19 Namespaces are one honking great idea -- let's do more of those! 

In this blog, we will look at an alternative solution to maximise CPU usage.

The diagram below from the part 1 blog we can notice that applications require CPU in busts and they have wait periods between the processing busts.

If we treat processing busts as events and queue events to be executed on a single thread, we can make sure the thread (CPU) is always occupied and thus giving us maximising CPU usage and avoiding unnecessary context switching.

Consider processing blocks as events without any I/O calls within them which can be executed non-sequentially/asynchronously. The code should be split into execution blocks (events) which can be executed separately with other requests events executed between them.

Event Loop

An event loop is a program loop where a thread waits for events and executes events that occur in a program. For more info on checkout Event Loop

Unix File Descriptors

Following Unix principle of everything is a file, file descriptor is used to detect events when reading/writing to file, network communication, device communication and inter-process communication. System calls “epoll” and “pselect” are used to detect file descriptor state change without blocking application.

The below diagram shows a simplified event loop which goes through four stages in each loop.

• Check if a new event is created (i.e. a web request has come in or a call back event has occurred) and add new event is added to the queue.
• Pick an event from the queue
• Execute event
• Create a call-back event (i.e. database query or network access call-back when the response has arrived. Call-back events are handled by epoll or pselect.

The solution described above is a simplified version of Reactor Pattern.

Advantages of Reactor Pattern

• Optimal usage of CPU.
• Can handle more requests with same hardware.
• Can scale above C10K limit

C10K Problem

Historically, Reactor Pattern came into prominence after C10K problem and Solution to C10K problem using Non-Blocking I/O succeeded.

Disadvantages of Reactor Pattern

• Asynchronous event based code base makes it difficult to understand and structure code. Promise library can help.
• Debugging code will be more difficult since stack trace begins from the call-back instead of start of request.

Some of the disadvantages can be mitigated by using fiber (ruby fibers, node fibers), will cover Fibers in another blog.

In part 3, we will look at Callback issue in more detail and how promises can help fix callback issues.

Applications with Blocking I/O

I am assuming a simple scenario of single threaded application like Ruby on Rails Application running on a computer with single CPU. In real world, OS splits CPU time to multiple applications and does a regular context switching.

In a single threaded application like Ruby on Rails Applications, requests are processed by a single thread. When the thread makes a I/O bound call like database query or network call, application/thread is blocked even though it could be used to work on other requests.

A solution to get around the above problem is to have multiple applications running on the same box. So when one application’s thread is blocked another application’s thread can proceed with another request processing. In below diagram there are two applications competing for CPU time and second application consumes CPU between a2 and a1 time. But we still see that CPU is idle between a1 and t1 and between t2 and a2.

To avoid any idle time on CPU we can add more applications, but this will lead to unnecessary context switch which will degrade performance further. For example in the diagram below, third application is switched with second application while third application is still processing a request and needs CPU time.

As shown in the above diagram, we can notice two issues

• OS switches CPU from an application which needs CPU for processing.
• OS switches CPU to an application which is still waiting for IO.

These two issues are due to pre-emptive thread switching. To achieve optimal CPU allocation, application should be able to request for CPU time or give-up CPU time in a cooperative manner instead of pre-emptive switching.

C10K Problem

Early in 2000, a single server could not handle more than 10000 connections at a time. It was a limitation under which applications worked and developers were not able to exceed 10000 connections limit on a single box. The solution that was found is to use nonblocking I/O on each thread i.e. Non-blocking IO stared as a scalability soultion to C10K Problem.

Reactor Pattern provides an work around for the above problem using epoll.

• http://en.wikipedia.org/wiki/Nonpreemptive_multitasking
• http://en.wikipedia.org/wiki/Preemption_(computing)
• http://en.wikipedia.org/wiki/Reactor_pattern

In part 2, we will look at Non-Blocking IO as an alternative solution to maximise CPU usage.

What usually happens is we

• move the problem (or issue) from one area to other area.

  Example :– In database normalisation we improve performance of add/update by normalising but causing read performance issues.

• solve one problem at the cost of introducing another problem.

  Example :– We introduce redundancy to gain retrieval performance at the cost of multiple updates (due to redundancy).

This shows that problems are not really solved but just moved around or converted to a different problem.

But we still find and use solutions you might say. Yes, we do and we do it by trading off one type of problem to another type of problem.

So the Art of finding a solution to a problem is identifying problems you are willing to live-with and problems you can not compromise on.

In fact, I have found different solutions by just compromising on a dimension of existing solution and to find a new solution.

Reference

1. No Free Lunch theorems

2. CAP theorem

Guidelines to get Restful Web Services right

Uniform Resource handling

• Allow discoverability of new resources. Similar to adding new link to existing page (resource) to allow access to new page (new resource).
• Take advantage of intermediary components between client and server. ( Caches, Proxies, Firewalls etc)
• Follow consistent approach to view/modify/create operations on any resource.
• Make conscious trade-offs between cache-ability, discoverability, performance and convenience

Safe Methods

• Client requests are readonly.
• Client can make duplicate requests without causing unintended side-effects. Similar to loading the page again in browser.
• Does not mean server will respond with same response to the new requests. Similar to reloading(F5) a page in browser may get different/modified response (new data) on dynamic web page.

Idempotent Methods

• Client can replay the request if client is not certain server has processed the request due network failure or other errors.

Follow HTTP standards

• Use Request Methods as defined by HTTP

  • Get, Head, Options methods
    • should be safe (readonly) i.e. should not cause side effects on resource representation.
  • Get, Head, Options, Put, Delete
    • should be idempotent i.e. replay of request (due to network failure or uncertainty) should not cause issues
  • Post
    • Can cause side-effects and does not guarantee safety or idempotent
  • Use Request MIME Types to encode representation
  • Use HTTP status code for responses status
  • Keep Restful Services Stateless — maintain state in client.

• URL of the resource

  • Url of the resource represents the hierarchy.
  • For example : www.school.com/class/<1>/subject//
    • In the above example school has classes
    • classes has subjects

Frequently Asked Questions :

• Is PUT Request for Creation and POST for updating ?

  • Both PUT and POST can be used for Creating new resource or Updating an existing resource.

• Deference between PUT and POST ?

  • PUT is idempotent and client can be replay the request if network failure or system error without causing issues on server. Use PUT to completely replace existing resource representation or create new resource representation.
  • POST is not idempotent and general purpose method without restrictions and corresponding benefits. Use POST when other verbs don’t fit.

• When to use POST ?

  • POST is general purpose method which can be used when other HTTP verbs don’t fit well.

[Using] POST only becomes an issue when it is used in a situation for which some other method is ideally suited:
e.g., retrieval of information that should be a representation of some resource (GET), complete replacement of 
a representation (PUT), or any of the other standardized methods that tell intermediaries something more valuable
than “this may change something.” The other methods are more valuable to intermediaries because they say something
about how failures can be automatically handled and how intermediate caches can optimize their behavior. POST does
not have those characteristics, but that doesn’t mean we can live without it. POST serves many useful purposes in
HTTP, including the general purpose of “this action isn’t worth standardizing.”
                    --- Roy T. Fielding (http://roy.gbiv.com/untangled/2009/it-is-okay-to-use-post)

• How does HTTP Safety and Idempotent work during concurrent requests?

  • Safety and Idempotent are defined in non concurrent condition.
    • Good analogy to understand is load a web page in browser and on reloading the page browser can get new version of the page. Server could also deny GET request if agreed number of requests has been reached. User should not worry about making duplicate requests.
  • For example,
    • A replay GET request can return new representation if resource is modified by another request.
    • A replay GET request can return modified representation like updated hit/access count.
    • A failed GET request after allotted number of calls are made to a given resource, does not violate Safety rule.
    • Get request with authentication token can fail on second request. Safety does not grantee same response every time.

• Can a single GET request return two or more different resources? Can a resource contain other resources?

  • Yes, they can but it comes at the cost of cache-ability.
  • For example:–

  Car Resource

  GET /car/:licenceNumber

  {
      make : 'Toyota',
      model : 'Rav4',
      color : 'Red',                              
      owner : {
                  link : '/car/:licenceNumber/owner',
                  firstName : 'John',
                  lastName : 'Smith',
                  address : {
                                  streetName : '300 Boylston Ave E'
                                  city : 'SEATTLE',
                                  state : 'WA',
                                  zipcode : '98102'
                                  country : 'USA'
                          }
              }
  }
  

The above GET request for resource Car given a license number will return information on car, owner and owner address. Server might expose owner and owner address as resources as-well. As shown below.

Car Owner Resource

GET /car/:licenceNumber/owner

{
    firstName : 'John',
    lastName : 'Smith',
    address : {
                link : '/car/:licenceNumber/owner/address',
                streetName : '300 Boylston Ave E'
                city : 'SEATTLE',
                state : 'WA',
                zipcode : '98102'
                country : 'USA'
             }
}

Car Owner Address Resource

GET /car/:licenceNumber/owner/address

{
    streetName : '300 Boylston Ave E'
    city : 'SEATTLE',
    state : 'WA',
    zipcode : '98102'
    country : 'USA'
}

In this case, Car Resource contains Owner resource which in turn contains owner address resource. But Server exposes two other end point for owner and owner resource.

How

First Level or basic level is when someone is told what needs to be done. He/She does what is told. At this level the design is handed over to you and you are just implementing someone else’s design without fully understanding implications of design.

Change Area :– Implementation. Software Design is fixed but implementation is altered by team.

Example :– Software company is provided with design documents with object class specifications and asked to implement the software.

What

Second Level or intermediate level is when someone is given a requirement and allowed to design and implement the software without major restrictions on how the requirement is designed or implemented. At this level developer is given freedom to choose different implementation options which achieve the desired requirement.

Change Area :– Software Design and Implementation. Requirements is not allowed to change by team.

Example :– Software company is provided with use case documents or requirement documents and asked to implement the software according to specifications given in requirement documents.

Why

Advanced Level is when someone understands the business goals of the application and is allowed to choose requirements, design and implementation to achieve the business goal. As Project Managers, Product Owners, Analysts and Developers start working on designing/implementing features and requirements, they understand the business goals and why something is being build.

Change Area :– Software Requirements, Design and Implementation. Business goal can not be changed by team.

Example :– Project team is formed with co-sourced members of business and software company and Teams goes through Inception process. During Inception process, the team gains first hand understanding of business goals and how they relate to features/requirements.