Monday, September 13, 2010

Subversion - Managing exclusions in commit hooks

Subversion is now a widely used version control system. It is easy to setup, supports branching and merging almost effortlessly and has a few good client tools (like TortoiseSvn) and plugins for major IDEs that make life very easy for users.
Subversion stores content in what are termed as repositories. One can visualize a repository as the base folder inside which various code lines exist. These can be trunk (the main code line), branches and tags (a tag usually corresponds to a release).
Subversion also has a concept of hooks which are programs (usually scripts) that can be invoked at specific events. There are hooks for pre-commit, commit and post-commit events. Hooks can be used for a variety of purposes e.g. checking for a proper commit comment, running static analysis tools and fail commits on violations (this is usually better done in CI but sometimes might need to be done in a hook), code format checking etc.

Activating and Deactivating hooks for specific artifacts
Hooks exist at the level of a repository. What this means is that if a pre-commit hook exists, it will be invoked on each commit transaction to any code line within the repository. Sometimes though, certain code lines or projects or files need to be excluded from the checks built into the hooks. In other words, one might need to "activate" or "deactivate" hooks for certain commits. This "activation" or "deactivation" is not available out of the box in SVN and needs to be coded and built in. Here's an example of how one can achieve the same. The example below uses bash shell script but can be in any language.

Let's take an example of a pre-commit hook that needs to be deactivated for a few branches, one project and one file in a specific branch with a certain word in its name. A pre-commit hook is invoked as part of a commit transaction just before the transaction executes and becomes a revision. If the hook executes successfully and returns 1, the transaction commits and a new revision gets created. If it aborts or returns 0, the transaction is rolled back by the SVN server.
A pre-commit hook is called by the SVN server with two arguments, the first is the absolute path to the repository and the second is the transaction id of the commit transaction in question. With this information in hand, the script can be made to find the changes that are part of a transaction using the svnlook command. The command for the same is:
svnlook changed -t $2 $1 ($1 is the repos path and $2 is the transaction id)

With this information now available, the hook can be made to ignore certain files, projects etc from its processing.
The example here uses a flat text file where each exclusion is listed in a line. An example exclusion file could be:

The hook can then be written as:


    while read line
        if [[ ${path} =~ ${regex} ]]
    done < ${exclusionFile}
    return ${exclude}
CHANGED=`${SVNLOOK} changed -t "${TXN}" "${REPOS}" | /bin/grep -v ^D | /bin/awk '{print $2}`  
# (ignoring deletes)
for artifact in ${CHANGED}
    if [ $? -eq 0 ]
        #do processing here
The benefit of taking this approach is that exclusions can be configured and maintained easily without any change to the hook itself.

Thursday, September 9, 2010

Holes in Application Architecture - Hidden single points of failure

Most enterprise applications and almost all customer facing web applications have high availability as a stated or implicit requirement. Ensuring high availability, among other things, also means building  in redundancy for those parts of the system which can become single points of failure.

While defining the deployment and application architectures of highly available systems, one needs to take both an outside-in and inside-out view of the system to identify all possible single points of failures. Many times, while defining the deployment architecture , there is a tendency to take just a outside-in view of the system. Same goes for the case of defining application architecture aspects related to high availability. This approach takes care of building redundancy for the "obvious" single points of failures such as web server, app server and database and many times that is good enough. Sometimes though, due to the nature of the application, components of the application interact with external entities and services which are deployed separately in their own deployment units. To ensure true high availability, redundancy needs to be built for such entities and services as well. Identifying them needs an an inside-out view without which these might just get the miss. 

In this post, I use the context of a web application that is accessible over the internet to highlight a few such hidden single points of failure.

A highly available web application is typically deployed in a setup where there are four vertical tiers:

  • Tier 1 is a load balancer (a hardware one quite often). 
  • Tier2 is a web server farm with the web servers also acting as reverse proxies.  These web servers are also configured to support failover to another application server if the one that a request is sent to fails to respond. 
  • Tier3 is a cluster of application servers.
  • Tier4 is a set of database servers (at least 2 in either active-active or active-passive mode). 
  • In case the application has filesystem dependencies (a video sharing site for example), the filesystem is mounted from network storage on individual application web server and/or application server nodes.

This setup takes care of all the obvious single points of failure. What it does not take care of are the not so obvious ones. Some of these are:

Outbound http calls
Application sometimes need to make calls to other urls on the internet. These could be calls to get RSS feed data or could be web service calls to third party services such as spam detection services or other services such as Feedburner or a host of other things for that matter. Quite often, security guildelines mandate that applications cannot directly make calls to internet urls and must get the requests proxied through proxy servers. In such a case it's important to  build in redundancy for proxy servers and applications should be coded correctly to resend requests through another proxy server if one fails.

Single Sign On
Some applications delegate authentication to a separate SSO solution (e.g. Siteminder, CAS). Allowing a user to login to an application via SSO usually means the following sequence of events:
a) redirecting the user to a login page hosted by the SSO application
b) If the login is successful, the user request then presents a token to the application requesting it to be allowed access. This token is generated by the SSO application.
c) Making a "behind the scenes" call to the SSO app to validate the token
The third step is a potential single point of failure. Assuming SSO has multiple instances for redundancy and the instances are clustered, the application should be coded correctly to contact a different SSO app instance in case the first doesn't respond.

Antivirus checks
Some applications which allow users to upload binary content (videos, audios, images etc) may need to run the uploaded content through an anti-virus check and quarantine the file if found infected. If the AV check is done via a service invocation on a remote process (which is usually the case), the application code should take care of invoking the service on another instance if the first instance is not responding.

Some applications need to send emails to various kind of recipients. This means talking to a SMTP server either directly or via a relay (e.g. a sendmail relay on Linux) . In either of the cases, assuming redundancy exits for the SMTP server or relay, the  applications need to be coded to send the email content to another instance if the first instance doesn't respond.

These are some example from my experience. I am sure there would be a few more. Thoughts on what those could be?