Why didn't I know about it?

Why do most organizations, especially mid-level start-ups, have complex processes that virtually dictate how things must be done? And usually why is there a high level of discontent/contempt for processes among employees?

Because when a manager asks "There was a problem? Why didn't I know about it?" the right answer is "Because you were not there!". Sadly, the usual response is to add more cruft into the system to "ensure that management becomes aware of a problem as soon as possible".

So now you have the team wasting time preparing and sending status reports by email to the manager, whose email software automatically filters these reports to a separate folder that he ignores, until the next time when there is a problem and he asks "Why didn't I know about it?".

This time the status reports have to be consolidated by the Team Lead into an spreadsheet, with "issues" marked in red, so that the manager can see them "at a glance".

Rinse, lather, repeat, until The Process becomes a nightmare of statuses, reports and meetings that drain the resources and productivity of the start-up.

No doubt, every organization need a process to ensure that team members know what they should be doing, and others know the progress of each team. The most effective process is one that achieves this and gets out of the way. The key to this, I believe, is to maximize automation, where team members do not have to spend more than 5 minutes in their day on updating status, and all reports are generated on the fly for anyone who wants to see it.

And a manager most definitely should not require a process to know when something went wrong - he should have seen it coming before it happened.

Are humans fundamentally vegetarians or non-vegetarians?

I believe humans are non-vegetarians from an evolutionary perspective.

The animal kingdom is split into predators and prey and there is one characteristic shared by all members of each group, regardless of species. That is the position of the eyes.

All prey have eyes at the side of their head, to get a wider view of their surroundings when grazing and looking out for predators. Whereas all predators have eyes at the front of the head to focus better on their prey during a chase. This applies everywhere - mammals, fish(sharks), birds(eagles), reptiles and so on.

Guess which group humans belong to?

Building re2c on Mac/OSX/XCode

Having recently got a Macbook Pro, I installed Xcode and tried to port zenlang over to it. The lemon parser was a piece of cake, but re2c... not so much. This is using XCode 4.3 on OSX 10.7.

Having downloaded and extracted the source tarball, I tried running ./configure and got the error "C++ compiler cannot create executables".
The config.log file showed that the test file created by ./configure wasn't getting linked, it could not find crt1.10.6.

So the next step was to find this library (using find /Applications/Xcode.app -name "crt1.10.6*") add it to the environment before running ./configure, as follows:

env LDFLAGS="-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.SDK/usr/lib" ./configure

Now configure goes through successfully, but make doesn't. It is unable to find the standard include files, so I had to re-run configure with the include paths, again, discovered using find.

env CPPFLAGS="-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.SDK/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.SDK/usr/c++/4.2.1"  LDFLAGS="-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.SDK/usr/lib" ./configure

Note that we are also adding the include directory for the C++ header files (iostream, iomanip, etc) as well.

Now make should run successfully and create the re2c binary.

A couple of blogs I found suggested using macports, but I was quite sure that autotools won't let me down :)

My theory about depression.

We usually think of depression as a negative force but I like to think of it as a positive influence, and my* theory is that it has its roots in an evolutionary survival instinct developed in all creatures.

It goes like this: When an animal is under attack by a predator, it instinctively loses interest in everything else - the food it was grazing, its surroundings... the only thing it can now think of is escape and survival.

As we evolved into humans and became more intellectual in our pursuits, this instinct too evolved to the intellectual level. Hence when we are "under threat", for example insecurity borne of personal or material loss or lack of opportunities, we get into a frame of mind where, among other things, we lose interest in everything around us.

Our whole mental focus is now on whatever is bothering us and on ways to fix it. We should use this mental state to single-mindedly find solutions to the problem at hand.  Unfortunately, it often goes the other way, we just keep thinking of the problem and not the solution, and that pulls us down into the depths of despair and possibly destructive tendencies.

* I have not heard of this anywhere else, but it is possible I haven't looked hard enough.

Free will and destiny - a false dichotomy

Most people think fate and free will are mutually exclusive arguments and believe in one or the other.

I like to see them as two interdependent sides of the same coin. Free will is our ability to make choices and take our own actions in any situation that fate brings us. If you accidentally fell in a river, that was fate, but whether you drown or survive is your free will.

Fate may play a role in the circumstances we find ourselves in, but has no role whatsoever in the choices we make. That is free will, and nothing else.

Zen: A cross-platform functional programming language.

My new programming language is out! Check it out at http://zenlang.sourceforge.net/

(Update: Now moved to https://github.com/renjipanicker/zenlang)

Scrum Tools

As a lightweight process, Scrum can in theory be implemented using nothing more than a spreadsheet application like Excel or OpenOffice. However, in practice it gets tiresome and ineffective after a while, and one prefers to use a centralized, custom application for implementing the process.

Scrumworks
A good tool I have come across so far is Scrumworks from Danube Technologies. They have a Pro and a Free edition, and I have found that while the Pro edition is very much on the expensive side (249 USD per user per year), the free edition is good enough for most small companies. It has one of the best user interfaces and reporting I have seen till date.

However I do not use it mainly because it does not integrate well into other tools:

• It doesn't have an integrated bug-tracking/ticketing system, or the means to integrate into one. This is a showstopper because, while one can track internal bugs as subtasks of high-level features, there is no way to track bug reports from the field.
• Another similar reason is the lack of an integrated document management system, again a showstopper.
• The inability to comment on tasks. I believe it is very important to keep track of the historical discussions that took place around tasks; questions that were asked, and their answers, reasons for decisions, etc, and to keep it in a central location that can be accessed at any time (as against email).
  

JIRA/Confluence
The next best tool I have used is Atlassian's JIRA+Confluence. This addresses almost all of the shortfalls of ScrumWorks, but being primarily a ticket tracking system, lacks built-in Scrum tracking abilities. However, JIRA is quite extensible and this lack can be overcome by using a third-party plugin, or developing a plugin in-house.

TRAC
The third option, which I have only cursorily evaluated but never used in a production environment, is TRAC. This is an open-source application, and is, again, an extensible ticket tracking system which has plugins for Scrum. Prima facie, this seems to have advantages all around. It is a freely available open-source application, hence can be customized as much as required, and at the same time, has an integrated documentation tracking system and comment tracking mechanism. Do have a look at Buildix, a complete system based on Linux, having TRAC, Cruisecontrol and SVN pre-configured to work with each other.

I'll be sure to post an update as soon as I have explored it in greater detail.

The Sprint

The sprint (or monthly iteration) is the most visible aspect of the SCRUM process, in particular to developers, since all development effort takes place within the sprint.

The sprint starts with a sprint planning meeting at the beginning of the month where all stakeholders, including customers, product managers, marketing personnel, developers, testers, technical writers and others are present.

At this meeting, the engineering team presents a list of tasks from the product backlog that they believe they are in a position to complete in the sprint being planned. All other stakeholders participate in the presentation by giving their inputs. At the end of this meeting, everyone has a clear understanding of the work to be done in the sprint, and the end-deliverables.

Likewise, the sprint ends with a similar meeting at the end of the month, where the engineering team gives a demonstration of the features that have been added to the product during the just-concluded sprint.

Once the sprint planning is done, the engineering team becomes largely self-organizing, and is left alone to figure out how best to achieve their sprint goals. However, there are two important mechanisms to track progress and provide visibility for management during the course of the sprint. These are the daily stand up meetings, and the burndown charts.

The daily stand-up meetings.
The engineering team holds a very short internal meeting every day, known as the daily scrum. This meeting is also known as the daily stand-up meeting because it is suggested that all participants be standing through the meeting, as a mechanism to keep it focussed and brief. This meeting is called for and conducted by a designated leader, known as the scrummaster. The designated scrummaster typically is the technical lead, project manager or any of the developers in the team.

During this meeting, all participants are required to explain what they have accomplished in the past working day, and what they intend to complete in the upcoming day. They are also expected to raise any blocks they are facing (also known as “impediments”) which are preventing them, or has prevented them from meeting their goals. It is then the team lead's (scrum master's) responsibility to follow up on and resolve the impediments. The scrum master ensures that this meeting is focussed only on status updates and identification of impediments, and does not digress into an actual discussion of the impediments.

Once this meeting is over, all engineers go back to their desks to begin or continue work on their stated goals, while the scrum master works towards resolution of impediments.

The burndown chart.
Immediately after the sprint planning meeting at the beginning of the month, all teams do a detailed analysis of the tasks to be completed in the sprint, and prepare a list of subtasks they need to perform for every task. These subtasks would include all development activities, testing activities, documentation updates, etc.

Along with the subtasks, all team members also provide an estimate, specifically in hours, of the amount of time they expect to take to complete each subtask. It is recommended that the subtasks be as granular as possible, such that each subtask is between 4-8 hours.

Every day, all team members updates the estimate for the subtask they are currently working on, and replaces it with a value that represents the amount of time now left on it. This is important: at all stages, the estimates, whether original or updated, must represent as accurately as possible, the amount of time left to work on it.

A snapshot of this data (the estimates) is collected every day, and plotted on a graph with date on the X axis. Ideally, this graph should show a line going downwards from left to right, something like the one shown below (copyrighted image from www.scrumalliance.org):



As can be seen, this graph provides a very clear picture of the state of the sprint to all stakeholders. If the graph is moving horizontally, or upwards, it could be an early indication that all is not well, and that the team might not meet their sprint goals.

In my experience, a very effective usage of the burndown chart is for an automated system to generate them once a day, and and immediately email them to all parties.

Estimation and Velocity

An iterative development model enables the engineering team to provide management with more accurate estimates for upcoming tasks, and consequently help management make more accurate plans and projections. This entry describes how.

When creating a project plan following traditional waterfall techniques, the engineering team or analysts typically do so by evaluating the scope of the tasks and the capabilities of the team to arrive at a number, usually in man-days or man-months, for each task. Next, the dependencies between tasks are considered and serialized accordingly to arrive at start and end dates for each task and the entire project.

This approach has two key disadvantages. The first disadvantage is that it makes the project plan inflexible and limits the management's options for speeding up the pace of work.

The second disadvantage is that the team's ability to perform at a certain rate can change due to unforeseen circumstances, such as an unexpected resignation or vacation by a key member, and this will affect the entire project plan. The productivity level of a team can change during the course of a project, specifically, an unexpected loss of a key team member can slow down the project, or perhaps an unforeseen technology requirement comes up which involves a learning curve and slows down the team, and so on.

When following the waterfall model, the options available to deal with these situations are expensive and/or detrimental to the quality of the end-product. For example, management may choose to drop or more affected features, which will require rework at almost all levels: planning, design, development, test plans, customer expectations, etc. Alternatively, one may choose to find a workaround to the unexpected technical obstacle, which might compromise the product quality, and result in increased post-release costs.

The first disadvantage: An inflexible project plan.
On a side note to start with, it is often said that “you cannot put together a group of 9 women to produce a baby in 1 month”. It is no coincidence that this analogy is usually stated by people following the waterfall model, where the project goes through a series of steps, and each step is completely dependent on the previous one, and where the final release can be made only at the end of all these steps.

A careful study of any waterfall project plan will reveal that dependencies are inevitable; a result of phase-wise planning. Clearly, you cannot start the design phase until the requirements phase is complete. You cannot start the development phase until the design is complete, and you cannot start testing until development is complete.

However, the woman-baby analogy does not hold good with the iterative model which requires that the project be split into independent tasks. Very rarely do we experience dependencies among individual features of a project. So by planning the project around features instead of phases, the organization has eliminated a good percentage of the dependencies right there, and makes the project more amenable to being executed in parallel.

If the iterative model has to respond to the woman-baby analogy, it would be “Yes, it is possible to have 9 women work for 9 months to produce one baby each, thus achieving an effective rate of 1 baby a month”.

The iterative model provides a very effective and flexible mechanism to increase productivity rates by increasing team strength.

The second disadvantage: Changing productivity levels.
An iterative process makes it very easy to detect whenever the productivity level of a team changes (due to loss of team member, or unexpected learning curve), and to rework the projections.

The key is, the team does not estimate the tasks directly in man-days or man-months. The team estimates the complexity of the tasks, instead of approximate time required. The exact scale against which complexity is estimated can vary from organization to organization. Some organizations may use a linear scale, like 1 to 10, while others may use exponentially increasing discrete values, like 1, 2, 5, 10, 20, 50, 100 etc. The important point to remember is that the scale and the estimates must be consistent all through the lifetime of the project. For the sake of discussion, we can call these values “complexity points”.

Once this estimation of complexity is complete, it becomes easy to track the number of hours required per point. This figure is called the velocity of the team.

As an example, if we consider a team of 5 people, we can assume 800 man-hours available per month (160 hours x 5 people). If this team is confident that they can consume tasks whose complexity points add up to 100, we have an intial velocity of 8 hours per point. As more and more iterations are completed, the velocity keeps getting revised for greater accuracy.

This approach makes it easy to monitor and optimize the performance of development teams. Management can introspect on questions such as:

• “Can we decrease the number of hours per point?” - For increasing performance.
• “Can we increase the points consumed per month by adding more people?” - For increasing production rate.
• “How many tasks can we complete for the next quarterly release?” - For making projections.
• "By when do we expect to make feature X available?" - For responding to market queries.
  

This approach also ensures that any change in the productivity of the team does not require the engineering team to revisit the estimates for all the tasks.

Benefits of Iterative Development

The iterative approach of SCRUM provides 3 key benefits to the organization.

1. Sets a Rhythm: It helps the organization to quickly establish an internal development rhythm, and all cross-functional teams begin to work by this rhythm.
   
   The development teams gets into the mode of planning and delivering a release every month with new features, the business analysts and product owner gets into the habit of ensuring that the highest priority backlog items are completely well-defined in time for the next sprint, and the customer learns to expect a new release every month, with added functionality that can be objectively seen, commented upon and accepted.
   
   Month after month, the rhythm is maintained, and soon management is able to identify key metrics for which data can be collected, monitored and fine-tuned.
   
   
2. Focuses on tangible deliverables: The waterfall model is also broken into multiple phases, but the deliverables at the end of each phase are intangible artifacts, such as requirement specifications, design documentation, project plans, test plans, untested code, etc.
   
   The only tangible delivery ever made in the project is when the product is released to the customer. no doubt, there is value in all the various artifacts generated at the end of each phase in the waterfall model, and indeed, many of these artifacts are also created and used in iterative processes.
   
   However, they are merely an aid to the development effort and nothing more. A customer has no use for a design document, nor will the company's share price go up because a project plan is complete. On the other hand, being able go to market with the first version of a product ahead of the competition, or being able to achieve faster turnaround times for bug fixes and new features will definitely go a long way towards keeping customers and shareholders happy.
   
   An iterative development model keeps everyone focussed on the software as the ultimate deliverable.
   
   
3. Emphasis on timebound delivery: The 3 main factors that impact the success of a product is features, quality and timeliness, and planning a project is typically an exercise in achieving the right balance between these 3 factors; one factor cannot be changed without having an impact on at least one of the others.
   
   The key difference between a waterfall-based project and an iterative project is that the former has a greater interest in keeping the feature set intact, while the latter is time-bound. That is, an iterative project places the emphasis on making a release at the specified time, even if it means shipping with one less feature.
   
   A company might, for example, have a policy to release a new version of their product every quarter. With such a policy, the organization can confidently make an announcement to their customers that the next release will be made on a particular date, with a certain set of features. All cross-functional teams such as development, QA, business analysis, technical writing, etc are in tune with this policy, and work towards maximizing the number of features and bug-fixes they can complete and integrate in time for the next quarterly release.
   
   Now, if there is a delay in one of the features, or something changed to make a feature redundant, the deadline is not affected. The organization can still make a release on the announced deadline, perhaps with a feature or two less. The delayed features are subsequently released in the next quarter.
   
   

Iterative Development

An iterative development model such as SCRUM is a very powerful mechanism for change control. In any project managed using typical waterfall models, every change proposal had to go through an evaluation process to estimate it's impact on the project deadline. Sometimes, even if a proposed change is beneficial to the product, it gets rejected because of the amount of rework to be done by the development team and/or its impact on the deadline.

An iterative process, on the other hand, deals with this problem very elegantly by maintaining a narrow time window within which specifications gets frozen, and development work is done only within this window. This window is an iteration, known in SCRUM as a Sprint, and is typically 1 month in duration. The other teams may modify any task on the backlog which has not yet been taken up by the development team.

At the start of an iteration, the development team looks at the Product Backlog (which has the highest priority tasks at the top) and identifies a certain number of tasks from the top which they believe they can complete within the one month sprint. A more detailed explanation of this estimation technique is given later.

Once the tasks to be undertaken in the current sprint (or iteration) is identified, they are moved into the sprint backlog and the team starts working on them. At this stage, none of the other teams and stakeholders may subsequently alter any of these tasks.

The work done on a task within the sprint is a mini-project in itself, and typically covers all stages of development, from design, development, testing, documentation and integration.

The Product Backlog

Managers who have worked with products that are well-established in the market would be familiar with the maintenance pattern, where all work becomes a series of new feature additions, change requests or bug fixes on that product.

They would also know that it is in fact much easier to manage a project which is in this stage, because all work is defined as a series of discrete tasks that can be individually assigned, tracked, developed and released. Progress and performance can now be measured numerically, based on the number of tasks completed and the complexity of each task, as in any other industry.

In contrast, a project for developing the first version of a product tends to be more chaotic, more so when developed using one of the traditional waterfall models.

• Multiple features are specified, defined and estimated by the project management team.
• The engineering team tries to design and develop all these features in one big step
• QA tries to test them all in another big step, and
• the customer expects to see only the end result which consists of all these features.

A delay in any of these steps affects the final deadline that has been committed to the customer.

Using SCRUM, one aims to bring the project to the “maintenance state” described above as early as possible, so that all subsequent work can be defined as a series of tasks in a pipeline.

This is done by identifying the bare minimum functionality required to demonstrate a working application. Having reached this state, all subsequent work is defined as a series of tasks that adds more functionality to the existing application.

Consider the example of a scientific calculator software. One can define the baseline product as a simple calculator, capable of performing basic arithmetic operations such as add, subtract, multiply and divide. Clearly, this baseline product is too immature to release to the general market, but it can definitely be released to various internal stakeholders.

For example:

• The QA team (and perhaps beta testers) can start testing for obvious bugs and edge cases.
• The business analysts can either verify that the product is in line with their expectations, or make changes to the original requirements they had defined.
• The user interface and usability experts now has a concrete application to provide recommendations against, and so on.

The significant benefit here is that all stakeholders can start providing inputs, and the company has the flexibility to make changes to the original requirements of the product based on these inputs.

It is important to note that all requirements are defined as a series of tasks. Each task can, at a minimum, be either a new feature to be added, or a change to an existing feature, or a bug fix. The job of the project manager now is to prioritize and manage this task pipeline efficiently.

In the calculator example above, subsequent feature requests could be for implementing square and square root functionality, sin and cos functions, etc, such that each of these functions become a separate task in the pipeline.

In SCRUM terminology, this pipeline is called the Product Backlog. Despite "Backlog" being a word with a slightly negative connotation, it is apt. The Backlog is the list of tasks that should be completed for the next deliverable version of the Product, and becomes the central axis around which all development efforts revolve.

Flatlands

A book that changed the way I look at life. Edwin Abbot was a man ahead (or later, depending on where you are from) of his times. In simple everyday language, he brings to the average person the concept of dimensions, something that's otherwise in the realm of mathematics and physics.

So what is great about this book?
He has painted a beautiful image of dimensions and trans-dimensionalism. He makes you imagine a world that exists in only 2 dimensions and how a person living in this world tries to imagine a 3rd dimension.

If you are a person you is trying to figure out the world, this is a must-read. Check it out sometime. Its available on Project Gutenburg.