Listening every day to identify project risks

There are plenty of resources available on project risk management. It’s an important topic. After all, if you mismanage risks, your project could run late, go over budget, fail to deliver its benefits, or fail to be completed at all. The topic of project risk management is often treated academically. Try reading most books, articles, or blogs on the subject and you are quickly submerged in frameworks, assessments, protocols, Black Swan theory, and Monte Carlo simulation. There is also some practical advice about holding an initial risk assessment meeting at the outset and again during the project, so risks can be logged and monitored.  Unfortunately, there are also things that happen that team members don’t really recognize as project risks, as they are situations that happen all the time and assumed to be ‘just the way it is’.

Identify risks by listening

Often, additional risks can be identified outside of formal risk meetings. Once identified, risks can be managed, so identification is critical. Project managers need to practice their listening skills throughout the project to identify additional risks. In any kind of project meeting or casual discussion, risks can be identified. Here are several examples I’ve heard at different times, and how I heard about them:

·         When we change the active directory security groups we need to make sure it isn’t month-end because there are always so many errors that users can’t work for a while. (In a meeting on scheduling)

·         At go-live, it’s always crazy trying to get the inventory to reconcile, the accountants end up doing lots of ‘plugs’ and estimates, then they figure it out correctly and fix it the next month. (In a status meeting, when the discussion got side-tracked to an unrelated topic)

·         When we launch new forms, the end users make a mess of them because they can’t figure out how to fill them in. Then we have to re-work the form and launch the revision. (At a kickoff meeting for a project to design and implement a new form)

·         We hope that Jenny never catches the flu, as no one else knows how to do that critical step we need at go-live. (At a status meeting, while explaining that a component is late due to Jenny’s lack of availability)

·         It seems like there’s an awful lot of work to be done on the go-live weekend, I wonder if it’s too tight. (In the kitchen, when the project manager was getting coffee)

·         We have a new vendor working on an internal web site for our HR department. (In a discussion about roles and responsibilities)

·         The vendor’s project manager is a sub-contractor; do you think she will keep the vendor’s senior management informed and engaged? (Informal chat, after the vendor’s project manager had been introduced)

The risks noted above did not come up in official risk identification meetings, but instead came up in other unrelated discussions.

Managing the risks

Because these came up outside of the formal risk meeting, and many were assumed as ‘just the way it is’, there was sometimes resistance when the question was asked “what can we do to manage this?” However, when pressed further, it was frequently possible to identify steps that could be taken to manage the risk. Let’s look at risk mitigation ideas for the first few risk examples:

·         If changes to AD security groups usually have a lot of errors, can they be subject to a rigorous review step before implementation, or can the testing approach be examined to see if there are gaps, or can part/all of the process be automated?

·         If it’s crazy trying to get the inventory to reconcile, how about a rehearsal, or customized reconciliation reports, or review the approach with the accounting department to see what additional controls are warranted?

·         If new forms are not well received, can they be vetted and changed as part of the project instead of after launch? Is it possible to create a user advisory panel, or run a pilot with a subset of users?

As you can see, once the risk is identified, it’s possible to come up with ideas on how to mitigate that risk.

Conclusion

Although the risk identification meetings can generate some ideas about risk, additional risks can be identified if the project manager is listening for them. Ask yourself every day: Did I hear something today that might suggest there’s a risk we aren’t managing?

Copyright 2015 Debbie Gallagher

Great by choice by Jim Collins and Morten T Hansen (book summary)

Have you read Great by Choice, by Jim Collins and Morten T. Hansen? The book describes nine years of research done to try and identify why some companies thrive in uncertainty and others do not. As I read the book, I realized that with the fast pace of change in technology, the question is relevant not just to companies, but to their IT projects as well.

The study

The authors studied the stock market for the period 1972 to 2002, and selected several companies who met three criteria: (a) stock price beat their industry index by ten times (10X); (b) the environment in which the company operated over that period was turbulent; and (c) the company was fairly small (and therefore vulnerable) at the beginning of the period.  These companies and their leaders were called 10Xers.

Then for each 10X company, they selected a comparison company in the same industry. The majority of the work was to research what was different between the 10X companies and the comparison companies to find out what could have been the cause of the 10Xer’s success.

The results of their research surprised them. Great creativity and risk taking turned out not to be key factors to success in the long run. Instead, the 10X companies and their leaders managed risk very well, and had three key behaviors: (a) Fanatic discipline; (b) empirical creativity; and (c) productive paranoia

Antarctica

The authors use the race for the South Pole in 1911 by Roald Amundsen and Robert Falcon Scott to illustrate some of the book’s findings. They liken the approach of Amundson’s successful expedition to the10X company leaders, while the comparison company leaders lean more toward the habits of Scott, whose team perished before completing the trek.

It’s an interesting step back in time and also illustrates very well not only the concepts of the study but also the relationship between risk management, success, and failure.

Fanatic discipline

In turbulent times, the companies that outperformed their industries had consistent goals and performance. Instead of pursuing massive high-growth strategies, they pushed for consistent returns every year, no matter how difficult it was to achieve. In addition, even during boom times in their industry, they held back from wild growth strategies despite pressure to do so.

Achieving consistent performance no matter what is happening in the marketplace requires concrete, clear, intelligent, and rigorously pursued performance mechanisms to keep on track. Fanatic discipline is not about bureaucracy, but about the ability to remain clear about goals and find ways to deliver consistent performance. It also requires the ability to be a non-conformist and avoid the herd instinct of the marketplace.

Empirical creativity

The most successful companies tested new concepts on a small scale and determined what worked and what didn’t work before launching significant new lines of business, markets and technologies.

These trials allowed the 10X companies to spend a lot of time and money on big launches only once they had tried the concept on a smaller scale and determined how to be successful. Alternatively, they sometimes learned that the concept didn’t work for them and avoided spending a great deal of money and resources to learn that.

Although innovation is necessary, the authors were surprised to find that the most innovative were not the most successful companies. Instead, a threshold level of innovation is required to compete in the industry, and beyond that the amount of innovation doesn’t matter very much. Instead, it matters more that innovation is paired with the ability to scale the innovation and deliver on commitments to customers.

Productive paranoia

Leaders of 10X companies prepared for the unknown and managed risks well.  They concerned themselves with what could go wrong and created buffers to deal with those known and also with unknown risks.

The leaders of the 10X companies avoided taking actions that had huge downside potential. In addition, they had the ability to look beyond daily operations to see the big picture and identify the biggest risks to their companies.

IT project risk

This book is based on research into US corporations and makes interesting references to the South Pole expeditions of 1911.  However, as I read it, I thought very often of IT projects I’ve been on, and how the lessons could be applied. Many projects fail to achieve their objectives (or even fail to complete) due to lack of discipline, trying to deliver untested concepts, or poor risk management. Examples include projects without clear objectives, sponsorship, scope, or requirements (lack of discipline). Also, there are projects attempting to launch big technological or business changes without pilots (lack of empirical information).  Probably anyone reading this can think of cases where big risks (new software, new hardware, new vendor) were taken, but the risks were either unrecognized or unmanaged (lack of productive paranoia).

Summary

The authors of the book Great by Choice have based their book on research into US companies that outperformed their industry competitors over a thirty-year span by at least ten times. They identify three key characteristics of 10X companies and leaders that have been key to their success: (a) fanatic discipline; (b) empirical creativity; and (c) productive paranoia.

They make no claim at all of the book’s relevance to IT projects. However, as I read the book, I thought the parallels were easy to see and recommend the book to those who are interested in managing the risk of IT projects.

Copyright 2015 Debbie Gallagher

It's just an upgrade

This is a true story. The company name has been changed.

The project manager was developing a project charter for his new project. He and his supervisor expected this project to be low risk and quite straightforward, as it was just an upgrade to an application that had already been running for many years in the organization. The application was used daily by approximately ninety percent of employees across the country.

The first risk assessment

The project manager met with his assigned mentor to discuss the project. At this company, it was a requirement that the project charter include a risk assessment section. The project manager included as risks some items that had been problems on previous projects, such as business resources and IT code promotion resources not being available. When questioned, the project manager said there was no particular reason to believe these would be problems on the new project, but he wasn’t sure what else was relevant to include as risks.

The real story

In discussion with the project manager, the mentor learned more about the project, which was in the planning stage prior to charter approval and project launch.

The application vendor had been slow to respond to requests for help with planning the upgrade.  The vendor was apologetic about the lack of assistance provided and explained that they had recently sold a very large engagement and were very busy as a result.

The existing application had numerous customizations and interfaces, almost none of which were documented. Upgrades of this application had been attempted before, but the projects were never completed due to the difficulty in replacing the undocumented customizations and interfaces.

The project sponsor had just gone on medical leave for several months and appointed a junior manager to act as sponsor in her place.

The mentor helped the project manager to identify three significant risks: (a) unavailable vendor resources; (b) undocumented customizations and interfaces; and (c) inappropriate sponsorship.

Actions taken

The mentor suggested several follow up actions to the project manager to try and manage the risks before launching the project.

The project manager approached the application vendor about alternatives for resourcing the project, but the answers were discouraging. The vendor had no business partners trained to perform implementations and also could not recommend any other companies or individuals who might to able to guide the company through the application upgrade process.

A business analyst was assigned to start documenting the customizations and interfaces, but the work was progressing very slowly as he was also assigned to a high-priority project.

Although the original project sponsor was available for a brief phone call, she was unable to suggest a more appropriate replacement sponsor for her expected several-month absence.

Unfortunately, none of the actions resulted in lower risks.

Updated risk assessment

After these follow up actions and further discussion with the mentor, the project manager had a much more realistic view of the project risk.

The vendor’s lack of availability during planning raised significant concerns about its ability to provide appropriate resources for the project itself. Noticeable lack of availability during the sales cycle made it clear that the vendor was overwhelmed by the large contract they had just signed and would not be focused on this company’s upgrade. Even if they managed to provide a few resources, the vendor would not be able to support them to the extent needed.

Undocumented customizations and interfaces had already caused previous upgrades of this application to fail. Proceeding with the project without developing a good understanding of the customizations and interfaces would certainly cause the project to fail once again.

The lack of an appropriate project sponsor is a risk that sounds alarm bells for any experienced project manager.  The appointment of a low-level resource shows a lack of understanding of the role of the sponsor. Someone too junior to determine business priorities and commit resources is not an appropriate choice.

Any one of the three risks described would put the project at significant risk of failure. Given all three risks, it would be nearly impossible for the project to succeed. 

Recommendations

The project manager discussed the revised risk assessment with his supervisor. The supervisor was reluctant to delay the project but did finally agree that it was too risky to proceed at that time.  However, the discussion of the risk with the sponsor would be a delicate matter, especially given the concern about the inappropriateness of the sponsor.

The supervisor arranged for the IT vice-president to have a discussion with the business vice-president (to whom the original project sponsor reported).

Epilogue

The two vice-presidents agreed that a different project was needed first, one with the purpose of documenting the existing customizations and interfaces. The IT vice-president agreed to commit resources to this predecessor project.

They expected that by the time that project was completed, the original project sponsor would be back from medical leave and the vendor would have had time to train new resources.

Conclusion

Given the risks identified for the project, the vice-presidents made the right decision. Delaying the project and starting with documenting the customizations and interfaces was the best option to set it up for success.

Evaluating project risk is frequently a challenge for junior project managers. It is typical for new project managers to focus on schedules and budgets, and develop their understanding of risks and issues at a later stage of their project management career. Providing a mentor for the project manager was a wise course of action for the company. The mentor was able to guide the project manager through the risk assessment process, leading to a more realistic evaluation of the project’s chance of success.

Copyright 2015 Debbie Gallagher

Is it really all about schedule and budget?

When I read job postings for IT project managers, there is a heavy focus on being able to deliver on schedule and on budget. In addition, I sometimes hear rather lively discussions about the role of the project manager: are they/should they be accountable for anything at all beyond schedule and budget? I’ve also observed from my own project management experience that the first questions from the sponsor and from the steering committee are invariably “how much will it cost” and “when will it be done”? Again, the focus is on schedule and budget.

Why are schedule and budget the big focus of project managers and the executives for whom they deliver projects? Is that where all the big risks are?

The big risks

In my opinion, schedule and budget overruns are not the big risks themselves. They are the outcome of other risks that have materialized into issues. So, what are the biggest project risks that drive schedule and budget problems?

In a previous article Identifying and Managing Project Risk by Tom Kendrick (book summary), I summarized Tom Kendrick’s book, in which he outlines his findings from examining hundreds of technical projects. In the book, all the risks that occurred were translated to a timeline impact so they could be compared. I think it’s a fair assumption that when the timeline is extended, the cost also grows. So, what are the risks that have the greatest impact? From Kendrick’s book, two risk areas with the greatest impact are: (a) non-mandatory scope changes; and (b) legitimate, but not anticipated, requirements gaps.

In other words, the big risks are related to scope and requirements. There are plenty of other risks, but let’s start with these two. When these risk events occur, there is an impact on the schedule and budget.

Managing scope and requirements

There are several ways to focus on scope and requirements.

First, these need to be defined as clearly as possible at the outset.  Too often there is a rush to get the project started, and it begins with inadequate descriptions of scope and requirements. This rush to delivery and related skimping on scope and requirements can lead to disastrously large impacts on schedule and budget.

Sometimes there is a good reason to have uncertainty at the start of the project.  In that case, identify the lack of clarity as a project risk. Then, make sure that everyone on the project and on the steering committee knows about that risk. Also be sure to include a re-assessment point in the schedule and funds in the budget to address the cost of potential changes.

Next, focus on requirements and scope during the project.  At the outset, meet with the project team(s) to review these in detail and address any questions. It is crucial that the team knows what is to be delivered.

Make sure there is a change control process and that it is followed.

Review and follow up on the project issues frequently. Issues are often the result of disagreement regarding what is in scope or what is needed. Too often, issues are logged in a database and neglected.

During the project there should be checkpoints to verify that the project team is on track. Hold walkthrough meetings with team members, and also with stakeholders outside the team. While often seen as time consuming, these walkthroughs are critical tools to ensuring the delivered product supports the requirements and scope of work.

An effective way to determine during the project if there are issues with scope and requirements is to ask team members informal questions about what they are working on, whether they are clear enough on what they need to do, and what problems they are having.  The project manager needs to make herself available when there are questions from the team regarding scope and requirements, and not put them off while heads-down adding up budget hours or updating Gantt charts.

Testing is another method of ensuring that the delivered product meets requirements and scope. Evaluate test plans to determine if they engage the right people and cover the right tests to identify gaps in scope and requirements. Ideally, gaps should be identified before testing, but even if found during testing, at least they are dealt with before go-live.

Conclusion

When hiring and evaluating project managers, the heavy focus on schedule and budget overlooks the relationship between schedule and budget, and project risks.  It’s usually the project risk events occurring that cause the budget and schedule problems.  Managing the schedule and budget is really about managing the project risks. In this article, I picked two top risks and discussed common techniques to manage them.

I’ll leave you with a couple of thoughts. First, I think the question about whether the project manager is responsible for anything besides schedule and budget has a definitive answer. Project Management Institute says yes, as the knowledge areas cover a lot more than schedule and budget.  Quite apart from PMI’s body of knowledge, the answer has to be yes, because if you aren’t managing the project risks, it isn’t possible to manage the schedule and budget.

My second parting thought: If the project comes in on time and on budget, but does not deliver the expected scope or meet requirements, what was really achieved?

Copyright 2015 Debbie Gallagher

Identifying and managing project risk by Tom Kendrick (book summary)

In reading Tom Kendrick’s book, Identifying and Managing Project Risk: Essential Tools for Failure-Proofing Your Project (2^nd edition, 2009), I found it a compelling read and thought I’d share my views.

The premise of Kendrick’s book is that the experiences from previous projects, specifically the historical data of project problems and their impacts can help us in identifying risk and managing those risks on our own projects.

PERIL database

Over a period of more than ten years, Kendrick collected data on project problems from hundreds of project managers. He assembled this data in his Project Experience Risk Information Library (PERIL) database. The database provides information on what types of things go wrong on projects, and attributes each to root causes. In addition, the database captures estimates of the impact of the problems that occurred. The PERIL data is mainly based on projects having a dependence on new or relatively new technology; includes mostly projects that had a planned duration between six and twelve months; and twenty or fewer project team members. 

Kendrick analyzed over six hundred projects, and has come up with an interesting approach to allow comparison. For each project, whatever the actual impact of the project issues, he has converted the impact to a time impact.  For example, if scope was reduced to meet the scheduled completion date, the timeline impact has been determined by calculating how late the project would have been if it had delivered its original scope. Kendrick’s approach may not stand up to rigorous scientific scrutiny (and he is open about the bias in the data), but in my opinion, it still provides a very useful way to compare project risks. From these risks come clear risk management strategies. 

Kendrick’s findings

Kendrick’s first analysis is high level and compares the total timeline impact in the PERIL database for eight root causes.  These include, for example, scope changes during the project, failure to meet deliverable requirements, schedule delays, internal staffing.

In later chapters, Kendrick then breaks down the categories of risk/impact into sub-categories. For example, the impact of scope changes during the project is broken down into non-mandatory scope changes, legitimate requirements gaps, and so on.

In the chapters covering sub-categories, the descriptions of issues from the project managers are included along with the frequency and impact figures. For example, end user not involved enough in requirements, supplier not meeting deadlines, system architect not available. In reading these issues from other project managers, some triggered project flashbacks!

The Panama Canal

Throughout the book Kendrick weaves stories of the Panama Canal building projects. The first project, which ran from 1879 to 1889, was a failure. Ten years after the start of the project, it was cancelled, thousands of workers were dead, the costs had spiraled out of control, and there was no canal. A failure indeed!

The second Panama Canal project started in 1902, started re-planning in 1905, and the canal opened in 1914. This project finished under budget and ahead of schedule, and was useable for its intended purpose. Although workers also died on the second project, the death and disease rates were dramatically reduced due to improved insect control, housing and other practices implemented on the project.

Although Kendrick placed the Panama Canal segments to tie each to a specific risk or approach being discussed in the book, for me they were very interesting, not just as project management solutions, but also as an interesting historical sidebar.

Lessons from the book

Although Kendrick makes project management recommendations regarding each sub-category, he does not make recommendations about which are the most critical practices for project managers to adopt overall. He leaves this to the discretion of each project manager, to be determined based on their own specific projects and environments.

In examining the breakdowns in the book, each project manager can determine what the biggest-impact problems are likely to be, either in general or on their own project, and then decide what to do about them. Every reader will likely take something different from the book depending on whether they are looking for general guidance or for insights for their own project.

An example

For purposes of this article, I decided to look at the data from the perspective of “If I wanted to improve my project risk management in just one area, where should I focus to address the project risks that occur most frequently and have the greatest impact?” So, I started by looking for the category that has the greatest number of issues and the greatest impact.  As described above, the greatest impact is from scope changes that occur during the project.  It is also the issue that occurs with the greatest frequency.

Then, within the scope change category, the two sub-categories that have the greatest frequency and impact are non-mandatory changes and legitimate requirements gaps.

My conclusion is that to improve my project risk management practices, I should start by improving scope management. By looking at the sub-categories, I see that the ways to improve scope management the most are: (a) eliminate the non-mandatory changes; and (b) reduce legitimate requirements gaps by either spending more time on requirements or by making a provision in the schedule right from the outset.

Summary

If I was looking at the data before starting a new project, I would be looking to see what might apply to my own project.

The value in Kendrick’s book is the analysis of many technical projects and the comparability of the various causes of problems.  This comparability allows each project manager to assess for themselves what recommendations can be quickly implemented for the highest improvement in their projects. 

The Panama Canal highlights provide interesting examples of the differences in project management techniques and capabilities.

Copyright 2015 Debbie Gallagher

Evidence of controls for your project audit

In previous articles, I described why your IT project might be audited, and an overview of the major steps in the audit for an application development or implementation project. In this article, I’ll provide some insight into the evidence needed to demonstrate for the audit that the project team actually executed the controls.

Examination and testing of evidence is something the auditor is required to do in order to satisfy auditing standards (see my earlier article What to expect when your IT project is audited).

Let’s follow some examples for a few common project controls. I’ll go over the kind of evidence that might be used, as well as some common problems with evidence.

Evidence that testing occurred

A common control on IT projects is that the application is tested to verify that it works and meets business requirements.

Relevant evidence that testing took place and that the application worked often focuses on the signed test scripts that were executed. These are particularly useful evidence if they include expected results, as well as a notation and signature indicating that the test passed. Additional supporting documents might be the task completion status for testing tasks on the project schedule, or a summary document capturing test pass/fail results for all scripts.

The auditor may also examine test planning documentation in order to assess the approach. For example, testing strategy, testing plans, project schedules showing testing tasks and resources, and test scripts.

As mentioned in my previous article Why your IT project is being audited, the various auditors examining your IT project may have differing scope and purpose. These differences will affect the evidence they examine. For example, if an internal audit group needed to verify that the company methodology was followed, they may need to see evidence to support specific tests mandated by your company’s own standards.

Over the course of performing many project audits, I noticed that often project teams had documentation to support test planning, but sometimes had problems with sufficient evidence supporting the test execution. This is a significant distinction and is not just fussiness on the part of the auditor. It is necessary that the evidence support the specific control.

Evidence problems related to test execution can occur when testers do not sign their completed test scripts, neglect to indicate that the test result was a pass, or even throw away their completed test results. In addition, sometimes a tester will keep scripts showing failure, but neglect to keep the follow up test results showing that the problem was corrected.

Evidence of business approval

In addition to the testing itself, it is customary for a key business representative to approve the test results for the project.

Sometimes the testing manager or project manager will ask the business approver to respond to an email, indicating approval of the test results. This approach sounds straightforward enough but in practice sometimes leads to inadequate evidence. 

A common problem with email signoffs is that requesters and approvers may treat them casually, leading to problems in using them as audit evidence later. One evidence problem occurs when the wording does not make it clear what the approver is approving.  For example, if the testing manager sends an email to the approver, “Please approve the attached,” attaches the test results, then the approver replies “Approved”, the test results are no longer attached and there is no indication in the email itself of what is being approved. 

A simple solution is to be more specific in the request email. For example, “Please approve the test results, which show successful completion of all general ledger scripts, details attached.” With this approach, if the attachment is removed in the email reply process, there is still an indication of what is being approved.

A significant evidence issue can arise when the approver adds irrelevant information to the approval email. For example, “Results ok, please let me know the resolution of the problem with ap-open.”

The approver has started an entirely new subject, as ap-open has nothing to do with the specific approval being requested. Unfortunately, there is no indication that the ap-open problem is different from the subject of the approval, so it leaves the auditor wondering how the approval could have been granted when there appears to be a problem outstanding.

The auditor has no choice, if presented with this email as an approval, except to examine the situation further and obtain further evidence that the ap-open problem had no relationship to the approval being granted.  The auditor, remember, is mandated to examine the documents to obtain sufficient evidence, and the wording of the email suggests that the approval may not be valid. This kind of email subject change invalidates the approval given until shown otherwise.

One more problem can occur with email approvals, when the email deletion policy eliminates the evidence prior to the audit! Remember to copy the approval emails to a place where they will be retained.

Evaluating your own evidence

In trying to determine in advance whether your evidence might be acceptable to the auditors, review the documents and ask yourself this question “If I wasn’t on the project team, and this set of documents was everything I knew about the whether the control happened, would I have proof that the work was performed and found to be acceptable?”  

Conclusion

What frequently happens, I think, is that project team members do perform the work and obtain the approvals.  However, in many cases, these aspects of the work may not be well documented, leading to problems in supporting the audit later.

Now that you know some of the evidence pitfalls, perhaps you can avoid them on your next audit.

Copyright 2015 Debbie Gallagher

Using a risk and controls matrix to support your project audit

In previous articles, I described why your IT project might be audited, and an overview of the major steps in the audit for an application development or implementation project. In this article, I’ll describe the IT project risk and controls matrix, which you can use to prepare for the audit.

Risk and controls matrix

A risk and controls matrix is one of the most common methods that companies and auditors use to document risks and controls. Although some companies have more formal applications, often Excel spreadsheets or Word tables are used for this documentation. The function of the matrix is simply to provide structure for the documentation. You could certainly write narratives instead, but the matrix is easy to read and is a familiar format for those who document or audit controls.

Columns in the matrix

The columns in the matrix (from left to right) generally include:

·         Risk description

·         Likelihood

·         Impact

·         Control objective

·         Control activity.

Documenting risks

First, the risks documented are inherent risks, which are the risks that exist assuming there are no controls in your environment. Keep in mind that although you may have risks that are specific to the project, there are some risks you can identify without even knowing what the specific IT project is. We are talking about inherent risk, so this is easier than it sounds.

Here’s an example, “The system may not meet the needs of the business, or deliver what management expects.”  Some companies write this in more formal language but it isn't necessary to do so. 

For each risk, indicate how likely it is and what the impact would be if the risk occurred. You can get numerically fancy if you like, but it is very common to use high, medium, and low for these indicators. Where the likelihood and impact are high, the controls should be particularly stringent, and their operation should be well documented for reference and audit. If you had no controls at all on your project, the likelihood of not meeting business needs or management objectives would be a high-probability and high-impact risk.

Control objectives

Generally, the control objectives are opposite to the risks, as they represent the outcome you want. For example, the control objective for the above risk would be, “The system meets the needs of the business and is what management expects it to be.”

No joke, it’s often as obvious as that. As a result, some companies don’t bother to document the risk and just start with the control objective. However, it is much more common to include the risk because the entire purpose of the controls is to address risk.

Control activities

The control activities are what your company or project team is doing to mitigate the inherent risk and satisfy the control objective.

For the example above, there would usually be several control activities. For example, one of the key ways to ensure that the application functions according to expectations is to test it in several different ways. You might write a control activity such as “Unit testing, interface testing, and user acceptance testing will be performed.”

Another control you likely have is formal sign off from the business user on the requirements, on the application design, or on the completed application test results.

In trying to ensure the application will meet the needs of the business and management when it is implemented, there are generally environmental controls that are enforced not just company wide, but also at the project level. For example development, testing, and production environments are restricted based on the role of the individual. In addition, you do all of the application testing in an environment that as closely as possible reflects the expected production environment. 

In ensuring the application will deliver its expected results, the project usually also trains end users to make sure they will use the application correctly and follow the associated new processes.

So, now you have a set of control activities that together can satisfy the control objective and mitigate the specified inherent risk.

Other typical risks

Now that you’ve seen a detailed example of one of the risks, here are some other risks that you may wish to include in your matrix:

·         Project does not meet the expectations of management: For this risk, the control activities are usually the project approval and project oversight processes you have. For example, project approvals by management, project advisory and steering committees, etc.

·         Systems implemented interfere with normal business operations: The control activities you might document for this risk are the system outage permission processes, as well as the updates to documentation for the application and the production operations teams. You may also obtain source code or arrange an escrow agreement for purchased applications.

·         Data converted to the new system may not be complete, accurate, or valid: This is a risk you would normally include if you are converting data from an older system to the new one.

Project-specific risks

The risks outlined above apply to nearly all IT application development or implementation projects. You may also have additional risks to consider based on the specific project you are delivering.

Conclusion

Documenting your project risks, control objectives, and control activities can help you prepare your project for an audit. The controls matrix is a typical approach used in other areas of the business and by auditors, as it provides structure for the assessment of controls.

Copyright 2015 Debbie Gallagher

What to expect when your IT project is audited

In a previous article, I described the various types of auditors that may audit your IT project, and why your project could be selected for audit. In this article, I’ll explain the major steps in the audit for an IT application development or implementation project. In addition, I’ll provide some information on what the audit is likely to cover, and some insight into audit evidence requirements.

Steps in the audit

The audit of your project is also a project, so you can expect similar steps to occur:

·         Planning and Scoping;

·         Execution; and

·         Closing.

Planning the audit

No two IT projects are the same, so before planning can start, the auditor needs to understand the project. To develop this understanding, the auditor will usually meet with IT and business representatives, as well as read background material that describes the project. The auditor will also find out what the audit requirements are. For example, if it is a financial statement audit, he or she will meet with the financial audit team to understand the areas of audit risk and what audit procedures they already have planned to cover that risk.

Based on the audit requirements and the understanding of the project, the auditor will develop a scope of work and an audit plan. Once the scope and plan are approved, the audit moves into execution phase.

Executing the audit

This step is the one you are likely most familiar with. The auditor will meet with you and other project team members to ask more detailed questions and will ask for documentation to support the answers that you provide, and to show that certain areas of the project were well controlled.

Closing the audit

At this stage, the work has been completed, and the auditor will issue recommendations for improvement, usually in the form of a formal report or management letter comments. You or someone else in your organization will also be asked to provide a response to the recommendations. After the responses are gathered, the auditor presents the report to management or to the audit committee.

Areas of audit interest

Although every project is different and the audit scope is defined for each one, there are some areas that are reviewed frequently enough that you should expect the auditor to include them. These areas of interest fall into two categories:

(1) the project itself; and

(2) the new processes and systems.

For the project itself, the auditor will usually be interested in:

·         Approvals for the project;

·         Project governance – for example, meeting minutes, scope change control, management oversight, issues management, go-live readiness assessment;

·         Testing of the new application, including unit testing, interface testing, integration testing, performance testing, business sign offs on test results;

·         Customization of the application (if it’s a package);

·         Security related to the data, the development environment, and application configuration; and

·         Completeness and accuracy of the data loaded to the new system, as well as data integrity.

For the new processes and new systems, the auditor is usually interested in:

·         Interfaces carrying data between applications;

·         Security;

·         Segregation of duties;

·         Changes to functionality and businesses processes – e.g. does the application now have multi-currency, or does it now have on-line purchase order approvals?

If you are subject to an annual audit where only a portion of the system is audited each year, the auditor also may need to change the rotation plan so that all elements are in scope for the go-live year of the new systems.

Audit evidence

According to the International Auditing and Assurance Standards Board (IAASB) Handbook, “The auditor should obtain sufficient appropriate audit evidence to be able to draw reasonable conclusions on which to base the audit opinion”. (International Standard on Auditing (ISA) 500, paragraph 2).

It goes on to describe in paragraph 7 that “Sufficiency is the measure of the quantity of audit evidence”, and that “Appropriateness is the measure of the quality of audit evidence”.

You can see that the auditor is required to ask you for relevant documentation, and will review it and test it. Testing means that the auditor performs some procedures to verify the usefulness of the document as audit evidence. There are numerous possible audit procedures available to the auditor and they are not covered here, but here are a couple of illustrations. For example, for your document that shows the reconciliation of data conversion, the auditor may select data from both systems and check that it agrees to what shows on your reconciliation document. Or, to verify your application testing, the auditor may select a sample of completed test cases to see if they have the results documented and signed by the tester.

Some of the examination of the project is done to provide insight for the auditor into areas of risk for the new processes and systems. For example, in reviewing the issues database for the project, the auditor will assess whether issues were tracked and followed up during the project. In addition, he or she will check to see if any high priority and high impact issues were still open at go live. If they are areas of audit interest, the auditor will likely follow up to determine what mitigation strategies were implemented in the new environment to deal with errors that may have occurred due to those open issues.

Conclusion

Because the auditors need to collect evidence as part of their review, you should not throw away any project documentation until after all of the relevant auditors have completed their work.

You can ask early in your project to meet with the various auditors. This should allow you to find out what areas are of particular interest, so that you can make sure your documentation is retained for their work.

Copyright 2015 Debbie Gallagher