The changing role of the technology team in ERP implementations (part 3)

In two previous articles, I described several areas where the role of the technology team in ERP implementations has changed in the last ten to twenty years.  Here is part 3, the final installment.

Single sign on

This capability makes life much easier for the end user. Although it is easier to implement than it used to be, it still requires set up, testing, and support by the technology team.

 

Disaster recovery

In a cloud environment, you won’t be managing databases and servers any more. The scenario where your entire data centre collapsed and you lost all your systems at once will change to multiple application-based or vendor-based scenarios. You need to re-think what disaster recovery planning and testing should cover. You also need to verify that participation in disaster recovery planning and testing exists in your contracts with cloud vendors.

IT controls

If you’ve been outsourcing some of your IT functions already, you’re familiar with this. Although your organization is responsible to ensure IT controls are operating effectively, many of those controls will now be implemented and operated by your cloud vendors. Relevant IT controls will need to be reviewed internally and with the vendors to identify if any controls need to be re-designed, and how they will be tested. Selection of your ERP system and negotiation of the contract needs to consider IT controls and associated Service Organization Control (SOC) reporting requirements.

Application support model

With ERP in the cloud, the internal support team no longer needs to be heavily weighted toward database analysis, operating system, and hardware experience. Instead, the team’s focus will be on the business process, how the application supports the business process, and reporting and data governance. This is a significant shift in mindset and skills for many IT support managers and staff.

If the existing ERP is supported by manual processes, your business users will certainly want to automate more when the new ERP is implemented. E.g. on-line workflow to replace email approvals. As a result, the number of ERP users you support is likely to be higher than it is today.

Vendor management

Given the number of areas where your IT organization now relies on vendors (ERP, integration, disaster recovery, IT controls), it is evident that a key responsibility and skill set in your operational support model is vendor management. This may be an area in which your team requires bolstering if you have not been outsourcing or using cloud very much to date.

Conclusion

Technology is so ingrained in business processes today that there’s no such thing as a fully manual business process any more.  Having the business define the new processes and handing off requirements to the technology team is not a workable process. Business analysts have to participate fully in the business process workshops, identifying requirements along the way for everything from mobile device needs to interface and data conversion requirements.

In addition, your technology team for the ERP project, whether internally available or contracted, needs to support a wide variety of requirements that didn’t often exist two decades ago: mobile technology, browsers, firewalls, single sign on. In addition, the approaches to disaster recovery, IT controls, and application support likely need to change.

As you can see from this article and the previous two, the role of the technology team in ERP implementations has changed significantly. So, if you are implementing ERP soon, and it’s been ten or twenty years since you did so, you should anticipate and plan for all or most of these differences.

The changing role of the technology team in ERP implementations (part 2)

In a previous article, I described several areas where the role of the technology team in ERP implementations has changed in the last ten to twenty years.  Here is part 2.

Importance of data

ERP and other application implementers have historically considered focus on People, Process, and Technology as the foundation for success. In the last several years, it has become apparent that Data is a fourth and equally important element.

Data, including its definition, conversion, integration, and governance have become critical to the success of the implementation and the ongoing usability of the ERP system. Although the data is owned by the business, they generally require support from the technology team in supporting data governance and enforcement of data standards.

Downstream systems

Due to changes in master data or the business meaning of data when you implement a new ERP, combined with the integrations, you may have many changes to data in applications you connect to/from.

One of the causes will be data re-numbering. For example, if you change significant master data such as customer number, vendor number, or part number, you will likely have a business requirement to update existing data in other systems besides the ERP.

In addition, when implementing ERP, the business may take the opportunity to change the meaning of the data. For example, the existing ERP may have Acme Toronto, Acme Vancouver, and Acme Calgary all set up as separate vendors. In the new ERP, the business may choose to eliminate these three vendors and instead create a vendor called Acme Canada with three sites in Toronto, Vancouver, and Calgary. You can see that this would have an impact on the way that integrations to/from other applications are designed; and the potential for changes to existing data in those systems.

Data cleansing

Data often requires cleansing in order to accommodate a new ERP. In addition to correction of issues with existing data, the business may want to develop data for the ERP that exists only in spreadsheets today. The technology team is often called upon to support data assessment and may also need to provide tools and processes to support data creation and standardization. 

Workflow

Workflow can enforce controls and retain documentation of approvals, and is built into modern ERPs. In a cloud environment, the notification emails will be crossing multiple corporate firewalls. Similarly, if you are using your ERP host or another external party to provide integrated services such as invoice scanning, there will be technology-enabled processes crossing internal and external environments. Your technology team will need to be involved in the security design and implementation.

Customizations

If your existing ERP system is customized, you know what a challenge it is to keep up with upgrades, as modifications need to be re-established with each upgrade. With cloud ERP, it is easier to push back on the business when they want to customize, as cloud lends itself to adherence to the vendor’s standard offerings. However, where there is a real business need, you may need to develop the customization outside of the ERP, leading to another system and interface to manage, and the necessity of supporting business questions on differences between the two systems.

System cutover

Along with the transition of business functions to the new application and processes, there will be a number of technical steps involved as well. Cutover tasks may include turning off/on interfaces and scheduled jobs, and providing updated charts of accounts and other master data (e.g. customer numbers) to other systems and business partners. It is also common to prevent data updates in legacy applications by modifying user roles when the new ERP goes live.

Stay tuned

There is more to cover on the changing role of the technology team in ERP implementation. Watch for part 3.

Project risk case study: Scarce resource management

Technology projects are heavily dependent on expert resources from the IT team and the business. Managing people whose work is critical to the success of the project is essential and managing their time can be challenging.

This case study is a situation where a business expert was proving to be unable to deliver to the timeline.

Case study

The project was development of custom reports. When engaging resources for the work, the project manager did the usual things to ensure availability of the resource: for example, arranged for dedicated team members, and obtained commitments from their managers that they would be free to concentrate on the project. For those who were not vendors, the project manager arranged for the individuals’ jobs to be filled by other workers so the project team could concentrate on the project work.

The situation

The development work proceeded but there were challenges and issues along the way (perhaps a case study for another time).  Due to the development issues, there were quite a large number of problems found in the testing phase. 

However, just as problematic, it was taking a very long time for the business expert doing the testing to identify the issues on the reports, and to re-test as defects were corrected.  As time went on, it became clear that the tester was a significant bottleneck in the process.

The project manager discussed the pace with the tester, who confirmed that he was concentrating on the testing tasks, and had not been pulled away to work on other non-project work. The tester claimed that it was just a temporary delay due to learning curve on what needed to be done, and that he would have no problem catching up.

Unfortunately, the tester’s prediction of increasing speed did not come true. The testing continued to take much longer than planned.  In addition, the tester claimed, and his manager agreed, that no one but himself was qualified to examine the reports and determine whether they were right or wrong.

Evaluation of the problem

The project manager decided to see for herself what was causing the delay, so she arranged to sit in the tester’s office while he worked. She did some other work, but by sitting close by was able to observe the tester’s process.

What she observed was that there were many time-consuming steps to complete the tester’s work.  He printed the legacy report; then printed the new report. Then, he went through every line and checked every heading, description, and number for accuracy.  Then, when he found a difference between the legacy and new report, he had to determine whether the difference was acceptable. If it was not acceptable, he had to log the defect in the tracking software.

The project manager realized that much of the work could actually be done by others. The only part of the work that the business expert tester really had to do for himself was to evaluate whether a difference was acceptable or not.

New approach

The project manager arranged to add two junior staff to the project. They were assigned to print the legacy and new reports, compare every heading, description, and number, and mark differences with highlighters and tape flags for review. 

The business expert was now able to focus on the one thing that no one but himself could do. He focused on evaluation of the differences between the legacy and new reports, and decided whether the difference was acceptable or a defect.

The project manager also arranged for someone else to log the defects for the tester, so he could concentrate on just the evaluation.

Conclusion

The project manager’s observation was essential to resolving the work bottleneck. Each report was taking two to four hours to test, but the portion that only the business expert could do was really only fifteen to thirty minutes.

By re-allocating much of the work to other resources, the business expert was freed up to complete the evaluation for many more reports.

Although their participation is critical to a project’s success, business experts can become a bottleneck on a project. In order to ensure the proper participation of the expert while also eliminating the bottleneck, it was necessary for the project manager to evaluate the work and break it down into its component parts. Then many of the steps could be completed by others, freeing up the expert to focus on the task for which he had the expertise.

Introduction to IT Project Risk

While working on a variety of IT projects, I’ve seen several project charters and risk registers. Over time, I’ve noticed that not all project managers have a solid grasp of how to identify and mitigate project risk. So, here’s an introduction to the subject.

These aren’t project risks

Perhaps it seems an odd place to start, but the most common problem I see is that many risks identified aren’t really risks.  So, what is not a project risk?

An issue is not a project risk.  An issue is something that is already a problem for the project. E.g. a risk item that states: Vendor is late delivering components. Since the vendor is already late, or you already know the delivery is going to be late, this is an issue, not a risk. A risk is something that could happen, but hasn’t happened yet.

An outcome is not a project risk. E.g. a risk item that states: The project might not go live on time. This statement is an outcome of one or more events that could happen, for which the outcome is that there could be an impact on the project timeline. A risk is the thing that might cause the project to go late or cost more, etc.

Also, a vaguely-defined concern isn’t a risk. Risks aren’t really properly identified if they are not specific. E.g. a risk item that states: Testing is a risk. This statement is too vague, as it doesn’t say what specifically about testing is a risk and why it is being identified.

These aren’t mitigation plans

In addition to incorrectly identified risks, I often see vague mitigation plans. E.g. Manage dependencies and milestones. E.g. Leverage vendor’s experience. These mitigation plans are too vague to be actionable.

What are we worried about? What will we do about it?

Every book on project management has a risk definition along the lines of “Project risk is an uncertain event or condition that, if it occurs, has a positive or negative effect on one or more project objectives such as scope, schedule, cost, and quality.” (PMBOK, Fifth Edition). Following that, there’s a definition of risk mitigation, such as: “Risk mitigation is a risk response strategy whereby the project team acts to reduce the probability of occurrence or impact of a risk.” (PMBOK, Fifth Edition)

Although accurate, these definitions can be simplified to aid understanding. The easiest way to identify risks is to ask yourself and others: What exactly are we worried about and why are we worried about it?

Then, once the worry and rationale are very clearly defined, the mitigation plan is based on:  What very specific actions are we going to take regarding that worry?

An example

Do you have reason to believe the vendor might deliver late – before it happens? For example, maybe the vendor has recently made a lot more sales than usual and you’ve had late deliveries when you’ve bought from other fast-growing vendors. If so, you can record the risk very clearly: Vendor has gained significant market share in the last three months. Based on our experience with other vendors, this market gain can have an impact on manufacturing capacity, which may result in late delivery.

If you want to avoid this event, think about what options you have to avoid your project being derailed by late delivery. For example, if the contract isn’t signed yet, can you build in a penalty clause, or pay a premium for on-time? Can you spend extra effort on vetting your requirements to ensure your own project doesn’t cause any impact on the vendor? Do you have access to other equipment you could use temporarily if the delivery is late? Does that strategy lead you define an additional risk and mitigation plan? Ask others for ideas on how they’ve managed this kind of risk before and how successful the strategy was for them. Be specific in your mitigation plan: go beyond platitudes such as good project management.

Do you have other worries?

Anything that causes you a specific concern is a candidate to be included as a risk.  Risks are as varied as the projects on which they arise. Here are a few examples I’ve seen: (1) Servers are being moved from a company-run to an external data centre during the same time period that our project needs to test connectivity for several interfaces. Testing connections to the old data centre will need to be repeated for the new data centre, with potential for delay to our project’s timeline. (2) Although the on-site project manager assigned by a major vendor has great leadership skills, he’s not particularly rigorous about scheduling and tracking against the schedule. Since he is managing a large team and a tight delivery timeline, we are concerned that we won’t have adequate visibility to progress or lack thereof. (3) Previous upgrades to this application have failed due to lack of understanding of the integration points.

Notice that all of these risks are very specific, and they provide a rationale for why the risk has been identified.

Schedule and follow up

Assign a responsible person and due date for every mitigation action item. If you aren’t able to assign a person and due date, the mitigation step probably isn’t defined clearly enough. Track these mitigation steps to make sure that they actually get accomplished.

Conclusion

Understanding of IT project risks and mitigation plans can be improved by simplifying the definitions to: (a) What you are worried about exactly and why; and (b) what you will do about it.

It is important to be very specific and provide the rationale for the risks. The mitigation plan needs to be detailed enough that each task in it can be assigned to a person and on a specific date. 

Project risk: Vendor management case study

Working with vendors is a fact of life on technology projects. It makes sense to engage vendors to work on your projects. They have product experience with many clients and can bring best practices in addition to technical expertise.

However, vendors are not your employees and it is wise to remember that their goals and your company’s goals are not always completely aligned. As a result, in addition to the benefits, there are risks in bringing on vendors to deliver your projects.

Case study

Here’s an example of a project gone off the rails due to issues with selecting and managing the vendor.

The client had issued a Request for Proposal (RFP) and selected a vendor to implement a new application and its supporting technology platform.

The vendor started missing deadlines early in the project. The client’s requirements for performance were not met. To resolve the performance issues, the vendor proposed hardware changes. The resulting procurement and installation time caused more delay.

Unfortunately, installation of the new hardware did not resolve the performance problems, so the vendor embarked on a series of changes to try and improve performance.  Time after time, the vendor said the performance changes would be done by the following week, but every time they were unable to deliver the improvements.  It was clear that the vendor’s team was working hard, including evenings and weekends, but still the system could not operate within the necessary timeframe.

After months of promises followed by non-delivery, the vendor asked for a three-month period to achieve the necessary performance. The client agreed to the three-month timeline, but once again, the performance metrics were not achieved.

At this point, the vendor asked if the client would make changes to other applications in the same business process. As long as all of the applications fit into the overnight process, the vendor’s own application performance would be of less concern. The client agreed to modify other applications, which caused further delays while requirements, development, and testing took place.

Unfortunately, the changes to the other applications were not sufficient. When done, it was clear that the vendor would still have to significantly improve the performance of its own product.  

The vendor continued to work at application changes and database tuning to try and correct the performance issues. When that failed to achieve the desired results, the client agreed to examine the business process to see if they could change it to accommodate the product’s performance issues. The process re-design did not create enough efficiency to solve the problem.

If you’ve ever been in this position, either as vendor or client, you know it’s a very unpleasant situation. The client and vendor blame each other for the problems. Millions of dollars are spent. The business does not have their new application. The vendor is losing money on the project. Discussions about legal action take place.

Causes

There were several problems with this situation.

In the RFP, the client did not specify performance requirements. This lack of guidance allowed the vendor to avoid considering performance when creating the proposal.

The vendor was a solid, well-established company, with an excellent reputation for delivery and support. However, the product was new, with no installations in the client’s industry or any other industry. When evaluating the vendor proposals, the client did not recognize the importance of the fact that the references provided by the vendor (for other products) were irrelevant to the project they were proposing.

The vendor’s size, stability, and reputation were a good thing, of course, in that the vendor was motivated to deliver to the client’s satisfaction, and had the financial resources to invest in efforts to correct the problems.  A smaller, less reputable vendor may have simply walked away early on.

As delays continued over many months and millions were spent, the client never had any discussion regarding whether the project should be cancelled and alternative products evaluated.  This inability to face the failure of a product and project is common. Many clients, once they’ve invested a lot of money, time, and resources, do not admit defeat and start over.

Conclusion

The interests of the client and vendor were never aligned right from the start. The client was looking for a product to install and add value to their business process right away.

The vendor was looking for a successful installation of its new product, so they could use it as a reference when selling to other clients.

If the client had realized up front the differences in their goals, they may have been able to negotiate an approach that worked for both. Instead, both the client and vendor have failed to achieve their objectives.

Listening every day to identify project risks

There are plenty of resources available on project risk management. It’s an important topic. After all, if you mismanage risks, your project could run late, go over budget, fail to deliver its benefits, or fail to be completed at all. The topic of project risk management is often treated academically. Try reading most books, articles, or blogs on the subject and you are quickly submerged in frameworks, assessments, protocols, Black Swan theory, and Monte Carlo simulation. There is also some practical advice about holding an initial risk assessment meeting at the outset and again during the project, so risks can be logged and monitored.  Unfortunately, there are also things that happen that team members don’t really recognize as project risks, as they are situations that happen all the time and assumed to be ‘just the way it is’.

Identify risks by listening

Often, additional risks can be identified outside of formal risk meetings. Once identified, risks can be managed, so identification is critical. Project managers need to practice their listening skills throughout the project to identify additional risks. In any kind of project meeting or casual discussion, risks can be identified. Here are several examples I’ve heard at different times, and how I heard about them:

·         When we change the active directory security groups we need to make sure it isn’t month-end because there are always so many errors that users can’t work for a while. (In a meeting on scheduling)

·         At go-live, it’s always crazy trying to get the inventory to reconcile, the accountants end up doing lots of ‘plugs’ and estimates, then they figure it out correctly and fix it the next month. (In a status meeting, when the discussion got side-tracked to an unrelated topic)

·         When we launch new forms, the end users make a mess of them because they can’t figure out how to fill them in. Then we have to re-work the form and launch the revision. (At a kickoff meeting for a project to design and implement a new form)

·         We hope that Jenny never catches the flu, as no one else knows how to do that critical step we need at go-live. (At a status meeting, while explaining that a component is late due to Jenny’s lack of availability)

·         It seems like there’s an awful lot of work to be done on the go-live weekend, I wonder if it’s too tight. (In the kitchen, when the project manager was getting coffee)

·         We have a new vendor working on an internal web site for our HR department. (In a discussion about roles and responsibilities)

·         The vendor’s project manager is a sub-contractor; do you think she will keep the vendor’s senior management informed and engaged? (Informal chat, after the vendor’s project manager had been introduced)

The risks noted above did not come up in official risk identification meetings, but instead came up in other unrelated discussions.

Managing the risks

Because these came up outside of the formal risk meeting, and many were assumed as ‘just the way it is’, there was sometimes resistance when the question was asked “what can we do to manage this?” However, when pressed further, it was frequently possible to identify steps that could be taken to manage the risk. Let’s look at risk mitigation ideas for the first few risk examples:

·         If changes to AD security groups usually have a lot of errors, can they be subject to a rigorous review step before implementation, or can the testing approach be examined to see if there are gaps, or can part/all of the process be automated?

·         If it’s crazy trying to get the inventory to reconcile, how about a rehearsal, or customized reconciliation reports, or review the approach with the accounting department to see what additional controls are warranted?

·         If new forms are not well received, can they be vetted and changed as part of the project instead of after launch? Is it possible to create a user advisory panel, or run a pilot with a subset of users?

As you can see, once the risk is identified, it’s possible to come up with ideas on how to mitigate that risk.

Conclusion

Although the risk identification meetings can generate some ideas about risk, additional risks can be identified if the project manager is listening for them. Ask yourself every day: Did I hear something today that might suggest there’s a risk we aren’t managing?

Copyright 2015 Debbie Gallagher

Great by choice by Jim Collins and Morten T Hansen (book summary)

Have you read Great by Choice, by Jim Collins and Morten T. Hansen? The book describes nine years of research done to try and identify why some companies thrive in uncertainty and others do not. As I read the book, I realized that with the fast pace of change in technology, the question is relevant not just to companies, but to their IT projects as well.

The study

The authors studied the stock market for the period 1972 to 2002, and selected several companies who met three criteria: (a) stock price beat their industry index by ten times (10X); (b) the environment in which the company operated over that period was turbulent; and (c) the company was fairly small (and therefore vulnerable) at the beginning of the period.  These companies and their leaders were called 10Xers.

Then for each 10X company, they selected a comparison company in the same industry. The majority of the work was to research what was different between the 10X companies and the comparison companies to find out what could have been the cause of the 10Xer’s success.

The results of their research surprised them. Great creativity and risk taking turned out not to be key factors to success in the long run. Instead, the 10X companies and their leaders managed risk very well, and had three key behaviors: (a) Fanatic discipline; (b) empirical creativity; and (c) productive paranoia

Antarctica

The authors use the race for the South Pole in 1911 by Roald Amundsen and Robert Falcon Scott to illustrate some of the book’s findings. They liken the approach of Amundson’s successful expedition to the10X company leaders, while the comparison company leaders lean more toward the habits of Scott, whose team perished before completing the trek.

It’s an interesting step back in time and also illustrates very well not only the concepts of the study but also the relationship between risk management, success, and failure.

Fanatic discipline

In turbulent times, the companies that outperformed their industries had consistent goals and performance. Instead of pursuing massive high-growth strategies, they pushed for consistent returns every year, no matter how difficult it was to achieve. In addition, even during boom times in their industry, they held back from wild growth strategies despite pressure to do so.

Achieving consistent performance no matter what is happening in the marketplace requires concrete, clear, intelligent, and rigorously pursued performance mechanisms to keep on track. Fanatic discipline is not about bureaucracy, but about the ability to remain clear about goals and find ways to deliver consistent performance. It also requires the ability to be a non-conformist and avoid the herd instinct of the marketplace.

Empirical creativity

The most successful companies tested new concepts on a small scale and determined what worked and what didn’t work before launching significant new lines of business, markets and technologies.

These trials allowed the 10X companies to spend a lot of time and money on big launches only once they had tried the concept on a smaller scale and determined how to be successful. Alternatively, they sometimes learned that the concept didn’t work for them and avoided spending a great deal of money and resources to learn that.

Although innovation is necessary, the authors were surprised to find that the most innovative were not the most successful companies. Instead, a threshold level of innovation is required to compete in the industry, and beyond that the amount of innovation doesn’t matter very much. Instead, it matters more that innovation is paired with the ability to scale the innovation and deliver on commitments to customers.

Productive paranoia

Leaders of 10X companies prepared for the unknown and managed risks well.  They concerned themselves with what could go wrong and created buffers to deal with those known and also with unknown risks.

The leaders of the 10X companies avoided taking actions that had huge downside potential. In addition, they had the ability to look beyond daily operations to see the big picture and identify the biggest risks to their companies.

IT project risk

This book is based on research into US corporations and makes interesting references to the South Pole expeditions of 1911.  However, as I read it, I thought very often of IT projects I’ve been on, and how the lessons could be applied. Many projects fail to achieve their objectives (or even fail to complete) due to lack of discipline, trying to deliver untested concepts, or poor risk management. Examples include projects without clear objectives, sponsorship, scope, or requirements (lack of discipline). Also, there are projects attempting to launch big technological or business changes without pilots (lack of empirical information).  Probably anyone reading this can think of cases where big risks (new software, new hardware, new vendor) were taken, but the risks were either unrecognized or unmanaged (lack of productive paranoia).

Summary

The authors of the book Great by Choice have based their book on research into US companies that outperformed their industry competitors over a thirty-year span by at least ten times. They identify three key characteristics of 10X companies and leaders that have been key to their success: (a) fanatic discipline; (b) empirical creativity; and (c) productive paranoia.

They make no claim at all of the book’s relevance to IT projects. However, as I read the book, I thought the parallels were easy to see and recommend the book to those who are interested in managing the risk of IT projects.

Copyright 2015 Debbie Gallagher

It's just an upgrade

This is a true story. The company name has been changed.

The project manager was developing a project charter for his new project. He and his supervisor expected this project to be low risk and quite straightforward, as it was just an upgrade to an application that had already been running for many years in the organization. The application was used daily by approximately ninety percent of employees across the country.

The first risk assessment

The project manager met with his assigned mentor to discuss the project. At this company, it was a requirement that the project charter include a risk assessment section. The project manager included as risks some items that had been problems on previous projects, such as business resources and IT code promotion resources not being available. When questioned, the project manager said there was no particular reason to believe these would be problems on the new project, but he wasn’t sure what else was relevant to include as risks.

The real story

In discussion with the project manager, the mentor learned more about the project, which was in the planning stage prior to charter approval and project launch.

The application vendor had been slow to respond to requests for help with planning the upgrade.  The vendor was apologetic about the lack of assistance provided and explained that they had recently sold a very large engagement and were very busy as a result.

The existing application had numerous customizations and interfaces, almost none of which were documented. Upgrades of this application had been attempted before, but the projects were never completed due to the difficulty in replacing the undocumented customizations and interfaces.

The project sponsor had just gone on medical leave for several months and appointed a junior manager to act as sponsor in her place.

The mentor helped the project manager to identify three significant risks: (a) unavailable vendor resources; (b) undocumented customizations and interfaces; and (c) inappropriate sponsorship.

Actions taken

The mentor suggested several follow up actions to the project manager to try and manage the risks before launching the project.

The project manager approached the application vendor about alternatives for resourcing the project, but the answers were discouraging. The vendor had no business partners trained to perform implementations and also could not recommend any other companies or individuals who might to able to guide the company through the application upgrade process.

A business analyst was assigned to start documenting the customizations and interfaces, but the work was progressing very slowly as he was also assigned to a high-priority project.

Although the original project sponsor was available for a brief phone call, she was unable to suggest a more appropriate replacement sponsor for her expected several-month absence.

Unfortunately, none of the actions resulted in lower risks.

Updated risk assessment

After these follow up actions and further discussion with the mentor, the project manager had a much more realistic view of the project risk.

The vendor’s lack of availability during planning raised significant concerns about its ability to provide appropriate resources for the project itself. Noticeable lack of availability during the sales cycle made it clear that the vendor was overwhelmed by the large contract they had just signed and would not be focused on this company’s upgrade. Even if they managed to provide a few resources, the vendor would not be able to support them to the extent needed.

Undocumented customizations and interfaces had already caused previous upgrades of this application to fail. Proceeding with the project without developing a good understanding of the customizations and interfaces would certainly cause the project to fail once again.

The lack of an appropriate project sponsor is a risk that sounds alarm bells for any experienced project manager.  The appointment of a low-level resource shows a lack of understanding of the role of the sponsor. Someone too junior to determine business priorities and commit resources is not an appropriate choice.

Any one of the three risks described would put the project at significant risk of failure. Given all three risks, it would be nearly impossible for the project to succeed. 

Recommendations

The project manager discussed the revised risk assessment with his supervisor. The supervisor was reluctant to delay the project but did finally agree that it was too risky to proceed at that time.  However, the discussion of the risk with the sponsor would be a delicate matter, especially given the concern about the inappropriateness of the sponsor.

The supervisor arranged for the IT vice-president to have a discussion with the business vice-president (to whom the original project sponsor reported).

Epilogue

The two vice-presidents agreed that a different project was needed first, one with the purpose of documenting the existing customizations and interfaces. The IT vice-president agreed to commit resources to this predecessor project.

They expected that by the time that project was completed, the original project sponsor would be back from medical leave and the vendor would have had time to train new resources.

Conclusion

Given the risks identified for the project, the vice-presidents made the right decision. Delaying the project and starting with documenting the customizations and interfaces was the best option to set it up for success.

Evaluating project risk is frequently a challenge for junior project managers. It is typical for new project managers to focus on schedules and budgets, and develop their understanding of risks and issues at a later stage of their project management career. Providing a mentor for the project manager was a wise course of action for the company. The mentor was able to guide the project manager through the risk assessment process, leading to a more realistic evaluation of the project’s chance of success.

Copyright 2015 Debbie Gallagher

Is it really all about schedule and budget?

When I read job postings for IT project managers, there is a heavy focus on being able to deliver on schedule and on budget. In addition, I sometimes hear rather lively discussions about the role of the project manager: are they/should they be accountable for anything at all beyond schedule and budget? I’ve also observed from my own project management experience that the first questions from the sponsor and from the steering committee are invariably “how much will it cost” and “when will it be done”? Again, the focus is on schedule and budget.

Why are schedule and budget the big focus of project managers and the executives for whom they deliver projects? Is that where all the big risks are?

The big risks

In my opinion, schedule and budget overruns are not the big risks themselves. They are the outcome of other risks that have materialized into issues. So, what are the biggest project risks that drive schedule and budget problems?

In a previous article Identifying and Managing Project Risk by Tom Kendrick (book summary), I summarized Tom Kendrick’s book, in which he outlines his findings from examining hundreds of technical projects. In the book, all the risks that occurred were translated to a timeline impact so they could be compared. I think it’s a fair assumption that when the timeline is extended, the cost also grows. So, what are the risks that have the greatest impact? From Kendrick’s book, two risk areas with the greatest impact are: (a) non-mandatory scope changes; and (b) legitimate, but not anticipated, requirements gaps.

In other words, the big risks are related to scope and requirements. There are plenty of other risks, but let’s start with these two. When these risk events occur, there is an impact on the schedule and budget.

Managing scope and requirements

There are several ways to focus on scope and requirements.

First, these need to be defined as clearly as possible at the outset.  Too often there is a rush to get the project started, and it begins with inadequate descriptions of scope and requirements. This rush to delivery and related skimping on scope and requirements can lead to disastrously large impacts on schedule and budget.

Sometimes there is a good reason to have uncertainty at the start of the project.  In that case, identify the lack of clarity as a project risk. Then, make sure that everyone on the project and on the steering committee knows about that risk. Also be sure to include a re-assessment point in the schedule and funds in the budget to address the cost of potential changes.

Next, focus on requirements and scope during the project.  At the outset, meet with the project team(s) to review these in detail and address any questions. It is crucial that the team knows what is to be delivered.

Make sure there is a change control process and that it is followed.

Review and follow up on the project issues frequently. Issues are often the result of disagreement regarding what is in scope or what is needed. Too often, issues are logged in a database and neglected.

During the project there should be checkpoints to verify that the project team is on track. Hold walkthrough meetings with team members, and also with stakeholders outside the team. While often seen as time consuming, these walkthroughs are critical tools to ensuring the delivered product supports the requirements and scope of work.

An effective way to determine during the project if there are issues with scope and requirements is to ask team members informal questions about what they are working on, whether they are clear enough on what they need to do, and what problems they are having.  The project manager needs to make herself available when there are questions from the team regarding scope and requirements, and not put them off while heads-down adding up budget hours or updating Gantt charts.

Testing is another method of ensuring that the delivered product meets requirements and scope. Evaluate test plans to determine if they engage the right people and cover the right tests to identify gaps in scope and requirements. Ideally, gaps should be identified before testing, but even if found during testing, at least they are dealt with before go-live.

Conclusion

When hiring and evaluating project managers, the heavy focus on schedule and budget overlooks the relationship between schedule and budget, and project risks.  It’s usually the project risk events occurring that cause the budget and schedule problems.  Managing the schedule and budget is really about managing the project risks. In this article, I picked two top risks and discussed common techniques to manage them.

I’ll leave you with a couple of thoughts. First, I think the question about whether the project manager is responsible for anything besides schedule and budget has a definitive answer. Project Management Institute says yes, as the knowledge areas cover a lot more than schedule and budget.  Quite apart from PMI’s body of knowledge, the answer has to be yes, because if you aren’t managing the project risks, it isn’t possible to manage the schedule and budget.

My second parting thought: If the project comes in on time and on budget, but does not deliver the expected scope or meet requirements, what was really achieved?

Copyright 2015 Debbie Gallagher

Identifying and managing project risk by Tom Kendrick (book summary)

In reading Tom Kendrick’s book, Identifying and Managing Project Risk: Essential Tools for Failure-Proofing Your Project (2^nd edition, 2009), I found it a compelling read and thought I’d share my views.

The premise of Kendrick’s book is that the experiences from previous projects, specifically the historical data of project problems and their impacts can help us in identifying risk and managing those risks on our own projects.

PERIL database

Over a period of more than ten years, Kendrick collected data on project problems from hundreds of project managers. He assembled this data in his Project Experience Risk Information Library (PERIL) database. The database provides information on what types of things go wrong on projects, and attributes each to root causes. In addition, the database captures estimates of the impact of the problems that occurred. The PERIL data is mainly based on projects having a dependence on new or relatively new technology; includes mostly projects that had a planned duration between six and twelve months; and twenty or fewer project team members. 

Kendrick analyzed over six hundred projects, and has come up with an interesting approach to allow comparison. For each project, whatever the actual impact of the project issues, he has converted the impact to a time impact.  For example, if scope was reduced to meet the scheduled completion date, the timeline impact has been determined by calculating how late the project would have been if it had delivered its original scope. Kendrick’s approach may not stand up to rigorous scientific scrutiny (and he is open about the bias in the data), but in my opinion, it still provides a very useful way to compare project risks. From these risks come clear risk management strategies. 

Kendrick’s findings

Kendrick’s first analysis is high level and compares the total timeline impact in the PERIL database for eight root causes.  These include, for example, scope changes during the project, failure to meet deliverable requirements, schedule delays, internal staffing.

In later chapters, Kendrick then breaks down the categories of risk/impact into sub-categories. For example, the impact of scope changes during the project is broken down into non-mandatory scope changes, legitimate requirements gaps, and so on.

In the chapters covering sub-categories, the descriptions of issues from the project managers are included along with the frequency and impact figures. For example, end user not involved enough in requirements, supplier not meeting deadlines, system architect not available. In reading these issues from other project managers, some triggered project flashbacks!

The Panama Canal

Throughout the book Kendrick weaves stories of the Panama Canal building projects. The first project, which ran from 1879 to 1889, was a failure. Ten years after the start of the project, it was cancelled, thousands of workers were dead, the costs had spiraled out of control, and there was no canal. A failure indeed!

The second Panama Canal project started in 1902, started re-planning in 1905, and the canal opened in 1914. This project finished under budget and ahead of schedule, and was useable for its intended purpose. Although workers also died on the second project, the death and disease rates were dramatically reduced due to improved insect control, housing and other practices implemented on the project.

Although Kendrick placed the Panama Canal segments to tie each to a specific risk or approach being discussed in the book, for me they were very interesting, not just as project management solutions, but also as an interesting historical sidebar.

Lessons from the book

Although Kendrick makes project management recommendations regarding each sub-category, he does not make recommendations about which are the most critical practices for project managers to adopt overall. He leaves this to the discretion of each project manager, to be determined based on their own specific projects and environments.

In examining the breakdowns in the book, each project manager can determine what the biggest-impact problems are likely to be, either in general or on their own project, and then decide what to do about them. Every reader will likely take something different from the book depending on whether they are looking for general guidance or for insights for their own project.

An example

For purposes of this article, I decided to look at the data from the perspective of “If I wanted to improve my project risk management in just one area, where should I focus to address the project risks that occur most frequently and have the greatest impact?” So, I started by looking for the category that has the greatest number of issues and the greatest impact.  As described above, the greatest impact is from scope changes that occur during the project.  It is also the issue that occurs with the greatest frequency.

Then, within the scope change category, the two sub-categories that have the greatest frequency and impact are non-mandatory changes and legitimate requirements gaps.

My conclusion is that to improve my project risk management practices, I should start by improving scope management. By looking at the sub-categories, I see that the ways to improve scope management the most are: (a) eliminate the non-mandatory changes; and (b) reduce legitimate requirements gaps by either spending more time on requirements or by making a provision in the schedule right from the outset.

Summary

If I was looking at the data before starting a new project, I would be looking to see what might apply to my own project.

The value in Kendrick’s book is the analysis of many technical projects and the comparability of the various causes of problems.  This comparability allows each project manager to assess for themselves what recommendations can be quickly implemented for the highest improvement in their projects. 

The Panama Canal highlights provide interesting examples of the differences in project management techniques and capabilities.

Copyright 2015 Debbie Gallagher

Evidence of controls for your project audit

In previous articles, I described why your IT project might be audited, and an overview of the major steps in the audit for an application development or implementation project. In this article, I’ll provide some insight into the evidence needed to demonstrate for the audit that the project team actually executed the controls.

Examination and testing of evidence is something the auditor is required to do in order to satisfy auditing standards (see my earlier article What to expect when your IT project is audited).

Let’s follow some examples for a few common project controls. I’ll go over the kind of evidence that might be used, as well as some common problems with evidence.

Evidence that testing occurred

A common control on IT projects is that the application is tested to verify that it works and meets business requirements.

Relevant evidence that testing took place and that the application worked often focuses on the signed test scripts that were executed. These are particularly useful evidence if they include expected results, as well as a notation and signature indicating that the test passed. Additional supporting documents might be the task completion status for testing tasks on the project schedule, or a summary document capturing test pass/fail results for all scripts.

The auditor may also examine test planning documentation in order to assess the approach. For example, testing strategy, testing plans, project schedules showing testing tasks and resources, and test scripts.

As mentioned in my previous article Why your IT project is being audited, the various auditors examining your IT project may have differing scope and purpose. These differences will affect the evidence they examine. For example, if an internal audit group needed to verify that the company methodology was followed, they may need to see evidence to support specific tests mandated by your company’s own standards.

Over the course of performing many project audits, I noticed that often project teams had documentation to support test planning, but sometimes had problems with sufficient evidence supporting the test execution. This is a significant distinction and is not just fussiness on the part of the auditor. It is necessary that the evidence support the specific control.

Evidence problems related to test execution can occur when testers do not sign their completed test scripts, neglect to indicate that the test result was a pass, or even throw away their completed test results. In addition, sometimes a tester will keep scripts showing failure, but neglect to keep the follow up test results showing that the problem was corrected.

Evidence of business approval

In addition to the testing itself, it is customary for a key business representative to approve the test results for the project.

Sometimes the testing manager or project manager will ask the business approver to respond to an email, indicating approval of the test results. This approach sounds straightforward enough but in practice sometimes leads to inadequate evidence. 

A common problem with email signoffs is that requesters and approvers may treat them casually, leading to problems in using them as audit evidence later. One evidence problem occurs when the wording does not make it clear what the approver is approving.  For example, if the testing manager sends an email to the approver, “Please approve the attached,” attaches the test results, then the approver replies “Approved”, the test results are no longer attached and there is no indication in the email itself of what is being approved. 

A simple solution is to be more specific in the request email. For example, “Please approve the test results, which show successful completion of all general ledger scripts, details attached.” With this approach, if the attachment is removed in the email reply process, there is still an indication of what is being approved.

A significant evidence issue can arise when the approver adds irrelevant information to the approval email. For example, “Results ok, please let me know the resolution of the problem with ap-open.”

The approver has started an entirely new subject, as ap-open has nothing to do with the specific approval being requested. Unfortunately, there is no indication that the ap-open problem is different from the subject of the approval, so it leaves the auditor wondering how the approval could have been granted when there appears to be a problem outstanding.

The auditor has no choice, if presented with this email as an approval, except to examine the situation further and obtain further evidence that the ap-open problem had no relationship to the approval being granted.  The auditor, remember, is mandated to examine the documents to obtain sufficient evidence, and the wording of the email suggests that the approval may not be valid. This kind of email subject change invalidates the approval given until shown otherwise.

One more problem can occur with email approvals, when the email deletion policy eliminates the evidence prior to the audit! Remember to copy the approval emails to a place where they will be retained.

Evaluating your own evidence

In trying to determine in advance whether your evidence might be acceptable to the auditors, review the documents and ask yourself this question “If I wasn’t on the project team, and this set of documents was everything I knew about the whether the control happened, would I have proof that the work was performed and found to be acceptable?”  

Conclusion

What frequently happens, I think, is that project team members do perform the work and obtain the approvals.  However, in many cases, these aspects of the work may not be well documented, leading to problems in supporting the audit later.

Now that you know some of the evidence pitfalls, perhaps you can avoid them on your next audit.

Copyright 2015 Debbie Gallagher

Using a risk and controls matrix to support your project audit

In previous articles, I described why your IT project might be audited, and an overview of the major steps in the audit for an application development or implementation project. In this article, I’ll describe the IT project risk and controls matrix, which you can use to prepare for the audit.

Risk and controls matrix

A risk and controls matrix is one of the most common methods that companies and auditors use to document risks and controls. Although some companies have more formal applications, often Excel spreadsheets or Word tables are used for this documentation. The function of the matrix is simply to provide structure for the documentation. You could certainly write narratives instead, but the matrix is easy to read and is a familiar format for those who document or audit controls.

Columns in the matrix

The columns in the matrix (from left to right) generally include:

·         Risk description

·         Likelihood

·         Impact

·         Control objective

·         Control activity.

Documenting risks

First, the risks documented are inherent risks, which are the risks that exist assuming there are no controls in your environment. Keep in mind that although you may have risks that are specific to the project, there are some risks you can identify without even knowing what the specific IT project is. We are talking about inherent risk, so this is easier than it sounds.

Here’s an example, “The system may not meet the needs of the business, or deliver what management expects.”  Some companies write this in more formal language but it isn't necessary to do so. 

For each risk, indicate how likely it is and what the impact would be if the risk occurred. You can get numerically fancy if you like, but it is very common to use high, medium, and low for these indicators. Where the likelihood and impact are high, the controls should be particularly stringent, and their operation should be well documented for reference and audit. If you had no controls at all on your project, the likelihood of not meeting business needs or management objectives would be a high-probability and high-impact risk.

Control objectives

Generally, the control objectives are opposite to the risks, as they represent the outcome you want. For example, the control objective for the above risk would be, “The system meets the needs of the business and is what management expects it to be.”

No joke, it’s often as obvious as that. As a result, some companies don’t bother to document the risk and just start with the control objective. However, it is much more common to include the risk because the entire purpose of the controls is to address risk.

Control activities

The control activities are what your company or project team is doing to mitigate the inherent risk and satisfy the control objective.

For the example above, there would usually be several control activities. For example, one of the key ways to ensure that the application functions according to expectations is to test it in several different ways. You might write a control activity such as “Unit testing, interface testing, and user acceptance testing will be performed.”

Another control you likely have is formal sign off from the business user on the requirements, on the application design, or on the completed application test results.

In trying to ensure the application will meet the needs of the business and management when it is implemented, there are generally environmental controls that are enforced not just company wide, but also at the project level. For example development, testing, and production environments are restricted based on the role of the individual. In addition, you do all of the application testing in an environment that as closely as possible reflects the expected production environment. 

In ensuring the application will deliver its expected results, the project usually also trains end users to make sure they will use the application correctly and follow the associated new processes.

So, now you have a set of control activities that together can satisfy the control objective and mitigate the specified inherent risk.

Other typical risks

Now that you’ve seen a detailed example of one of the risks, here are some other risks that you may wish to include in your matrix:

·         Project does not meet the expectations of management: For this risk, the control activities are usually the project approval and project oversight processes you have. For example, project approvals by management, project advisory and steering committees, etc.

·         Systems implemented interfere with normal business operations: The control activities you might document for this risk are the system outage permission processes, as well as the updates to documentation for the application and the production operations teams. You may also obtain source code or arrange an escrow agreement for purchased applications.

·         Data converted to the new system may not be complete, accurate, or valid: This is a risk you would normally include if you are converting data from an older system to the new one.

Project-specific risks

The risks outlined above apply to nearly all IT application development or implementation projects. You may also have additional risks to consider based on the specific project you are delivering.

Conclusion

Documenting your project risks, control objectives, and control activities can help you prepare your project for an audit. The controls matrix is a typical approach used in other areas of the business and by auditors, as it provides structure for the assessment of controls.

Copyright 2015 Debbie Gallagher