Retiring applications

A frequently-neglected part of the plan for implementation of a new application is the retirement of the one that is being replaced.

However, there is good reason to do that work at the same time as your new project, as there are risks and costs related to keeping an old system running after it is no longer actively used.

Why decommission

The most obvious concern with maintaining applications that are no longer used is the cost of licensing and support agreements. There are also less obvious costs. For example, you may be running certain hardware or operating systems for the sole purpose of maintaining the legacy applications. You may also pay consultants with legacy expertise to maintain that legacy hardware and software. Perhaps it is no longer feasible to apply some types of security to older systems.

Here is an approach to consider and tailor to your own application and organization.

Business ownership

Although managed by IT, there must be a business owner who will participate in retirement planning and approve any decisions that are relevant to the business.

Contracts

Unless the application being retired is fully home-grown, you likely have some contracts in place. For example, application licenses for users and administrators, and support agreements. The infrastructure or security support may also be covered by service contracts. Review all of the relevant contracts and determine what your rights and responsibilities are. E.g. if users are not maintaining any data, but are read-only, do you still have to pay? What is the notice period, and when is the deadline to give notice? Are your responsibilities truly clear in the contract, or do you need a lawyer to review it?

Continuing access to the data

Depending on the business use of the application, it may be necessary to keep the data for some period of time. Start by identifying the data that is kept in the system. Is it general ledger, purchase orders, energy usage? Also identify whether the application is the system of record for that data. If the application is just for reporting or transmitting, and the system of record is somewhere else, it may much easier to eliminate the application’s data set.

If the application is the system of record, map the data sets in the system to the corporate data retention policy. This policy will have been developed in alignment with compliance, legal, and regulatory requirements. Then, discuss with the business owner what their business needs are for ongoing access to the data. This may be different from the corporate data retention policy.

While you’re having that discussion with the business, find out what kind of access to the data is going to be needed. Do they still need to use the application to view the data, or will it be enough to be able to query the database?

Also ask the business owner how many users should have continuing access to the application. Perhaps you can reduce the number of users to just a few, either right away, or after a year or so.

If the application being retired is on some technology that you no longer use for any other applications, it may be tempting to copy the database to a technology that you support as a standard. However, this may not provide what the business needs. If the data structure is complex, you may no longer have an employee with the experience to query that structure and get the right answer. In addition, if there is a need to re-generate reports or documents, it may not be feasible to do so without the application.

Operations

Perhaps based on either the retention policy, or the business needs, you have learned that the application is still needed for inquiry and reporting for several years. There still may be steps you can take to reduce the effort of operating the application.

For example, since the data isn’t changing, it may not be necessary to do daily backups or manage high availability. Perhaps the business would now be fine with a monthly backup, so you have fresh media. Maybe the business would agree with a longer recovery in the event of failure.

It’s also likely that you could eliminate the development and testing environments for the retiring application.

Security and controls

You can also examine the security and controls for the retiring application. For example, discuss with the business whether they still require two-factor authentication.

It’s also a good time to discuss whether the controls for the application and its supporting infrastructure should be tested any more as part of the audit. Some of the testing may no longer be needed, which could reduce the effort of internal or external auditors.

Other considerations

If you are generally moving applications to the cloud, or no longer wish to support the particular technology platform of the retiring application in your data centre, you may consider options to have someone else manage it for you. There are usually vendors who would agree to host the hardware and application in their own data centre. Note that this approach requires considerable planning, and you should engage the potential vendor early to discuss requirements.

Conclusion

When you implement a new system, planning for the retirement of the old one should be planned at the same time. It should be possible to manage costs and risks by appropriate planning and execution.

This article provided some high-level guidance on retirement planning to get you started. You can use it as a framework, and customize it to your own organization’s needs

The changing role of the technology team in ERP implementations (part 3)

In two previous articles, I described several areas where the role of the technology team in ERP implementations has changed in the last ten to twenty years.  Here is part 3, the final installment.

Single sign on

This capability makes life much easier for the end user. Although it is easier to implement than it used to be, it still requires set up, testing, and support by the technology team.

 

Disaster recovery

In a cloud environment, you won’t be managing databases and servers any more. The scenario where your entire data centre collapsed and you lost all your systems at once will change to multiple application-based or vendor-based scenarios. You need to re-think what disaster recovery planning and testing should cover. You also need to verify that participation in disaster recovery planning and testing exists in your contracts with cloud vendors.

IT controls

If you’ve been outsourcing some of your IT functions already, you’re familiar with this. Although your organization is responsible to ensure IT controls are operating effectively, many of those controls will now be implemented and operated by your cloud vendors. Relevant IT controls will need to be reviewed internally and with the vendors to identify if any controls need to be re-designed, and how they will be tested. Selection of your ERP system and negotiation of the contract needs to consider IT controls and associated Service Organization Control (SOC) reporting requirements.

Application support model

With ERP in the cloud, the internal support team no longer needs to be heavily weighted toward database analysis, operating system, and hardware experience. Instead, the team’s focus will be on the business process, how the application supports the business process, and reporting and data governance. This is a significant shift in mindset and skills for many IT support managers and staff.

If the existing ERP is supported by manual processes, your business users will certainly want to automate more when the new ERP is implemented. E.g. on-line workflow to replace email approvals. As a result, the number of ERP users you support is likely to be higher than it is today.

Vendor management

Given the number of areas where your IT organization now relies on vendors (ERP, integration, disaster recovery, IT controls), it is evident that a key responsibility and skill set in your operational support model is vendor management. This may be an area in which your team requires bolstering if you have not been outsourcing or using cloud very much to date.

Conclusion

Technology is so ingrained in business processes today that there’s no such thing as a fully manual business process any more.  Having the business define the new processes and handing off requirements to the technology team is not a workable process. Business analysts have to participate fully in the business process workshops, identifying requirements along the way for everything from mobile device needs to interface and data conversion requirements.

In addition, your technology team for the ERP project, whether internally available or contracted, needs to support a wide variety of requirements that didn’t often exist two decades ago: mobile technology, browsers, firewalls, single sign on. In addition, the approaches to disaster recovery, IT controls, and application support likely need to change.

As you can see from this article and the previous two, the role of the technology team in ERP implementations has changed significantly. So, if you are implementing ERP soon, and it’s been ten or twenty years since you did so, you should anticipate and plan for all or most of these differences.

Evidence of controls for your project audit

In previous articles, I described why your IT project might be audited, and an overview of the major steps in the audit for an application development or implementation project. In this article, I’ll provide some insight into the evidence needed to demonstrate for the audit that the project team actually executed the controls.

Examination and testing of evidence is something the auditor is required to do in order to satisfy auditing standards (see my earlier article What to expect when your IT project is audited).

Let’s follow some examples for a few common project controls. I’ll go over the kind of evidence that might be used, as well as some common problems with evidence.

Evidence that testing occurred

A common control on IT projects is that the application is tested to verify that it works and meets business requirements.

Relevant evidence that testing took place and that the application worked often focuses on the signed test scripts that were executed. These are particularly useful evidence if they include expected results, as well as a notation and signature indicating that the test passed. Additional supporting documents might be the task completion status for testing tasks on the project schedule, or a summary document capturing test pass/fail results for all scripts.

The auditor may also examine test planning documentation in order to assess the approach. For example, testing strategy, testing plans, project schedules showing testing tasks and resources, and test scripts.

As mentioned in my previous article Why your IT project is being audited, the various auditors examining your IT project may have differing scope and purpose. These differences will affect the evidence they examine. For example, if an internal audit group needed to verify that the company methodology was followed, they may need to see evidence to support specific tests mandated by your company’s own standards.

Over the course of performing many project audits, I noticed that often project teams had documentation to support test planning, but sometimes had problems with sufficient evidence supporting the test execution. This is a significant distinction and is not just fussiness on the part of the auditor. It is necessary that the evidence support the specific control.

Evidence problems related to test execution can occur when testers do not sign their completed test scripts, neglect to indicate that the test result was a pass, or even throw away their completed test results. In addition, sometimes a tester will keep scripts showing failure, but neglect to keep the follow up test results showing that the problem was corrected.

Evidence of business approval

In addition to the testing itself, it is customary for a key business representative to approve the test results for the project.

Sometimes the testing manager or project manager will ask the business approver to respond to an email, indicating approval of the test results. This approach sounds straightforward enough but in practice sometimes leads to inadequate evidence. 

A common problem with email signoffs is that requesters and approvers may treat them casually, leading to problems in using them as audit evidence later. One evidence problem occurs when the wording does not make it clear what the approver is approving.  For example, if the testing manager sends an email to the approver, “Please approve the attached,” attaches the test results, then the approver replies “Approved”, the test results are no longer attached and there is no indication in the email itself of what is being approved. 

A simple solution is to be more specific in the request email. For example, “Please approve the test results, which show successful completion of all general ledger scripts, details attached.” With this approach, if the attachment is removed in the email reply process, there is still an indication of what is being approved.

A significant evidence issue can arise when the approver adds irrelevant information to the approval email. For example, “Results ok, please let me know the resolution of the problem with ap-open.”

The approver has started an entirely new subject, as ap-open has nothing to do with the specific approval being requested. Unfortunately, there is no indication that the ap-open problem is different from the subject of the approval, so it leaves the auditor wondering how the approval could have been granted when there appears to be a problem outstanding.

The auditor has no choice, if presented with this email as an approval, except to examine the situation further and obtain further evidence that the ap-open problem had no relationship to the approval being granted.  The auditor, remember, is mandated to examine the documents to obtain sufficient evidence, and the wording of the email suggests that the approval may not be valid. This kind of email subject change invalidates the approval given until shown otherwise.

One more problem can occur with email approvals, when the email deletion policy eliminates the evidence prior to the audit! Remember to copy the approval emails to a place where they will be retained.

Evaluating your own evidence

In trying to determine in advance whether your evidence might be acceptable to the auditors, review the documents and ask yourself this question “If I wasn’t on the project team, and this set of documents was everything I knew about the whether the control happened, would I have proof that the work was performed and found to be acceptable?”  

Conclusion

What frequently happens, I think, is that project team members do perform the work and obtain the approvals.  However, in many cases, these aspects of the work may not be well documented, leading to problems in supporting the audit later.

Now that you know some of the evidence pitfalls, perhaps you can avoid them on your next audit.

Copyright 2015 Debbie Gallagher

Using a risk and controls matrix to support your project audit

In previous articles, I described why your IT project might be audited, and an overview of the major steps in the audit for an application development or implementation project. In this article, I’ll describe the IT project risk and controls matrix, which you can use to prepare for the audit.

Risk and controls matrix

A risk and controls matrix is one of the most common methods that companies and auditors use to document risks and controls. Although some companies have more formal applications, often Excel spreadsheets or Word tables are used for this documentation. The function of the matrix is simply to provide structure for the documentation. You could certainly write narratives instead, but the matrix is easy to read and is a familiar format for those who document or audit controls.

Columns in the matrix

The columns in the matrix (from left to right) generally include:

·         Risk description

·         Likelihood

·         Impact

·         Control objective

·         Control activity.

Documenting risks

First, the risks documented are inherent risks, which are the risks that exist assuming there are no controls in your environment. Keep in mind that although you may have risks that are specific to the project, there are some risks you can identify without even knowing what the specific IT project is. We are talking about inherent risk, so this is easier than it sounds.

Here’s an example, “The system may not meet the needs of the business, or deliver what management expects.”  Some companies write this in more formal language but it isn't necessary to do so. 

For each risk, indicate how likely it is and what the impact would be if the risk occurred. You can get numerically fancy if you like, but it is very common to use high, medium, and low for these indicators. Where the likelihood and impact are high, the controls should be particularly stringent, and their operation should be well documented for reference and audit. If you had no controls at all on your project, the likelihood of not meeting business needs or management objectives would be a high-probability and high-impact risk.

Control objectives

Generally, the control objectives are opposite to the risks, as they represent the outcome you want. For example, the control objective for the above risk would be, “The system meets the needs of the business and is what management expects it to be.”

No joke, it’s often as obvious as that. As a result, some companies don’t bother to document the risk and just start with the control objective. However, it is much more common to include the risk because the entire purpose of the controls is to address risk.

Control activities

The control activities are what your company or project team is doing to mitigate the inherent risk and satisfy the control objective.

For the example above, there would usually be several control activities. For example, one of the key ways to ensure that the application functions according to expectations is to test it in several different ways. You might write a control activity such as “Unit testing, interface testing, and user acceptance testing will be performed.”

Another control you likely have is formal sign off from the business user on the requirements, on the application design, or on the completed application test results.

In trying to ensure the application will meet the needs of the business and management when it is implemented, there are generally environmental controls that are enforced not just company wide, but also at the project level. For example development, testing, and production environments are restricted based on the role of the individual. In addition, you do all of the application testing in an environment that as closely as possible reflects the expected production environment. 

In ensuring the application will deliver its expected results, the project usually also trains end users to make sure they will use the application correctly and follow the associated new processes.

So, now you have a set of control activities that together can satisfy the control objective and mitigate the specified inherent risk.

Other typical risks

Now that you’ve seen a detailed example of one of the risks, here are some other risks that you may wish to include in your matrix:

·         Project does not meet the expectations of management: For this risk, the control activities are usually the project approval and project oversight processes you have. For example, project approvals by management, project advisory and steering committees, etc.

·         Systems implemented interfere with normal business operations: The control activities you might document for this risk are the system outage permission processes, as well as the updates to documentation for the application and the production operations teams. You may also obtain source code or arrange an escrow agreement for purchased applications.

·         Data converted to the new system may not be complete, accurate, or valid: This is a risk you would normally include if you are converting data from an older system to the new one.

Project-specific risks

The risks outlined above apply to nearly all IT application development or implementation projects. You may also have additional risks to consider based on the specific project you are delivering.

Conclusion

Documenting your project risks, control objectives, and control activities can help you prepare your project for an audit. The controls matrix is a typical approach used in other areas of the business and by auditors, as it provides structure for the assessment of controls.

Copyright 2015 Debbie Gallagher

What to expect when your IT project is audited

In a previous article, I described the various types of auditors that may audit your IT project, and why your project could be selected for audit. In this article, I’ll explain the major steps in the audit for an IT application development or implementation project. In addition, I’ll provide some information on what the audit is likely to cover, and some insight into audit evidence requirements.

Steps in the audit

The audit of your project is also a project, so you can expect similar steps to occur:

·         Planning and Scoping;

·         Execution; and

·         Closing.

Planning the audit

No two IT projects are the same, so before planning can start, the auditor needs to understand the project. To develop this understanding, the auditor will usually meet with IT and business representatives, as well as read background material that describes the project. The auditor will also find out what the audit requirements are. For example, if it is a financial statement audit, he or she will meet with the financial audit team to understand the areas of audit risk and what audit procedures they already have planned to cover that risk.

Based on the audit requirements and the understanding of the project, the auditor will develop a scope of work and an audit plan. Once the scope and plan are approved, the audit moves into execution phase.

Executing the audit

This step is the one you are likely most familiar with. The auditor will meet with you and other project team members to ask more detailed questions and will ask for documentation to support the answers that you provide, and to show that certain areas of the project were well controlled.

Closing the audit

At this stage, the work has been completed, and the auditor will issue recommendations for improvement, usually in the form of a formal report or management letter comments. You or someone else in your organization will also be asked to provide a response to the recommendations. After the responses are gathered, the auditor presents the report to management or to the audit committee.

Areas of audit interest

Although every project is different and the audit scope is defined for each one, there are some areas that are reviewed frequently enough that you should expect the auditor to include them. These areas of interest fall into two categories:

(1) the project itself; and

(2) the new processes and systems.

For the project itself, the auditor will usually be interested in:

·         Approvals for the project;

·         Project governance – for example, meeting minutes, scope change control, management oversight, issues management, go-live readiness assessment;

·         Testing of the new application, including unit testing, interface testing, integration testing, performance testing, business sign offs on test results;

·         Customization of the application (if it’s a package);

·         Security related to the data, the development environment, and application configuration; and

·         Completeness and accuracy of the data loaded to the new system, as well as data integrity.

For the new processes and new systems, the auditor is usually interested in:

·         Interfaces carrying data between applications;

·         Security;

·         Segregation of duties;

·         Changes to functionality and businesses processes – e.g. does the application now have multi-currency, or does it now have on-line purchase order approvals?

If you are subject to an annual audit where only a portion of the system is audited each year, the auditor also may need to change the rotation plan so that all elements are in scope for the go-live year of the new systems.

Audit evidence

According to the International Auditing and Assurance Standards Board (IAASB) Handbook, “The auditor should obtain sufficient appropriate audit evidence to be able to draw reasonable conclusions on which to base the audit opinion”. (International Standard on Auditing (ISA) 500, paragraph 2).

It goes on to describe in paragraph 7 that “Sufficiency is the measure of the quantity of audit evidence”, and that “Appropriateness is the measure of the quality of audit evidence”.

You can see that the auditor is required to ask you for relevant documentation, and will review it and test it. Testing means that the auditor performs some procedures to verify the usefulness of the document as audit evidence. There are numerous possible audit procedures available to the auditor and they are not covered here, but here are a couple of illustrations. For example, for your document that shows the reconciliation of data conversion, the auditor may select data from both systems and check that it agrees to what shows on your reconciliation document. Or, to verify your application testing, the auditor may select a sample of completed test cases to see if they have the results documented and signed by the tester.

Some of the examination of the project is done to provide insight for the auditor into areas of risk for the new processes and systems. For example, in reviewing the issues database for the project, the auditor will assess whether issues were tracked and followed up during the project. In addition, he or she will check to see if any high priority and high impact issues were still open at go live. If they are areas of audit interest, the auditor will likely follow up to determine what mitigation strategies were implemented in the new environment to deal with errors that may have occurred due to those open issues.

Conclusion

Because the auditors need to collect evidence as part of their review, you should not throw away any project documentation until after all of the relevant auditors have completed their work.

You can ask early in your project to meet with the various auditors. This should allow you to find out what areas are of particular interest, so that you can make sure your documentation is retained for their work.

Copyright 2015 Debbie Gallagher

Why your IT project is being audited

You’re the manager of a system implementation or custom development project and have just been told that the project is being audited. There’s a possibility that you aren’t exactly thrilled about this. After all, the project is going well, or maybe it’s over and the new system is humming along just fine. A common response by IT folks who haven’t been exposed much to audits is “but I didn’t do anything wrong, so why am I being audited?”  The purpose of this article is to provide some information on the reasons why your IT project could be audited. You’ll notice that the reasons have nothing to do with you!

Types of auditors

First of all, you may have noticed by now that there are different types of auditors checking out your IT environment. Here are some of the most common types of auditors that audit systems and IT processes in your organization every year:

·         Financial statement;

·         Controls certification readiness;

·         Internal audit; and

·         Regulatory.

The financial statement auditors are responsible for signing an opinion regarding the financial statements of your organization. Let’s include here the folks who have to audit your controls for Sarbanes-Oxley, and any other internal controls audits. They will be from the same audit firm as the financial statement auditors and will generally perform one integrated audit to support both the financial statement opinion and the internal controls opinion. The systems focus of these audits is on IT processes and systems that could have an impact on the financial statements.

In preparation for the internal controls certification and audit, your organization may have hired auditors from another firm to help you get ready. They may be documenting your IT environment and providing feedback on control weaknesses, so that you can remediate prior to the controls certification audit. These auditors also will be financial-statement focused.

If your organization has an internal audit department, you will be used to seeing them almost every year. You may have also noticed that their focus is often different from that of the financial statement auditors. The internal auditors go beyond the financial statements to look at systems that affect operations, whether or not they have no financial statement impact. They frequently also have a mandate to verify that the organization’s policies and procedures are being followed, or that value is being achieved for funds spent. The internal audit group may or may not be involved in assisting the financial statement auditors, depending on their mandate in your organization.

Regulators frequently also perform audits. For example, if you work for a financial institution, you are subject to audits by the Office of the Superintendent of Financial Institutions. Other industries also have regulatory auditors, and if you are in one of those industries, you know that these auditors will visit you every year. Their purpose is to verify that relevant regulations are being adhered to by the IT systems and processes in your organization.

The annual audit

The financial statements for your organization rely heavily on the systems in your organization – gone are the days of binders with 14-column paper! In performing their work, the auditors also rely on reports and data from those systems. As a result, the auditors are obligated to verify the management of many aspects of your financial systems.  They will check out security, networking, communications, databases, and application implementation and maintenance. They may also evaluate business continuity planning and IT strategy.

Depending on a variety of factors, the financial statement auditors may look at different aspects of your systems environment each year, say networking this year and databases next year. There may also be elements, like security, that do not follow a rotation plan, so are examined each year.

The internal auditors generally have a specific set of objectives each year, based on your organization’s risk assessment and audit plan. As a result, they will look at different aspects each year that they visit you.

As for regulatory auditors, the work they do will depend on the type of regulatory body they are and other factors relating to the industry and regulatory environment.

The project auditor

In addition to the usual array of annual auditors, you may also receive a series of visits from a specialist in IT project risk. This person or team will be hired by your project sponsor or client, who is looking for regular feedback throughout the project, so that risks can be managed timely. The hiring of this additional audit team is likely to occur in industries where there are a number of stakeholders to satisfy, or when the project is very large and high risk. The project risk audit team is likely to examine your project more frequently and in more detail than the various types of annual auditors.

The project audit

As part of the annual audit planning process, the audit team evaluates your organization and the risks in your environment. In this risk assessment, they are considering inherent risk, which is the level of risk that exists if there are no controls in your environment. So, for example, what would be the impact if everyone in the organization could sign a cheque, or if no one tested that new application system?

The audit team will then focus their work on the areas identified as highest risk. Although the areas of greatest risk vary by type of industry, a new system nearly always has a high probability of being identified as a medium or high risk. As a result, your IT project has a high likelihood of being audited, by the financial statement and controls certification auditors, the certification readiness team, the internal audit team, regulatory auditors, and maybe also project risk auditors.

As noted earlier, the various auditors have different focus areas and objectives. As a result, you may find that your project is audited several times.

It’s not about you

So, as you can see, the project audit isn’t about you at all. It’s driven by the audit team’s assessment of risk.

Copyright 2015 Debbie Gallagher