This is the third in a series of articles on data analytics and internal audit. This article looks at the audit finding statement: Criteria. The focus will be on the use of data analytics and how the criteria play an important role in the development of analytics to support and inform the audit.
IIA Standard 2210.A3 states that “adequate criteria are needed to evaluate controls.” Simply put, the audit criteria is a set of policies, procedures and requirements against which audit evidence is compared. For example, the audit of payroll may have a criterion that states that employees must be paid in accordance with their job classification and rate. If you are performing a manual audit of a sample of payroll transactions, this may be sufficient. The auditor would select an employee’s pay and identify their classification and pay rate; then compare the actual pay to the pay rate. However, when applying analytics to this criteria, more detailed and specific information must be applied. But let me step back to the audit objective because for me, the data analyst, this will drive all the analytics. I will use an actual example of an audit for which I supplied the analytics and how the back-and-forth between myself and the team lead was required to produce a good analytic result.
Audit Objective: The weekly pay is accurate, timely and in accordance with employee classifications and rates.
As a data analyst examining this objective to determine how to support the auditors, I would first look at the relevant data sources. In this case the main source of data is the payroll transactions file for the period being audited. I would also need a snapshot of the employee master file with the employee classification and the pay rate table. I would relate the pay transaction file to employee master to obtain the employee classification and to the pay rate table to get the associated pay rates. The analysis would be to identify any whose pay did not match their pay rate. This seems to be sufficient to address the stated objective, but in determining the data sources and developing the analytics, I noticed a couple of potential issues.
- What was the scope of the audit? In determining the data source for the pay transactions, I found that there were separate pay systems for full time employees and part time/casual employees.
- The objective was not considering employee who should have been paid and weren’t.
- It was also not considering whether an employee should be paid or not. For example, them may be on leave without pay, terminated (but still on the employee master), etc.
In determining the data sources, I was able to understand the pay process better. The pay transactions were coming from different pay systems: employees on salary; and employees paid by the hour. This prompted me to ask a question about the audit objective. Were they auditing the payroll function or the pay system or both? And a question about the scope. Was the audit looking at all employees (salaried and hourly)?
This raises an important point. The data analyst supporting the audit must understand the business process, data sources, and audit process. This knowledge must be conveyed to the audit team leader to ensure that the implications of the objective and the criteria are understood. I have seen too many organizations that hire a junior programmer to run the analytics for audit. These data analysts are not able to challenge the team leader, to inform the audit process, or provide insight into impact of the analytics being requested on the audit objectives.
A sub-objective (which I considered to be more of a criterion) was that each employee should only be paid once per pay period. From an analytics perspective, I interpreted this as a test for duplicates – employees paid more than once. However, running a duplicates test requires additional criteria, namely how do you identify a duplicate. The team lead said that an employee should only get one check (transaction) per pay period – providing me with the specific criteria I needed: duplicate pay transaction if the same employee, for the same pay period, required two or more checks. The results identified thousands of potential duplicates. (Note: I get particularly worried when there are too many records, or none identified by the analytics.) Too many probably means the criteria for the analytics is incorrect; and none could mean that the logic was incorrect.) I examined the duplicates and noticed that the amounts were different for the duplicate records. This was not part of the criteria, so was not surprising. Digging into the results a bit further, I noticed that the GL account was different. One of the records was typically regular pay, the other was usually overtime. The analysis had shown that overtime was paid by a different process and resulted in a separate pay transaction. This raised a question about the audit scope. Was the audit looking at all pay transactions (regular, overtime, special allowances, bonuses, etc.) or just regular pay.
In the end, the audit decided to focus on regular pay of salaried and hourly employees and to modify the scope to include the notion of ‘all active employees’ . This changed both the audit objective and the scope. Separate audits of overtime and ‘other pay transactions’ were conducted later. It was easy to modify the source data and the criteria to support the different audits.
In another example, I was asked to perform various analysis to support the audit of advanced payments. In my experience, which seemed to agree with the team leader, advances were used when employees were travelling. Unfortunately, neither of us had a good understanding of the advance payments process. While I correctly identified a transaction type that was the source of all travel advances, there were advanced payments that were used for other purposes and had a different transaction type. The result was that the analytics (and audit results) could only be applied to travel advances. We did not know this until the draft report was issued and the client questioned why advanced payment only amounted to $19M when he thought it should have been $26M for the year.
In another audit which was looking to identify people flying business class instead of economy, I obtained the source data from the airline system. However, I used the ‘booking’ data and not the ‘flown’ data. People can change their travel class at check-in, so I did not have complete or accurate data.
The good thing about data analysis is that with the correct data source and logic, you can test 100% of the transactions. You can also quickly and easily refine the logic or even the data source if the initial review of the analytic results is incorrect. I use these examples to highlight several key points about data analysis and criteria. First, applying analytics is not simply coding SQL or other queries. The data analyst must understand what the audit objective, scope and criteria are and how these can be transformed into programming statements. While doing this, the data analyst and team lead will need to be in constant communication to ensure that both parties understand the implications or decisions regarding data sources, analytics criteria, etc. Secondly, verify with the client that you have the correct data source (complete, timely, appropriate) to support the audit objective and scope. Lastly, the criteria must be more specific than for a manual review to code the logic required by the analytic e.g., employees paid twice versus salaried employees should only have one transaction per pay period for regular pay. And the results should be carefully examined before drawing conclusions. In the case of the duplicate pay, looking the results immediately identified a problem with the criteria.