Sampling Guidelines

1.0 Introduction

There are two main types of sampling:

• Statistical sampling is used to draw conclusions about populations.
• Non-statistical sampling is used to establish the existence, and determine the extent of, suspected conditions.

Statistical Sampling

Statistical sampling permits auditors to project characteristics of the sample to the population from which the sample s drawn, whereas non-statistical sampling only allows you to draw conclusion about the sample itself. Statistical sampling also differs from non-statistical sampling in that it allows the measurement of risk through the application of mathematical rules.

In statistical sampling, samples must be randomly chosen such that each item has an equal chance of being selected. Ensuring random representative samples rests on the proper application of probability theory. Two common approaches to sampling are:

• attribute sampling, which is used to estimate error rates in populations; and
• dollar unit sampling (DUS), which is a modified form of variable sampling, and is better suited to estimating quantities of error rather than occurrence rates when financial impacts are of primary interest.

Non-statistical Sampling

Non-statistical sampling is the selection of sample items without following structured techniques or established methods and therefore, cannot be used to project the characteristics of the sample to the population. The most popular form of non-statistical sampling is directed sampling. Directed sampling is employed when auditors suspect errors and wish to obtain evidence to support their suspicions, or find as many cases of suspected items as possible.

Auditors cannot draw any conclusions about populations from directed samples beyond what is actually found. For instance: a sample of all invoices with the same invoice number and date is selected and examined. A total of $400,000 in duplicates is found in $500,000 worth of invoices. Auditors, therefore, can conclude that there is at least $400,000 of duplicates, but they cannot conclude that 80 percent (400/500) of all invoices are duplicates.

3.0 Sampling Steps

Statistical and non-statistical sampling are similar in that they both involve planning, selection and evaluation.

Planning

Planning ensures audit tests are performed in a manner that minimizes non-sampling error. It involves establishing audit objectives so that auditors can determine sampling approaches best fitted to realize these objectives.

Selection

Selection determines how auditors choose sample items and perform required tests. Selecting the proper approach affects all other steps in the audit process, including sampling methodology.

For example, two possible objectives for a financial audit could be: ensure that the maximum dollar error in a population is less than $500,000; and verify if controls are working as intended. The sampling approach applied to achieve each objective may differ.

For the first objective, dollar unit sampling would be ideal. Even though DUS samples favour large dollar items, given the objective in terms of maximum dollars in error, auditors would not have to be as concerned with small dollar items since their impact on the value of dollars in error would be small as well. The sample would focus attention on the higher dollar items, where the risk or potential materiality is higher.

For the second objective, auditors may be concerned about the controls over specific processes, such as credit-card payments. In this case, DUS is not the proper sampling method. Instead, auditors should use attribute sampling, which randomly selects records regardless of their dollar value. The sample item tested can have only one of two attributes—either the control has been properly performed or it has not. The outcome of testing is known as a deviation rate, representing the proportion of transactions tested that have not been processed in accordance with established control procedures.

Evaluation

Evaluation of results enables auditors to formulate conclusions. The sampling approach chosen by auditors will affect what they are in a position to conclude. The primary differences between DUS and attribute sampling are:

• DUS treats each dollar equally, allowing auditors to form conclusions on maximum-dollar errors in populations.
• Attribute sampling treats every record equally, permitting auditors to form conclusions on deviation rates in populations but not the dollar impact of deviations.

Sampling enables auditors to:

• understand the businesses of audit subject areas;
• develop clear statements of audit objectives;
• formulate and test hypotheses;
• define appropriate populations; and
• fulfill audit objectives.

4.0 Understanding the Businesses of Audit Subject Areas

Understanding the businesses of audit subjects helps auditors determine the scope and objectives of audits; it also assists auditors in devising appropriate sampling approaches. Several ACL techniques can be employed, including:

• summarizing or classifying data;
• stratifying data to determine ranges;
• identifying potential duplicates;
• sorting or indexing data to examine transactions at the top and bottom of data ranges; and
• joining or relating data from previous years, or other audit subjects, to examine trends across time or organizations.

In addition, three ACL tests enable auditors to examine patterns in data:

• Benford’s Law examines the frequency distribution of the first ‘x’ digits of amounts and compares them to theoretical distributions.
• MAX/MIN identifies ranges between highest and lowest, and ratios of highest to second highest (available in version 12+ as part of Classify or Summarize).
• Amounts that occur frequently or infrequently.

Sampling during early phases of audits helps auditors develop sufficient understanding of audit subject areas, and determine the scope and objectives of audits.

5.0 Developing Clear Statements of Audit Objectives

Audit objectives often dictate approaches used to satisfy audit requirements—the use of directed sampling or statistical sampling for instance. Assurance audits could employ statistical sampling, while audits conducted to determine the existence of a problem or to assess the adequacy of a control framework could use directed sampling.  In other cases, a combination of direct and statistical sampling may provide the best results.

6.0 Formulating and Testing Hypotheses using Directed Samples

Directed sampling assesses all transactions for specific criteria faster and easier than statistical sampling. The first step to successful directed sampling is creating accurate statements of hypotheses—either negative or positive. The next step is to establish criteria, using directed sampling to select transactions for review. Finally, review results are examined to prove or disprove the hypotheses.

The following scenario provides an example of hypotheses formulation and directed (non-statistical) sampling in action.

Example 1: 

1. Hypothesis: 90 percent of invoices are paid on time.
2. Define criteria to highlight invoices that are paid late (more than 45 days following receipt of invoice).
3. Extract selected records and verify a sample to determine if these records are late payments and whether late payment is a systemic problem. If the records prove to be late payments, the hypothesis is rejected.
4. Total the amount of all transactions identified as late payments.

 7.0 Identifying Populations, Sampling Units and Items

Auditors must carefully identify populations, sampling units and items. In particular, auditors must consider audit objectives to ensure that populations and sampling units are appropriate.

Populations are entire sets of data from which samples are selected, and about which auditors wish to draw conclusions. Populations should be defined in the same terms as sampling items.

Sampling units are the type of record selected in the sampling process—for instance, an individual transaction, record or dollar.

 Sampling items are the individual items within populations that are reviewed. For instance, in attribute sampling, auditors review items selected; for DUS sampling, auditors review items containing selected dollars.

Populations should also be appropriate for audit objectives. For example, if auditors are testing an accounts-payable process and only include invoices in the population, then the population may not appropriate or complete since many other types of transactions affect the accounts-payable process, such as credit-card payments. Further, if auditors only include transactions from periods 1 to 12, they will not have a complete population if transactions can be posted in periods 13 and 14.

Sampling units and items should be given careful consideration. The definition of sampling units and items will affect populations, and may be influenced by sampling methods. For example, when sampling financial events, are auditors selecting lines from a financial document (e.g. an invoice) or the entire document? If auditors sample individual lines, then they cannot examine an entire invoice when determining the amount of an error.

Definitions of populations, and sampling units and items are critical to making defensible and meaningful conclusions of sample errors to populations, and thus meeting audit objectives.

8.0 Determining Sample Size

Sampling is only valid if samples selected for testing are representative of populations such that the incidence of error or deviation in the samples closely approximates the errors or deviations in the populations. In determining sample size, the main objective of auditors is reducing sample risk to an acceptable level for audit purposes while keeping audit work to a manageable level. The level of sampling risk has an inverse relationship with sample size.

Surprisingly, the population size for attribute sampling has a negligible effect on the sample size for populations over 10,000. Some factors that influence sample size are:

Factor	Effect on Sample Size
Increase in auditors’ intended reliance on the control system	Increase
Increase in rate of errors or deviations auditors are willing to accept	Decrease
Increase in total error auditors are willing to accept (tolerable error)	Decrease
Increase in rate of errors or deviations auditors expect to find in populations (expected error)	Increase
Increase in auditors’ required confidence level	Increase
Stratification of populations when appropriate	Decrease
Increase in population size	Negligible effect

 

Notes

• Expected error rate will have an effect on sample size. If the expected error rate is not known, auditors can take a small preliminary sample, the results of which can be used to define the exception rate for the audit sample. A new sample is drawn with the revised expected error rate and the results from the first sample are included in the ultimate sample. For example, if 30 items are chosen for the preliminary sample (to estimate the expected error rate) and, using the expected error rate, the total sample size is determined to be 100 items, auditors need to select an additional 70 items only.
• ACL should NOT be used to determine sample size when expected error rate is high (e.g. 10%). ACL, however, can still be used to select sample records.

 

Determining sample size using:

ACL should be used to determine the sample size when the audit objective relates to simply verifying whether or not an error rate in the population is below a certain level.  For example, in a financial audit you may want to verify that the number of incorrect invoices is at most 2 percent in the population.  ACL will determine the sample size and the maximum number of errors (maximum tolerable errors).

NOTE: If you have more than the maximum tolerable errors, you cannot verify the population error rate is at most 2 percent.

 

9.0 Selecting Samples

Auditors may employ a number of ways to select samples. Two fundamental rules, however, must always be followed when making statistical sample selections.

• Auditors must know populations, because audit opinions will be based on what has been sampled rather than on what has not been sampled; and
• Each unit in a population must have an equal, or known, chance at being selected.

For example, while DUS favours high dollar items, low dollar items still have a chance of being selected. If a population, however, includes negative- or zero-value items, these items have no chance of being selected by DUS. Accordingly, auditors have broken the fundamental principle that every item should have a chance of being selected.  In addition, DUS is not appropriate for testing for understatement and the error cannot be more than the reported book value of the item.

Note

 

• For DUS, ACL will automatically address the issue of negative-valued items by making selections based on absolute value. Population amounts, therefore, should also be the absolute value. Alternatively, auditors can perform a separate sample of negative-valued items. Zero-value items can also be treated separately, but that is beyond the scope of this guideline.

10.0 Defining Errors and Deviations Being Sought

Auditors must consider what constitutes an error by referring to audit objectives. The type of error or deviation is related to objectives. Before drawing samples, audit teams should be clear on what types of errors or deviations are being sought, and what evidence is required to assess whether sample items are in error or not.

Audits may also reveal errors not specifically sought. For example, while examining invoices to verify whether invoice amounts agree with contract amounts, auditors may notice some other discrepancies in the invoices, such as errors in financial coding on vendor invoices. Before projecting to a population, auditors should ensure that audit objectives, population, sampling plan and previous sample examinations have been validated or amended to reflect this change in purpose.

11.0 Performing Audit Procedures

Having drawn samples, auditors should then perform proper audit procedures to test audit objectives on each item selected. If several members of an audit team are involved in testing, it is imperative that all members apply the same criteria in a consistent manner to ensure that results are identical regardless of which auditor reviewed the sample item. Initially, this procedure may require all team members to review the same transactions, carefully discussing any discrepancies and documenting more precise criteria for further review. Results of the testing should be carefully and completely documented, with team leaders performing quality-assurance of results.

12.0 Evaluating Results

Statistical sampling is used to project quantifiable sample results to populations. Auditors, however, should not be content with mathematical results of their audit samples. While these results provide measurable assurance that samples are representative of audited populations, as well as provide objective estimates of the number of errors in these populations or the maximum dollar values of these errors, these results are not necessarily what the management of audit subjects areas need or want.  Auditors should be able to identify causes and effects of errors, and impact of findings. Further, auditors should consider whether errors affect entire populations, or are isolated or localized occurrences. For example, if sample errors are primarily evident at one branch or region, auditors should not necessarily be projecting these errors to an entire department.

13.0 Projecting Errors to Populations

 

With attribute sampling, you are using a sample to test your hypothesis (e.g. we want test at 95 percent confident that the error rate is no larger than 4 percent with a precision level of +/-2 percent).

If the error rate in the sample is no higher than 6 percent and no lower than 2 percent (i.e. falls in the range 4 percent +/- 2 percent) you can conclude that your hypothesis is true.  Thus, if you took a sample of 60 items and found two deviations (a sample error rate of 3.3 percent), then you can conclude that 19 times out of 20 (95 percent) the projected deviation rate (error rate) for the population will be 4 percent +/-2 percent.

If the sample error rate differs from the expected rate (4 percent) by more than the level of precision (+/- 2 percent) (e.g. a 6.5% error rate in the sample), you cannot conclude that there is a 4 percent +/- 2 percent error rate in the population.  In this case, you must either reassess your hypothesized error rate by using the error rate found in the sample and recalculating the sample size, or project the sample error rate without stating a level of confidence.

• Expected error: hypothesized rate of error or deviation expected in a population based on experience, part audits, or industry/corporate standards.
• Precision level: confidence interval around the expected error rate (expressed as a percentage).

For DUS samples, projections to populations are not expected errors but statements of the maximum dollar value of the errors. When extrapolating errors in samples to entire populations, auditors must determine the percentages of ‘sample dollars’ in the errors and apply these figures to the values of entire populations.

For example, if auditors draw $300 from a population of $100 million, each dollar sampled is tied to a record. Auditors examine the 300 dollars selected by looking at the 300 records (invoices, for example). If auditors find 15 records each with a 100-percent rate of error, the auditors can conclude that 15 of the 300 dollars are in error, yielding an error of 5 percent (15/300). This error percentage can be applied to the population: five percent of $100 million equals a maximum of $5 million of errors in the population.

If a sample record is partially wrong, however, auditors are not sure if the sampled dollar is in error or not. For example, if an amount paid was $200 and an audit invoice amount was $190, then only $10 was in error. In this case, auditors can consider $0.05 of the sampled dollar to be in error (10/200). So, if auditors found a total of 60 errors, and these errors were all of this type, auditors would consider the total amount of sampled dollars in error to be 60*$0.05=$3.00. Auditors would use this amount to calculate the percentage dollars in error (3/300=1 percent) and apply this percentage to the total value of the population (1 percent of $100 million=$1 million).

The approach becomes increasingly complicated when there is a combination of 100 percent error and partial errors; however, the basic approach is as follows:

• Determine the amount of the sampled dollar in error (error/book value).
• Total the amounts of the sampled dollars in error.
• Compute percentage error (total of sample-dollar error amounts/total number of sampled dollars).
• Apply percentage error to entire population (percentage error*total value of population).

For example, given the errors and book values below, for a population of $100 million and a DUS of 300 items, calculations are as follows:

Error Amount         Book Value           Sample Dollar        Error Calc

$ 12,000                   $12,000                $12,000/12,000       =   1.00

$ 75                           $100                     $75/100                       = 0.75

$ 300                       $500                     $300/500                    = 0.60

$ 2,000                   $2,000                  $2,000/2,000            = 1.00

Total                                                                                                  3.35

Percentage error is 3.35/300=1.12

Error in population is 1.12% of $100M = $1.12M

14.0 Conclusion

Sampling can be an effective audit tool when auditors use it with an understanding of what they are assessing (population) and the select a proper sample—both in size and items. Audit objectives will drive sampling approaches and methodologies. In many cases, a combination of statistical and directed sampling can produce valid results.

Sampling, however, can easily produce results that are not representative and cannot be defended. Care should be exercised, therefore, when employing directed and statistical sampling. Moreover, auditors are strongly encouraged to involve the methodology section at every step of the sampling process.  In addition, where required, the services of a statistician should be sought.