ROM 10-04 - Review of Controls Over Job Creation and Retention Statistics Reported by SBA under the American Recovery and Reinvestment Act of 2009

To: Eric R. Zarnikow

Associate Administrator for Capital Access

/s/ Original Signed

Date: December 4, 2009

From: Debra S. Ritt

Assistant Inspector General for Auditing

Subject: Review of Controls Over Job Creation and Retention Statistics Reported by SBA under the American Recovery and Reinvestment Act of 2009

ROM-10-04

This memorandum presents the results of our initial review1 of job creation and retention statistics reported by the Small Business Administration (SBA) under the American Recovery and Reinvestment Act of 2009 (Recovery Act). The Recovery Act provided SBA with $730 million to expand the Agency’s lending and investment programs and to create new programs to stimulate lending to small businesses. The additional funding was intended to stimulate the economy by promoting job creation and preserving existing jobs. SBA selected job creation and retention as one of its performance measures to assess the success of its Recovery Act programs. Based on our initial review of program statistics reported by SBA, we identified issues involving the reasonableness of reported job creation and retention statistics for loans approved in June 2009, which raised questions about whether SBA had the appropriate controls to ensure the integrity of its job data. At the time of our audit, the June 2009 data was the only full month of Recovery Act data available for review.

To assess the reasonableness of job creation and retention statistics reported for June 2009, we examined job data maintained in SBA’s Electronic Transaction System (E-Tran). System records showed that lenders approved 3,841 Recovery Act 7(a) loans in June 2009 valued at $957 million and 182 American Recovery Capital (ARC) loans valued at $6.1 million. To ensure that we had the appropriate data, we compared it to the statistics reported by SBA in its June 2009 Monthly Recovery Act Program Performance Report.

Using data analysis software, we analyzed the reported job creation and retention statistics to identify statistical outliers. We calculated the mean and standard deviation for the 3,841 7(a) Recovery Act loans, and using this information, we calculated the z-scores or “standard scores” for jobs reported on each loan. The z-score indicates how many standard deviations each item is from the mean. For purposes of this review, we considered loans with z-scores exceeding +/- 3 to be statistical outliers. See Appendix II for more details on our statistical analysis.

We also examined how jobs were reported for multiple loans to the same businesses. In addition, we interviewed SBA officials from the Office of Capital Access, reviewed the 7(a) loan application to determine how job data is captured for 7(a) loans, and reviewed guidance to lenders for reporting job creation and retention statistics for 7(a) loans. We performed our analysis on job data from August 2009 to September 2009 in accordance with Government Auditing Standards prescribed by the Comptroller General of the United States.

BACKGROUND

On February 17, 2009, President Obama signed the Recovery Act into law to stimulate lending and promote economic recovery. The Recovery Act called for unprecedented levels of transparency and accountability in carrying out Agency programs and established the Recovery Accountability and Transparency (RAT) Board to coordinate and conduct oversight of funds distributed under the Act in order to prevent fraud, waste, and abuse. Subsequent guidance issued by the Office of Management and Budget (OMB),2 required Agencies to develop performance measures to assess the quantifiable outcomes of their Recovery Act programs. One of the performance measures selected by SBA for its Recovery Act loan programs was job creation and retention.

Shortly after its formation, the RAT Board emphasized to Offices of Inspector General (OIG) the importance of ensuring the reliability of job creation and retention data being reported by Federal agencies. In response to the RAT Board’s concern, in May 2009, we performed a preliminary review of SBA’s job creation and retention data for the 7(a) loan program. Upon completion, we informed SBA officials that SBA’s E-Tran, which contains the data used to report Recovery Act performance statistics, included erroneous loan approvals,3 management removed erroneous loan approvals from its reported loan approval statistics, but did not address the other deficiencies.

SBA’s Monthly Recovery Act Program Performance Report (See Appendix I) included 44,990 jobs created or retained by businesses that received 7(a) Recovery Act loans4 and 2,000 jobs created or retained by businesses that received American Recovery Capital (ARC) loans in June 2009. This job data was self-reported by borrowers on SBA 7(a) loan applications and submitted to SBA by lenders. Lenders are not required to validate the borrower-reported information.

RESULTS IN BRIEF

The SBA-reported number of jobs created and/or retained in June 2009 may have been overstated by at least 16 percent because it included (1) statistical outliers, (2) negative jobs reported, (3) double-counted jobs, and (4) jobs created by ARC loans. We identified 40 statistical outliers that resulted in an apparent overstatement of 6,200 jobs, or 14 percent of the 44,990 total jobs reported. We also identified negative job numbers reported on three loans, which demonstrated a data input control weakness. Reported jobs also may be overstated as a result of double counting. Some of this occurred because SBA did not take into account that the same jobs were reported multiple times for businesses that received multiple loans. In addition, some lenders reported new jobs as both jobs created and jobs retained, resulting in a 2 percent overstatement of jobs created/retained by 7(a) loans. Furthermore, even though SBA elected to use job retention as its performance measure for the ARC loan program, 5 percent of the jobs associated with ARC loans were reported as new jobs.

The reported job creation and retention statistics may be further misleading as many of the loans causing job creation/retention had not been disbursed. As of June 30, 2009, SBA approved 7(a) Recovery Act loans totaling $3.1 billion, which were reported to have created or retained approximately 155,000 jobs. However, only $1.7 billion of the approved loans, representing 95,000 jobs, had been disbursed. SBA could improve the transparency of the reported job data if it also disclosed the number of jobs associated with disbursed loans.

SBA’s reported job statistics may be inaccurate because SBA’s E-Tran system did not include the necessary input, reporting, and documentation controls to ensure that job statistics were properly recorded, classified, and accounted for. While we notified SBA officials of this in May 2009, as of September 10, 2009, SBA had not taken appropriate corrective action to implement the necessary controls. Furthermore, unlike the 504 loan program, SBA has not provided lenders with adequate guidance on how to report job creation and retention statistics for 7(a) loans. As a result, lenders have employed inconsistent methodologies when reporting job statistics.

We recommended that the Associate Administrator for Capital Access:

(1) implement the necessary controls to check the reasonableness of job data;

(2) implement a data quality review and testing process to determine if job data is properly recorded, classified, and reported; (3) determine whether the new jobs reported for ARC loans are data anomalies and if not, revise performance measures to report on jobs created; (4) improve the transparency of the reported data by disclosing the number of jobs associated with disbursed loans; and

(5) establish adequate guidance on how lenders should report job statistics for 7(a) loans.

Management agreed with recommendations 1, 2 and 3, disagreed with recommendation 4, and neither agreed nor disagreed with recommendation 5. Management stated it was committed to accurate and transparent reporting regarding the impact of its Recovery Act programs in stimulating the economy and supporting job creation and retention. Management also acknowledged the challenges associated with measuring the job creation impact and agreed to take additional steps to enhance the data quality and controls related to the job creation metrics.

RESULTS

Job Creation and Retention Figures in SBA’s June 2009 Report Were Overstated

SBA may have overstated the reported number of jobs created and retained in its June 2009 Recovery Act Program Performance Report by at least 16 percent. Our review identified anomalies in SBA’s reported job data that occurred because the data was not adequately reviewed for reasonableness. We also determined that jobs were double counted for businesses receiving multiple loans and when lenders reported the same jobs as being both created and retained.

SBA Did Not Establish Reasonableness Checks to Prevent Anomalies in Reported Job Creation and Retention Data

The majority of reported jobs per loan fell within statistically reasonable ranges. However, we identified a small number of loans that represented statistical outliers5 to have created and/or retained 16 percent of the total jobs reported. Based on SBA’s estimate that one job is created for every $26,000 in 7(a) loan value or $9,000 in SBAExpress loan value, job creation and retention figures for these 40 loans may have been overstated by approximately 88 percent. Based on these results, we estimate that 6,200 jobs, or 14 percent of the 44,990 total jobs reported in June 2009 may be overstated. Although SBA was aware of the outliers, they were included in the Agency’s June 2009 Recovery Act Program Performance Report. For example, SBA reported that a $658,000 loan created/retained 700 jobs when, using SBA’s estimation criteria, it was expected to result in only 25 jobs. Another $350,000 loan was reported to create/retain 600 jobs, when it should have been estimated to result in only 13. See Appendix III for details on all 40 outliers.

We also identified three loans in SBA’s reported statistics that had negative job numbers. For example, -20 jobs were reported for a $570,000 loan. These loans did not qualify as statistical outliers and did not materially impact the overstatement of jobs reported. However, they demonstrate an input control weakness that SBA should address.

Job Creation and Retention Statistics Included Jobs that Appeared to Be Double Counted

The apparent overstatement of jobs in June 2009 was also due to the double counting of jobs when aggregating multiple loans to the same businesses. We determined that 224 businesses received 2 or more of the 3,841 loans approved in June 2009. While it is often appropriate for a business to receive multiple 7(a) loans, in many cases, lenders reported to SBA the same number of jobs created and retained for each of the borrower’s loans. For example, one business was approved for 2 loans on the same day in June, totaling $75,000 and $125,000 and the number of jobs reported to have been created or retained under each loan was 85. As a result, SBA reporting reflected a total of 170 jobs created or retained by this business. We did not quantify how much the statistics were overstated as a result of the apparent double counting of jobs associated with multiple loans to the 224 businesses, as it would have required a detailed review of each loan file.

Lenders also double counted loans because job data on 7(a) loan applications was categorized differently than that shown in the E-Tran system used to transmit the statistics to SBA. For example, the 7(a) loan application requires borrowers to report “the number of employees at loan application” (jobs retained) and “the number of employees if the loan is approved,” (jobs created and retained). This information is captured in three different fields in E-Tran–“current employee quantity,” “jobs created quantity,” and “jobs retained quantity.” We noted that some lenders were using information from the loan application incorrectly, which resulted in the reporting of new jobs as both jobs created and jobs retained. Such reporting appeared to have resulted in a 2-percent overstatement of jobs reported for 7(a) loans in June 2009.

SBA Lacked the Necessary System Controls and Lender Guidance to Ensure the Reasonableness of Reported Job Creation and Retention Statistics

In accordance with OMB Circular A-123, SBA is required to implement the appropriate controls to ensure that job data is properly recorded, classified, and accounted for. However, SBA’s E-Tran system did not include input controls, or reasonableness checks, to ensure that the number of jobs reported was appropriate relative to the dollar value of each loan. For example, reasonableness checks should have been embedded in E-Tran to prompt lenders or prevent submission of their reports when (1) the number of reported jobs fell outside an acceptable range, (2) negative job figures were entered, or (3) a previous loan was made to the same business. While we notified SBA officials of the missing controls in May 2009, as of September 10, 2009, SBA had not taken appropriate action to implement the necessary controls to be responsive to the transparency and accountability requirements of the Recovery Act. In addition, SBA did not implement a data quality review and testing process to determine if job data was properly recorded, classified, and reported. This is necessary regardless of the input controls that are implemented to ensure the accuracy and integrity of the reported data.

SBA also had not provided lenders with specific guidance on how to report job creation and retention statistics for 7(a) loans in E-Tran. While SBA issued an information notice on September 22, 2009, explaining its job reporting methodology for Capital Access Programs, the notice did not provide instructions for reporting job retention and creation statistics for 7(a) loans. Because we found that lenders applied inconsistent methodologies when reporting job statistics, providing specific instructions for E-Tran data entry would help prevent future misreporting of job statistics from borrower-provided information.

SBA Counted New Jobs toward Meeting its Job Retention Goals for the ARC Loan Program

For all jobs reported for the ARC loan program in June 2009, 91, or 5 percent, represented new jobs. However, this program was established to help sustain businesses and retain jobs by providing loans of up to $35,000 to businesses experiencing financial hardship. Consequently, SBA defined a goal of retaining 31,000 jobs. SBA will need to determine whether the new jobs reported are data anomalies, and if not, revise its performance measure to report on jobs created.

Transparency of Reported Job Statistics Could Be Improved

The reported job creation and retention statistics for June 2009 were further misleading because they were based solely on loans that had been approved, regardless of whether they were ever disbursed. Between February 17, 2009, and June 30, 2009, SBA approved Recovery Act loans totaling $3.1 billion that it reported had created or retained approximately 155,000 jobs. However, lenders disbursed only $1.7 billion, or 55 percent of the total loan dollars approved, which resulted in the creation or retention of 95,000 jobs. We believe that reporting on just approved loans may inflate the job statistics and economic impact reported as many loans are cancelled after approval or not disbursed immediately. To its credit, SBA did disclose that approximately 15 to 20 percent of the gross loan approval value would not get disbursed, however, greater transparency would be achieved if SBA were to also report those jobs associated with disbursed loans as the real economic impact from job creation and retention is realized when loans are disbursed.

RECOMMENDATIONS

We recommend that the Associate Administrator for Capital Access:

Implement the necessary controls to check the reasonableness of data, including user prompts, range checks, and the prevention of negative figures, in E-Tran to ensure the accuracy of lender-reported job creation and retention statistics.

Implement a data quality review and testing process to determine if job data is properly recorded, classified, and reported.

Determine whether the new jobs reported for ARC loans are data anomalies and if not, revise performance measures to report on jobs created.

Improve the transparency and accountability of the reported data by disclosing the number of jobs associated with disbursed loans in the monthly Recovery Act Program Performance Reports.

Establish adequate guidance on how lenders should report jobs created and retained for 7(a) loans.

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL RESPONSE

On November 10, 2009, we provided a draft of this report to SBA for comment. On November 30, 2009, SBA submitted its formal comments, which are contained in their entirety in Appendix IV. Management agreed with recommendations 1, 2 and 3, disagreed with recommendation 4, and neither agreed nor disagreed with recommendation 5. Management stated it was committed to accurate and transparent reporting regarding the impact of its Recovery Act programs in stimulating the economy and supporting job creation and retention. Management also acknowledged the challenges associated with measuring the job creation impact and agreed to take additional steps to enhance the data quality and controls related to the job creation metrics. Specific management comments on the report recommendations, and our evaluation of them, are summarized below.

Recommendation 1

Management Comments

Management agreed with the recommendation and stated it was looking at the Agency’s systems, including E-Tran, to determine the appropriate long and short term business rules around fields for job creation and retention numbers.

OIG Response

Management’s comments were responsive to this recommendation. In response to the final report, please identify the specific controls you plan to implement to ensure the accuracy of lender-reported statistics and a target date for final action.

Recommendation 2

Management Comments

Management agreed with the recommendation and is in the process of procuring a data quality contractor to assist in the review and testing of Recovery Act program data, including job creation data.

OIG Response

Management’s comments were responsive to this recommendation. In response to the final report, please explain how you plan to implement a data quality review and testing process to determine if job data is properly recorded, classified and reported and provide a target date for final action.

Recommendation 3

Management Comments

Management agreed with this recommendation and stated that although the ARC loan program was established to help sustain small businesses, it is also possible that these loans help employers create new jobs. As a result, management stated it will review the information reported for job creation in the ARC loan program and determine whether to revise goals and reporting to include jobs created and retained.

OIG Response

Management’s comments were responsive. In response to the final report, please provide a target date for final action.

Recommendation 4

Management Comments

Management respectfully disagreed with this recommendation. Management stated that it considered our recommendation of reporting job creation both at the point of approval and disbursement. Management decided, however, that it was more appropriate to report job creation at the point of approval since disbursements can lag for several months and cross into another fiscal year, increasing the complexity of the reporting process. Furthermore, management stated that both SBA and OMB believe that reporting information for both approved and disbursed loans would be confusing to users of the information. Management stated it would continue to adjust the job creation and retention figures at the end of the fiscal year to reflect only those jobs that were reported as created or retained for loan approvals, net of cancellations.

OIG Response

We continue to support our position that SBA could improve the transparency and accountability of the reported data by reporting the number of jobs associated with disbursed loans in its monthly Recovery Act Program Performance Reports. We made this recommendation because the economic impact from job creation and retention of 7(a) loans is not realized until these loans are disbursed. Our recommendation was intended to provide an additional and more substantive measure of the Recovery Act’s program impact. Furthermore, we are not requesting that these loans be reported on a fiscal year basis, and therefore, do not believe our recommended actions would increase the complexity of the reporting process. We plan to discuss with OMB officials any concerns they may have with our recommendation. We will seek a management decision for this recommendation through the audit resolution process.

Recommendation 5

Management Comments

Management stated that unlike the 504 program, job creation/retention is not a required element of the 7(a) program. Further, management stated that SBA makes clear in its reporting of job creation and retention figures that they are self-reported by borrowers. Nevertheless, management stated it would review the consistency of the self-reported job creation/retention data and determine what level of guidance might be appropriate for borrowers, who include this information on their loan forms.

OIG Response

Management’s comments were not responsive to this recommendation as it did not specifically state whether it agreed or disagreed with the recommendation. We continue to support our position that guidance should be provided to the lenders on how to report jobs created and retained for 7(a) loans. As explained in our audit report, borrowers are required to report “the number of employees at loan application” and “the number of employees if the loan is approved” on their loan applications. It is the lenders’ responsibility, however, to report to SBA the number of jobs created and retained and it is this lender-reported data that is used by SBA in its monthly Recovery Act Performance Reports. We will seek a management decision for this recommendation through the audit resolution process.

ACTIONS REQUIRED

Please provide a written response providing additional details for implementing the recommendations within 30 days from issuance of this report, using form 1824 for all recommendations in our report.

We appreciate the courtesies and cooperation of the Office of Capital Access during this audit. If you have any questions concerning this report, please call me at (202) 205-[FOIA ex. 2] or Debra Mayer, Director, Recovery Oversight Group, at (202) 205-[FOIA ex. 2].

APPENDIX I. JUNE 2009 RECOVERY ACT PROGRAM PERFORMANCE REPORT

To view Appendix I, please see the attached document.

APPENDIX II. STATISTICAL ANALYSIS

Using IDEA® data analysis software (IDEA), we performed a statistical analysis of the job data reported for the 3,841 7(a) Recovery Act loans approved in June 2009. This included 2,333 7(a) loans(6) and 1,508 SBA Express loans. The loans were separated in this manner to be consistent with SBA’s methodology for calculating the dollar value of loans per job created or retained. SBA estimated that the SBA Express loan program created one job for every $9,000 in loan value, while all other 7(a) loans created one job for every $26,000 in loan value.

For both groups of loans, IDEA calculated the total reported job mean and standard deviation. The mean is the natural center or origin of the values in the universe and the standard deviation is the distribution of the data. Using this information, we calculated the z-scores or standard scores for each loan. The z-score indicates how many standard deviations each item is from the mean and is calculated as follows, where x represents the jobs reported, m represents the job mean and sd is the standard deviation:

Z = sd/x – m

According to the Empirical Rule of statistics, approximately 99.7-percent of all items in a data set lie within a range of +/- 3 standard deviations from the sample mean and loans outside of +/- 3 standard deviations are outliers. Therefore, we considered loans with z-scores beyond + / - 3 to be statistical outliers.

APPENDIX III. JUNE 2009 JOB CREATION AND RETENTION OUTLIERS

To view Appendix III, please see the attached document.

APPENDIX IV. AGENCY COMMENTS

U.S. SMALL BUSINESS ADMINISTRATION

WASHINGTON, D.C. 20416

DATE: November 30, 2009

TO: Debra S. Ritt

Assistant Inspector General for Auditing

/s/ Original Signed

FROM: Eric R. Zarnikow

Associate Administrator for Capital Access

SUBJECT: Draft Report on the Review of Controls Over Job Creation and Retention Statistics Reported by SBA under the American Recovery and Reinvestment Act of 2009. Project 9514A

Thank you for providing the opportunity to comment on this report.

The Small Business Administration is committed to accurate and transparent reporting regarding the impact of its Recovery Act programs in stimulating the economy and supporting job creation and retention.

We are proud to report that Recovery Act funding of the SBA’s 7(a) and 504 loan programs have played a significant role in boosting access to capital and helping create and retain jobs for small businesses. Increased loan guarantees in the 7(a) program and reduced fees in both the 7(a) and 504 loan programs have led to an increase in SBA loan volumes of more than 75 percent in the weeks following passage of the Recovery Act compared to the weeks preceding the Recovery Act's passage. This has translated into nearly $15 billion of lending support to small businesses across the country. These small businesses are reporting that 7(a) and 504 loans approved will help them create over 136,000 jobs and retain 290,000 more. While these reported job figures are not included in the direct recipient reports collected through the Office of Management and Budget to track the impact of the Recovery Act, SBA is committed to helping ensure that this borrower-reported information is materially accurate.

At the same time, we acknowledge some of the challenges associated with measuring the job creation impact of SBA’s Recovery Act programs. Throughout the implementation of the Recovery Act, we have been actively engaged in discussions across the SBA and OMB to identify the most appropriate reporting performance metrics—particularly given the temporary nature of these Recovery Act programs.

We appreciate and agree with several of the Inspector General’s recommendations in this review, as we believe that we can take additional steps to help enhance the data quality and controls related to the job creation metrics.

Following are the Agency’s responses to the IG’s recommendations:

Recommendation 1: Implement the necessary controls to check the reasonableness of data, including user prompts, range checks, and the prevention of negative figure, in E-Tran to ensure the accuracy of lender-reported job creation and retention statistics.

And, Recommendation 2: Implement a data quality review and testing process to determine if job data is properly recorded, classified and reported.

We agree with these recommendations, and the SBA is currently considering additional controls to ensure improved data quality. We are very focused on these issues, and have committed additional resources to address them. In July funds were approved for OCA to hire a data quality contractor to assist in the review and testing of Recovery Act program data, including job creation data. We have been in the process of procuring the data quality contract.

We are looking at our systems, including E-Tran, to determine appropriate long and short term business rules around fields for job creation and retention numbers.

Recommendation 3: Determine whether the new jobs reported for ARC loans are data anomalies and if not, revise performance measures to report on jobs created.

Before SBA’s ARC program was launched, the Agency made a logical assumption that that the most significant job impact from the program would likely be job retention rather than creation. Although this program was established to help sustain small businesses and retain jobs, it is also possible that these loans help employers to create new jobs.

SBA agrees with this recommendation. SBA will review information reported for job creation in the ARC program and determine whether to revise goals and reporting to include jobs created and retained.

Recommendation 4: Improve the transparency and accountability of the reported data by disclosing the number of jobs associated with disbursed loans

SBA respectfully disagrees with this recommendation. The Agency has considered the IG’s recommendation of reporting job creation both at the point of approval and disbursement. However, given guidance from OMB, the SBA has decided that it would be most appropriate to maintain reporting on approvals, so job creation impact can be most closely tied with the timing of program activity—particularly as disbursement could lag for several months after loan approval, could cross into another fiscal year, and could increase complexity of the reporting process. Both the Agency and OMB believe that reporting information on both approved and disbursed loans would be confusing to users of the information.

To ensure transparency, clarity and consistency, the SBA will continue to measure job creation and retention from its Recovery Act loan programs using a methodology similar to the one used for reporting of non-ARRA job creation/retention in SBA’s Fiscal Year 2009 annual report. These figures will be adjusted at the end of the fiscal year to reflect only the jobs that are reported as created or retained on gross approvals net of cancellations.

Recommendation 5: Establish adequate guidance on how lenders should report jobs created and retained for 7(a) loans.

The 7(a) program is open to a broad range of businesses for diverse purposes; unlike the 504 program, job creation/retention is not required element of the 7(a) program. SBA makes clear in its reporting of job creation and retention figures that they are self-reported by borrowers.

SBA will review the consistency of self-reported job creation/retention data and determine what level of guidance might be appropriate for borrowers, who include this information on their loan forms.

Thank you again for your review, and we look forward to continued engagement and discussion with you on this critical issue.

FOOTNOTES

1Our initial review focused on the reasonableness of job statistics reported by SBA and did not include a validation of the statistics against information reported by borrowers.

2 Memorandum M-09-15, Updated Implementing Guidance for the American Recovery and Reinvestment Act of 2009. questionable job statistics on several loans, and double counting of jobs created and/or retained from multiple loans to the same small businesses. As a result of this discussion,

3 Erroneous loan approvals are those approved for $0.

4 These loans do not include American Recovery Capital (ARC) loans.

5 Statistical outliers are those loans for which total reported jobs exceed three standard deviations of the mean i.e. have z-scores exceeding +/- 3. See Appendix II for details on our statistical analysis. that may have caused job data to be significantly overstated. Specifically, 40, or 1 percent, of 7(a) loans approved in June 2009 were reported.

6 These include all types of 7(a) loans except for SBA Express loans.