The challenges of estimating the economic impacts of spending on economic development, and a review of Scottish Enterprise’s approach

This report discusses the broad issues and challenges in assessing the benefits to an economy of economic development spending; such as business support, infrastructure development, inward investment and sector development. It critiques Scottish Enterprise’s (SE’s) approach to estimating economic impacts, drawing comparisons, as appropriate, with recent work completed for the English Regional Development Agencies (RDAs) and the Department for Business, Innovation and Skills, that looked at the impact of the RDAs. The report focuses on whether the overall methodology is appropriate, and provides SE with recommendations for any improvements.

The methodology included a review of the documentation SE has produced to explain its approach, in light of the aims with which it is tasked (as set out in the Scottish Government’s Economic Strategy (GES)). It also included a meeting to discuss SE’s approach.

The report concludes that organisations must address a number of key challenges when attempting to ensure that impact assessment is a key driver for what their organisation does. Thirteen main areas of challenge were identified, although not all of these are relevant to SE’s activities. Those seen as being of central relevance include: the complexity of impact estimation; securing organisational buy in for a corporate approach to evaluation; building the organisational capacity for evaluation and estimating impact; mitigating the effects of selection bias relating to how programmes and projects are selected for estimation of impact; addressing inconsistencies employed by different types of estimation techniques; the use of different methodologies and valuation methods; the use of standardised approaches for collecting data; identifying the differing time paths of impact such as how much time it takes for benefits to accrue; identifying the differing levels of persistence or sustainability of impacts; mitigating the effects of optimism bias; and ensuring adequate explanation and interpretation of complex results to wider audiences. SE’s approach to estimating impact was found to be a sensible one which has been informed by approaches taken by other agencies. Several characteristics of the approach are highlighted as particularly positive: the use of evaluation evidence from a range of sources to inform the estimates; the aim to use survey and beneficiary data to drive estimates of impact; and the flexibility to undertake impact estimation at the appropriate time in the project or programme life cycle.

Recommendations are made for SE in relation to: improving organisational capacity for evaluation and estimating impact; addressing inconsistencies employed by different types of estimation techniques; the use of standardised approaches for collecting data; and exchanging knowledge with other agencies.