public procurement effectiveness via bibliographic and
case studies is performed. As a result, the majority of methods cover four
components of assessing the public procurement efficiency - targeted
efficiency, cost-effectiveness, organizational efficiency, efficiency of budget
expenditures for public procurement. This does not provide an assessment of the
automated systems’ impact on the procurement procedures results and on possible
savings due to the use of certain procurement procedures. To comprehensively
assess the procurement efficiency in e-bidding, the authors propose considering
four key risks: the risk of cancellation of the procurement procedure, the risk
that the procurement procedure will not take place, the risk of appealing the
procurement, the risk of disqualification. As a result of risks calculations
under the sub-threshold and above-threshold procurement, individual values of
risks and their aggregate indicator are determined. This will adjust the scope
of audit procedures to verify individual procurements and identify weaknesses
in the procurement management system. We believe that the methodology of auditing the
procurement effectiveness, taking into account the quantitative and qualitative
parameters of procurement risks, will be a useful audit tool to determine the
effectiveness of the use of public funds under individual procurements and
identify areas of cost-effectiveness
for the state budget funds.
Keywords: Public Procurement; Risks; Audit; Audit Assessment;
Efficiency; Finance; Enterprises
1.
INTRODUCTION
National economies have recently
been negatively affected by pandemics and quarantine measures. The negative
dynamics of the revenue side of the budget and the forced reduction of
expenditures necessitate a more careful approach to spending on procurement for
the needs of the state. Taxpayers who face the threat of increasing taxes and
fees and/or cutting services hope that the state budget expenditures on the
purchase of goods, works and services will be accompanied by the effective
management of taxpayers’ funds. This is not always the case. Recent advances in
the public procurement theory suggest that complex auctioning schemes create
opportunities for collusion and corruption (Lambert-Mogiliansky & Sonin,
2006).
Audits aimed at assessing the value
for money (VfM), which emerged in the late 70’s of the twentieth century in
response to the economic crisis and the budget deficit of the world (Pollitt et
al., 1999) began determining the efficiency of use in addition to legality.
Examining the types and forms of auditing (external, internal, public and
private (Bowerman, Raby & Humphrey, 2000), considered that
auditing is only one aspect of a broader but a rapidly evolving “performance
measurement society” that includes other important elements, which include
increased inspection and self-assessment. (Mayne &
Ontario, 2006) divide the purpose of audit and evaluation. Examining the
essence of performance audits, these authors name the result of the
methodologies used as one of the difficulties in fulfilling their roles to the
public sector.
The main task of the performance
audit, as one of the types of public audit is a constant and comprehensive
audit of three “E” - economy, efficiency and effectiveness (McCrae & Vada, 1997) of public procurement is carried out by all higher institutions of
public audit.
Increasing the analytical potential
of the audit in the current situation is an important direction in the
development of its methodology (Slobodyanik, Kondriuk & Haibura, 2019).
Mega-trends that will change the way
we do business in the next decade will not only require a rethinking of the
vision of procurement, make the necessary changes in their strategy, but also
create the conditions for changing methods of assessing efficiency and
effectiveness. Among the most important modernizing influences are the use of
big data and the global network, the new role of procurement in decision-making
based on this data (Spiller et al., 2013) and volatility as a new norm:
transfer of supply risk to competitive advantage (Spiller et al., 2013).
Adi and Dutil (2018) note the presence of asymmetry in the selection of objects of
performance evaluations, when many ministries are underestimated and others are
systematically overestimated. Therefore, increasing the productivity of the
audit organizations themselves is also in the application of more acceptable
criteria for the selection of the object and the implementation of assessments
according to modern adapted methods.
The audit results will help raising
public awareness about the results of assessing the effectiveness of the public
funds use in organized bidding.
2.
LITERATURE REVIEW
In the modern economic literature,
research on the category of “efficiency” and diverse methods and approaches to
its evaluation are widely presented. Research on performance analysis can be
divided into two main categories: (a) research related to the development of
regulatory decision-making procedures; and (b) those that discuss the
application of regulations to empirical data (Kroll & Levy, 1980). The main problems of analytical audit
evaluation of the results of procurement procedures in real life have received
limited attention in the theoretical literature.
Our contribution offers an
integrative nature of the audit evaluation methodology of the procurement
procedures effectiveness, which links the evaluation indicators of the three
“E” with the procurement risks. The effectiveness
of public procurement depends on a combination of factors: regulatory framework
for procurement, macroeconomic and political processes, the degree of
saturation of commodity markets, professionalism of customers, the adequacy of
the formulation of conditions and features of structural diversity of public
procurement.
The study of literature sources
showed a variety of approaches to assessing the effectiveness of procurement,
as well as the lack of consensus on the methodology of its implementation. This
diversity of approaches is explained by the fact that in Ukraine there is no
single methodology for assessment, including those defined by regulations.
Thus, some scientists in assessing the effectiveness of public procurement take
into account the calculation of the absolute and relative effect of public
procurement by comparing prices in a single competition (Methodological, 2008).
The method of Lapin, Kiseleva and Kumundzhieva (2016) is an example of such a technique,
which is designed to evaluate the activities of customers in the field of
public procurement. The researchers suggests calculating
two indicators: cost-effectiveness and the validity of the initial contract
price. Cost-effectiveness is calculated as the difference between the initial
(maximum) contract price and the price at which the contract is concluded. The second indicator - the assessment of the
validity of the initial (maximum) contract price is determined by the deviation
of the initial (maximum) contract price, which is stated by the customer, from
the average contract price offered by the bidders. The researchers proposes, in addition to
savings, i.e. the difference between the initial (maximum) contract price and
the price at which the contract is concluded, to evaluate the validity of the
initial (maximum) contract price as a deviation of the initial (maximum)
contract price from the average contract price, offered by the participants of
the order.
In our opinion, this approach has a
number of disadvantages. Thus, comparing the contract price with the average
price of suppliers’ bids, it is possible to determine the “effectiveness of the
tender within its participants”, but with a small number of such participants,
and even more so in case of conspiracy, the resulting performance evaluation
will be ineffective.
Ivanova (2010) proposes estimating
the savings in public procurement as the difference between the sum of the
average bid prices of suppliers and the sum of the bid prices for which the
contract is concluded and to determine such an indicator for each sector of
public procurement separately.
In addition, the scientist’s method
involves assessing the feasibility of determining the initial price of
contracts as a deviation from the average savings in the industry. We believe
that the comparison of the final value of the contract and the market price for
similar goods is a more acceptable indicator, in our opinion. However, in this
case, determining such a market price can be quite problematic. We will explain
why.
The market price is calculated on
the basis of information about the concluded identical agreements with
homogeneous goods, works or services. Thus, in most cases, customers use the
information posted on the websites of potential suppliers, and accordingly on
the basis of such data and determine the average market price. In the case of
impossibility to determine the market value of the order, and this is usually a
common practice in the procurement of works or services, then customers use the
cost method of determining the price.
In practice, there is another option
for determining the comparative efficiency of the order, namely on the basis of
contract prices for previous tenders. The
task of the customer is to develop and implement effective processes for
evaluating suppliers and determining the criteria for the winner (Sollish &
Semanik, 2012), which determines the prerequisites for
effective procurement. However, the disadvantage of this approach is that it
does not take into account the transaction costs of customers and does not
reflect changes in prices for such goods. In addition, it is clear that some
groups of goods may become cheaper over time, while others may become more
expensive.
In our opinion, the main
disadvantage of methods for assessing the public procurement effectiveness is
that they are based solely on price indicators. Therefore, the qualitative
characteristics of goods, works and services purchased on a competitive basis
are ignored, thus losing the meaning of such a competition. Moreover, the
evaluation of the public procurement effectiveness is not taken into account at
all. The latter comes down only to determining a monetary efficiency.
The authors evaluate cost models for materials,
consumables, and equipment in order to examine the potential cost savings from
specific procurement procedures. The results of the cost model show that the
use of competitive tenders, procurement calendars, central warehouses and lists
of tenderers is associated with significant cost savings (Duncombe & Searcy, 2007).
We should agree with the thought of
certain scientists on the need to take into account the assessment of the
public procurement quality, in other words, the level of the consumer’s
satisfaction with the services, works, or the delivery of goods, if “efficiency
depends on the value and productivity perceived by the consumer” (Karlöf, 1996).
Such interpretation can be
considered to be true, taking into account that quality is an indicator of the
sufficiency of the product and service’s power, which is responsible for the
assessment of the consumer’s satisfaction quality, in accordance with the goal
of that particular commodity. In this case, quality assessment is primarily a
diagnosis of the properties of usefulness and reliability, outlined by the
customer for the delivered goods or work and services performed. However, in
our opinion, this approach has its drawbacks, as there are some difficulties
with such an assessment.
So, according to the theory of
benefits, it seems that all benefits are classified into the inspectorate,
experimental and fiducial. A more detailed qualification allows the inspection
goods to be featuring such characteristics that may be defined only after the
delivery.
Specific features of experimental
benefits provide additional characteristics, as it is possible to reconfigure
it for less than an hour to get specific goods. In this case, fiducial goods
are a type of goods, the evaluation of which is characterized by a subjective
nature.
Therefore, given the above, it
should be noted that the time gap can serve as an important point in the
process of qualitative evaluation of efficiency.
A group of scientists - proponents
of a slightly different position, propose using the criteria of economy,
productivity of resources used, cost-effectiveness, taking into account the
time factor, with a thorough analysis of the type, conditions and completeness
of contracts (Nesterovich, 2008), evaluate the implementation of planned
indicators (Karlöf, 1996), which allows determining the
degree of goals’ achievement.
3.
DATA AND METHODOLOGY
The main input data parameters that
were used to test the hypothesis of the possibility and feasibility of using
risks in assessing the effectiveness of public procurement for audit purposes,
are formed in the open e-procurement system Prozorro (2020).
The time range of the study consists
of the interval 2018-2019 in terms of procurement data for each month. With regard to the choice of research methods,
the key task is the development of such arrays of indicators that allow
calculations of risks using both quantitative and qualitative parameters.
Quantitative risk parameters were
calculated using calculations of probabilistic indicators of procurement risk
assessment, such as the ratio of the number of relevant procedures to the total
number of procurement procedures (Table 1). Statistical methods were used for
cost assessments and calculation of the aggregate indicator of public
procurement risk (Table 1).
Table 1: The main indicators that
characterize the effectiveness of public procurement
|
No
|
Name of the
indicator
|
Method of calculating the indicator
|
|
Probabilistic indicators of public procurement risk
assessment
|
|
1.
|
Probability of cancellation of the procurement procedure
|
Number of canceled procedures / Total number of procedures
|
|
2.
|
Probability that the procurement procedure will not take place
|
Number of procedures that did not take place / Total number of
procedures
|
|
3.
|
Probability of appealing the procurement procedure
|
Number of complaints satisfied / Total number of procedures
|
|
4.
|
Probability of disqualification of the participant
|
Number of price offers of the disqualified participants / Total number
of price offers
|
|
Cost indicators of public
procurement risk assessment
|
|
5.
|
Cost at risk of cancellation of the procurement procedure
|
Quantile of the function of distribution of
the expected value of purchases that have been canceled, with the selected
level of confidence of 95%:
, 
where: k is the absolute value of the quantile of the distribution of
a random discrete quantity, including  - the relative value;  - i-th value in
ascending order of the random variable;  - estimation of the relative location of the
i-th value of a random variable in a set of its values; n - is the number of
values of the random variable under consideration.
|
|
6.
|
Cost at risk that the procurement procedure will not take place
|
Quantile of the function of distribution of the expected value of
purchases that did not take place
|
|
7.
|
Cost at risk of appealing the procurement procedure
|
Quantile function of the distribution of the expected value of the
procured purchases
|
|
8.
|
Cost at risk of disqualification of the participant
|
Quantile of function of distribution of the price offers sum for the
purchases which have been disqualified
|
|
9.
|
Aggregate value at risk
|
The amount of value at risk analyzed
|
|
10.
|
Cumulative public procurement risk indicator
|
Cumulative value at risk / Expected value of procurement
|
Source:
Pysmenna (2017)
We used the unique content for
analytical research in terms of sub-threshold and above-threshold procurement,
which is largely due to the availability of the necessary information (Prozorro, 2020). At the same time, we did not differentiate
according to the sectoral distribution of procurement customers.
All empirical data on individual
public procurement, reflected on ProZorro, were used.
We agree that at the organizational stage of their implementation,
public and private sector customers have differences (Beuve, Moszoro & Saussier, 2018). However,
audit evaluation according to our methodology can be used to determine the
effectiveness of public or private procurement. The reasoning in favor of this conclusion is the common rules for
conducting such procurement in the electronic system, on the basis of which we
conducted analytical research.
We have disregarded the detection by
analytical methods of conspiracy and/or corruption of procurement during
performance evaluation, although we agree with Lambert-Mogiliansky and Sonin
(2006) on the significance of such effects on procurement results. However,
methods of detecting fraud and its implications for procurement efficiency, the
risks of collusion and corruption have not been the subject of this article.
The purpose of the
study was to
propose a method of auditing the effectiveness of procurement procedures, which
links the indicators of efficiency, productivity, economy with the risks of
public procurement. To this end, the existing methods of assessing the use of
funds for public procurement and indicators that determine the factors
influencing the category of efficiency are analyzed. It was determined that a
comprehensive performance assessment is possible provided that a procurement
risk assessment is used.
4.
RESULTS
Scientific analysis proves that in a
broad sense, efficiency is an indicator that characterizes the relationship
between the result of the process and the cost of its implementation. In
addition, given the variety of targets and outlined results, it is possible to
qualify economic, social, production and other types of efficiency.
In view of the above, the efficiency
of the public procurement system should be understood as a complex concept
consisting of the following components:
first,
the target efficiency as the degree of achievement of the system results;
second,
economic efficiency as the ratio of the economic effect and the cost of
resources needed to achieve such an effect;
third, organizational efficiency,
which characterizes the infrastructural and competitive environment of order
placement, the level of development, implementation and use of regulatory,
methodological, informational, analytical support of the public procurement
system;
fourth, the efficiency of budget
expenditures for public procurement (Figure 1).
Emphasizing the application of the
method of comparing the costs and benefits of public procurement, it should be
noted that expert assessment of all costs play an important role. The latter
can be divided into direct, such as the contract price and insurance costs, and
indirect, arising from the occurrence of any adverse events.
Figure 1:
Decomposition of the definition “efficiency of the public procurement system”
Source:
developed by the authors
If
the benefits to the public or, in other words, the benefits of meeting needs
are higher than the cost
of purchasing and placing an order, then such a project is considered
potentially effective (Arrowsmith, Linarelli & Don Wallace, 2000). In this case, the cumulative effect of public procurement can be expressed
through various effects (Table 2).
Table 2: Features
of the set of effects
|
No
|
Types of effects
|
The essence of the concept
|
|
1.
|
Direct savings
|
are lower prices compared to the planned amount of funding
|
|
2.
|
Indirect or implicit savings
|
are characterized by the purchase of goods, works and services of
higher quality and on more favorable terms than usual. For example, no
advance payment, reduction of delivery time, longer warranty period,
availability of additional services, etc.
|
|
3.
|
The side effects
|
are manifested, for example, in reducing the level of corruption,
increasing the degree of openness of public procurement procedures,
increasing the business reputation of the customer and the investment
attractiveness of the region
|
Source: compiled by the
authors using Lapin, Kiseleva and Kumundzhieva (2016)
The costs of achieving the economic
effect of public procurement are usually expressed in the amount of:
·
labor
costs for the implementation of organizational measures for procurement;
·
material
costs of public procurement, such as the cost of consulting services,
consumables, postal and courier services, the cost of equipping workplaces and
renting additional premises, to ensure the functioning of the infrastructure of
the public procurement system (including official websites and publications),
training staff.
Waters (2015) considers it necessary
to take into account the degree of customer satisfaction when assessing the
effectiveness of procurement, i.e. compliance with consumer demands, ensuring
the best conditions for public procurement, reliability and qualification of
the supplier. In this case, we often use the method of comparison of suppliers,
which analyzes the proposals of all suppliers in order to identify those, who
are able to qualitatively fulfill the government order with the lowest contract
price.
However, in our opinion, there is an
important problem of information asymmetry, because only suppliers know their
true so-called “cost curves”. It is clear that under such conditions, customers
can analyze the history of relationships with suppliers, which is the most
effective tool for obtaining information on their costs for the amount of losses
due to the supply of substandard products, the number of erroneous deliveries,
reliability of delivery, etc.
It is possible to eliminate these
shortcomings in the case of using another assessment method of procurement efficiency - the method of
analysis of the main provisions of the contract. This method involves analyzing
the presence and essence of certain provisions that govern the relationship
between supplier and customer. In this case, the evaluation is based on certain
criteria that must be taken into account in the contract, namely: the presence
of possible risks during procurement, achieving a certain result given the
identified planned costs, delineation of rights and obligations of the parties,
establishing a monitoring procedure for orders, features of the dispute
resolution process, opportunities for communication between the parties, etc.
Thus, the contract is evaluated in terms of its completeness: the more detailed
are all the necessary provisions, the more efficient is the procurement.
Of course, the analysis of the
contract implementation can be carried out both during the whole cycle of
public procurement and after its completion. The analyzed method uses a number
of qualitative indicators, such as staff qualifications, the process of interaction
and communication, the degree of satisfaction of all parties involved in the
public procurement process, compliance with the deadlines for placing orders
and work.
Such an assessment is usually
conducted by questioning all participants, from senior management of the
authorized body for the coordination of procurement of goods, works and
services at public expense, suppliers, and ending with employed officials who directly
place orders or submit proposals for public procurement (Perov, Dashkov &
Abdrakhimov, 2006). The methodology for assessment the
public procurement systems MAPS (2018) of the World Bank and the Organization
for Economic Cooperation and Development (OECD) is scientifically meaningful.
The MAPS methodology was developed
in 2003-2004 and is constantly being improved, taking into account the
Recommendations of the OECD Public Procurement Council (RPP, 2015) and reflecting the leading international framework for such
procurement, in particular the Model Law of the United Nations Commission on
International Trade (Uncitral, 2011), EU Public Procurement Directives (Public Procurement in the EU, 2016).
The MAPS analytical structure
consists of a basic evaluation methodology and various additional modules,
focuses on specific aspects of public procurement policy and can be used by
countries depending on their needs. Thus, the analysis of the country’s
conditions Moncrieffe, Luttrell (2005) should be based on a limited number of
factors potentially important for procurement reforms, namely:
1) political, economic and geostrategic
situation in the country;
2) links between the procurement system
and public administration and public finance management systems,
3) national policy objectives that
affect the strategy, quantity and quality of public procurement (White,
Parfitt, Lee & Mason-Jones, 2016);
4) the environment for the
implementation of reforms in the field of public procurement.
The originality of the MAPS methodology
is that the system of indicators is based on four panels:
a) the existing legal and political
structures that regulate the procurement process in the country;
b) institutional framework and
management efficiency;
c) the system functioning and the
competitiveness of the internal market;
d) accountability, integrity and
transparency of the procurement system.
Each panel, in turn, includes
several indicators and sub-indicators to be evaluated. In total, such a system
has 14 indicators and 55 sub-indicators, which correspond to the relevant
criteria for “instant” comparison of the current system with these principles.
The studied indicators are expressed in qualitative and/or quantitative terms.
What are the advantages of the
method of evaluating the effectiveness of public procurement by MAPS? The
analysis carried out in the process of scientific research made it possible to
identify the following advantages of this technique:
·
first,
the formulation of the concept of “sustainable public procurement” in line with
the integration of the three components of sustainable development, i.e.
economic, social development and environmental protection (Spiller et al., 2013);
·
secondly,
public procurement objectives are usually aimed at reducing the demand for
resources, minimizing the negative impact of goods, works or services
throughout their life cycle, ensuring fair contract conditions, including
ethical rights, human rights and employment standards;
third,
the possibility of using public procurement and other innovative methods in
evaluation.
In practice, the audit evaluation of
public procurement uses evaluation criteria that do not take into account
public requests for information about their effectiveness (Figure 2). Under the
procurement audit by the Accounting Chamber (Accounting Chamber of Ukraine,
2019), the conclusions on the criteria of transparency, timeliness and
completeness of management decisions were not evaluated in terms of their
impact on the effectiveness of procurement procedures.
In
our opinion, the assessment of the public procurement effectiveness mainly in
terms of budget savings (in absolute terms and as an average percentage
reduction in the price of goods, works, services purchased) is insufficient for
audit purposes. This conclusion is made by the auditors of the Accounting
Chamber of Ukraine, assessing the report of the Ministry of Economy, Trade and
Agriculture of Ukraine (Accounting Chamber, 2020), in which
the Ministry informs about “savings” of 1.02 billion UAH in public procurement
in 2019, but does not indicate the causes of this process. The auditors of the
Accounting Chamber do not confirm the amount of public savings announced by the
Ministry of Economy, as there is no analysis and reliable indicators of
efficient and transparent public procurement, the report does not analyze the
factors, including those affecting the expected value of tenders.
In our opinion, the use of public
savings as the main criterion for the effectiveness of the public procurement
system is insufficient for an objective and comprehensive assessment, as the
calculation of budget savings does not take into account a reasonable
determination of the initial (maximum) contract price and pricing, resulting in
the problems in calculating the savings rate.
Figure 2: Criteria for assessing the
effectiveness of the use of public funds allocated for the purchase of goods,
works and services
Source: compiled by the
authors
It should be added that the analyzed
approach to assessing the effectiveness of public procurement by the only
criterion of saving budget resources, in our opinion, is not entirely correct.
This conclusion is substantiated by:
·
first,
insufficient development of methodological support in terms of calculating the
initial contract price;
·
secondly,
the leveling of attention to the specifics of non-price factors that affect
both the choice of supplier and terms of supply, and the level of qualification
and reliability of public procurement executors;
·
third,
not taking into account the costs of ensuring the functioning of the public
procurement system.
One cannot but agree that the
calculation of savings depends on the initial contract price.
At the same time, in case of
substantiation of the latter, it is necessary to have special knowledge of
market research in order to conduct an in-depth market research by public
customers. Therefore, in most cases, the initial (maximum) price of contracts
is fixed at the level of financial resources allocated for the purchase of a
particular type of product, which, in turn, does not reflect the real market
situation. But, under such conditions, when achieving the effect of saving
budget resources, there is usually a problem of incomplete use of allocated
budget funds for certain items. For the customer, this means a reduction in
funding for the costs in subsequent periods.
In addition, it should be noted that
significant savings can be achieved either as a result of unreasonable
overestimation of the initial contract price, or as a result of non-fulfillment
of public procurement plans and the presence of tenders, which did not lead to
the conclusion of such contracts.
Thus, the traditional assessment of
procurement efficiency based on the definition of budget savings does not fully
reflect the degree of procurement efficiency and can be used only for
operational analysis. Achieving savings of budget resources in public
procurement is, although the main, but still only a part of the holistic
process of efficient spending of budget funds, so it can be used only as an
additional criterion of efficiency. Such an assessment levels out a set of
important indicators, in particular - the quality of the customer’s planning
work, discipline in the execution of contracts, compliance with the principle
of competitive procurement, the customer’s compliance with the requirements of
public procurement legislation, etc.
Thus, the efficiency of public
procurement is replaced by the saving of budgetary resources, while the
efficiency of their spending includes their saving. Hence, it is clear that
such an assessment will determine only one aspect of the effectiveness of
public procurement.
We believe that the lack of
assessment of the risks faced by bidders and procurement customers is a
significant disadvantage of this approach, because determining the level of
such risks makes it possible to form additional characteristics of the
effectiveness of the public procurement system. In our opinion, the key risks
of public procurement that need to be evaluated and monitored are:
·
risk
of cancellation of the procurement procedure;
·
risk
that the procurement procedure will not take place;
·
risk
of procurement appeal;
·
risk
of disqualification of the participant.
Risk assessment, or in other words,
determining the quantitative and qualitative consequences of its
implementation, is possible using various statistical methods.
In our opinion, by using one or
another method of risk assessment, the following information about the risk can
be obtained: first, its probabilistic characteristics; secondly, its
quantitative assessment (Shevchuk, 1998):
|
|
(1)
|
where, - the value of the assessment
of the consequences of the occurrence of a risky event;
- function of parameter (x);
P - the
probability of a risky event;
I - the potential consequences of risk.
Risk assessment methods, which are traditional
and well-tested (Goncharenko & Filin, 2016) to evaluate the risks of public
procurement should be supplemented by the method of cost-effectiveness
assessment (Zaiets, 2017). It involves comparing the benefits and costs of
participation in the competitive public procurement procedures. In the field of public procurement, risk assessments can be obtained using the methods presented in Figure 3.
Figure 3: Methods for assessing the risks of
public procurement
Source: authors’ development
The results of the assessment of the
probabilities of risks realization are summarized and illustrated in Figure 4. Based on the performed calculations, it can
be concluded that the risk that the procurement procedure will not take place
is the highest risk of public procurement.
Figure 4: Probabilistic characteristics of
public procurement risks
Source: calculated by the
authors
The level of this risk reached 45%
in January 2018 on both sub-threshold and above-threshold procurement and most
of the time during 2018-2019 remained at a level above 35%. At the end of the
year, all types of risks were reduced.
The risk of disqualification of
public procurement participants is the second risk most likely to be realized.
The level of this risk is higher in
above-threshold procurement (from 15% to 30%) against sub-threshold procurement
(from 10% to 20%). This is due to the fact that the qualification requirements
for participants are much higher in the implementation of above-threshold
procurement. Risk assessments have been calculated using a similar approach.
In order to objectively assess and
characterize the level of risks of public procurement, relative to the total
expected value of public procurement in terms of sub-threshold and
above-threshold values, we calculate VaR-indicators of public procurement (risk
value indicators) equal to the quantile of the value distribution function
procurement at the selected confidence level of 95%.
Since the determination of VaR is
possible for quantities with a normal function of distribution of their values,
in order to check whether the distribution of values of the expected value of
public procurement can be considered normal, we calculated tables of
frequencies of distribution of the expected value of procurement.
The results of calculations of
quantitative assessments of public procurement risks for above-threshold
procurement are summarized in Table 3 and sub-threshold procurement - in Table
4.
Table
3: Statistical estimates of the cost characteristics of public procurement
risks in above-threshold procurement
|
Indicators
|
Risk of cancellation of the
procurement procedure
|
Risk that the procurement
procedure will not take place
|
Risk of appealing the
procurement procedure
|
Risk of disqualification of
the participant
|
|
2018
|
|
Average value of risk assessment
|
522522891,09
|
468289944,77
|
161540793,22
|
30369610571,04
|
|
The
standard deviation of the cost estimate of risk
|
475977186,07
|
494551086,83
|
196395124,37
|
73913648117,30
|
|
Variation of the cost estimate of risk
|
91,09%
|
105,61%
|
121,58%
|
243,38%
|
|
2019
|
|
Average value of risk assessment
|
745970646,78
|
1007978193,72
|
96946432,16
|
1312023241,27
|
|
The
standard deviation of the cost estimate of risk
|
699553623,49
|
609072408,93
|
63621341,81
|
724218388,76
|
|
Variation of the cost estimate of risk
|
93,78%
|
60,43%
|
65,63%
|
55,20%
|
|
2018-2019
|
|
Average value of risk assessment
|
475543722,49
|
509924334,26
|
137714986,33
|
13018768698,69
|
|
The
standard deviation of the cost estimate of risk
|
587119348,44
|
617085748,07
|
159853292,97
|
48649237510,91
|
|
Cost
at risk, VAR
|
1441269112
|
1524940065
|
400650255
|
93039643467
|
|
The total value at risk
|
96406502899
|
|
Cumulative risk of public procurement
|
47,85%
|
Source: authors’ calculations
Table 4: Quantitative risk
assessments of public procurement in sub-threshold procurement
|
Indicators
|
Risk of cancellation of the
procurement procedure
|
Risk that the procurement
procedure will not take place
|
Risk of appealing the
procurement procedure
|
Risk of disqualification of
the participant
|
|
2018
|
|
Average value of risk assessment
|
4526153302,26
|
4738741838,41
|
4339546073,76
|
3830891947,16
|
|
The
standard deviation of the cost estimate of risk
|
6656071484,08
|
4625735620,49
|
8729026880,50
|
3259557149,33
|
|
Variation of the cost estimate of risk
|
147,06%
|
97,62%
|
201,15%
|
85,09%
|
|
2019
|
|
Average value of risk assessment
|
5733219792,15
|
8573202986,72
|
6081006510,89
|
18654570799,30
|
|
The
standard deviation of the cost estimate of risk
|
4301519882,77
|
2441631225,58
|
3469600566,59
|
29250724848,36
|
|
Variation of the cost estimate of risk
|
75,03%
|
28,48%
|
57,06%
|
156,80%
|
|
2018-2019
|
|
Average value of risk assessment
|
5190039871,70
|
6847695469,98
|
5297349314,18
|
11983915315,84
|
|
The
standard deviation of the cost estimate of risk
|
5363959804,53
|
3997205454,98
|
6261664278,55
|
22628308940,75
|
|
Cost
at risk, VAR
|
14012968611,00
|
13422513360,28
|
15596870513,50
|
49204171348,81
|
|
The total value at risk
|
92236523833,60
|
|
Cumulative risk of public procurement
|
12,05%
|
Source: authors’ calculations
As these data show, the total risk
of above-threshold public procurement in 2018-2019 is 47.85%, while the total
risk of the sub-threshold public procurement in 2018-2019 is equal to 12.05%.
In order to assess the value of the
aggregate risk indicator of public procurement, we suggest using the classical
approach, which involves the allocation of four risk zones depending on the
indicator values (Table 5).
Table
5: Gradation of risk zones depending on the values of the aggregate risk
indicator
|
The value of the aggregate
risk indicator
|
Risk Zone
|
|
0 – 0,1
|
Minor risk
|
|
0,1 – 0,3
|
Permissible risk
|
|
0,3 – 0,6
|
Increased risk
|
|
> 0,6
|
Unacceptable risk
|
Source: Stupakov and Tokarenko (2005)
In our opinion, such a scale is
acceptable for assessing the public procurement effectiveness. Therefore,
according to these criteria, the aggregate risk of above-threshold public
procurement is increased, while the aggregate risk of sub-threshold public
procurement is insignificant.
Thus,
the aggregate risk in above-threshold procurement is increased due to the high
probability that, firstly, the procurement procedure will not take place and,
secondly, the disqualification of participants.
For
audit purposes, this necessitates the adjustment of the scope of audit
procedures, which depend on the assessed risk of the audited entity.
The estimated risk of individual
procurement procedures according to the above method can be used when
identifying the effectiveness of the organization of procurement by individual
regions (periods).
Thus, the quantitative and cost
analysis of risk assessment in this case is carried out on the basis of
selected analytical data of the Prozorro system of territories, or one or a
group of customers.
5.
CONCLUSIONS AND RECOMMENDATIONS
Estimation of efficiency Performance management
will vary significantly from one context or situation to another, so adapting
general approaches to determining the effectiveness of the use of public funds
requires methodologies that would answer the questions of specific situations
“What are the generative mechanisms of this particular design of PMS?” and “How
do they interact with the particular context?” (Pollitt, 2013).
The research methodology is based on
the application of statistical and analytical methods on the data on the
conducted procurement procedures during 2018-2019.
In Ukraine, the operation of an open
electronic public procurement system has created a unique opportunity for
accessible analytics in the context of all announced procurement procedures.
The calculations were performed in
terms of sub-threshold and above-threshold procurement. The total amount of
processed data for 2018 is 1.084 million procurement procedures, and for 2019 -
1.238 million procurement procedures, which is 100% of the declared procurement
procedures for this period. Based on the results of the study, it is proposed
to supplement the existing methods of assessing procurement risk indicators to
conduct an audit assessment of procurement effectiveness.
Procurement efficiency was assessed
via four types of risk: the risk of cancellation of the procurement procedure,
the risk that the procurement procedure will not take place, the risk of
appealing the procurement, the risk of disqualification of the participant
separately for sub-threshold and above-threshold procurement. The riskiest
periods of procurement were established and the aggregate risk indicator of
public procurement was calculated at the level of 47.85%. This enabled
assessing the impact of certain types of risk on the procurement effectiveness
and adjusting the scope of audit procedures to verify individual procurement.
We see the further direction of the
research in determining the risks of procurement by types of selected
procedures, which will generate analytical data for risk management in the
supply chain management.
REFERENCES
Accounting Chamber (2019). Report on the analysis of the functioning of the public procurement
system. Available:
https://rp.gov.ua/upload-files/Activity/Collegium/2019/12-2_2019/Zvit_12-2_2019.pdf.
Access: 1th April, 2020.
Accounting
Chamber (2020). Information on
«Savings» of UAH 28.2 billion in public procurement in 2019 is contradictory. Available: https://rp.gov.ua/PressCenter/News/?id=908.
Access: 6th April, 2020.
Adi, S., & Dutil, P. (2018). Searching for
strategy: Value for Money (VFM) audit choice in the new public management era. Canadian Public Administration, 10.1111/capa.12254, 61(1),
(91-108). DOI: https://doi.org/10.1111/capa.12254.
Antoniuk, O., Chyzhevska, L., & Semenyshena, N. (2019). Legal
regulation and trends of audit services: what are the differences (evidence of
Ukraine). Independent Journal of
Management & Production, 10(7),
673-686. DOI: http://dx.doi.org/10.14807/ijmp.v10i7.903
Antoniuk, O., Kuzyk,
N., Zhurakovska, I., Sydorenko, R., & Sakhno, L. (2020).
The Role of «Big Four» Auditing Firms in the Public Procurement Market in
Ukraine. Independent Journal of Management &
Production, 11(9), 2483-2495.
DOI: http://dx.doi.org/10.14807/ijmp.v11i9.1432.
Arrowsmith, S., Linarelli, J., & Don
Wallace, JR. (2000). Regulating Public
Procurement. National and International Perspectives. Kluwer Law International.
Beuve, J.,
Moszoro, M. W., & Saussier, S. (2018). Political contestability and public
contract rigidity: An analysis of procurement contracts. Journal of Economics & Management Strategy.
doi:10.1111/jems.12268.
Bowerman, M., Raby, H., & Humphrey, C.
(2000). In Search of the Audit Society: Some Evidence from Health Care, Police
and Schools. International Journal of
Auditing, 4(1), 71–100.
DOI:10.1111/1099-1123.00304.
Duncombe, W.,
& Searcy, C. (2007). Can the Use of Recommended Procurement Practices Save
Money? Public Budgeting & Finance, 27(2), 68–87. DOI:10.1111/j.1540-5850.2007.00875.x.
Fesenko, V., Vakulchyk,
O., Guba, O., Ostapchuk, S., & Babich, I. (2020). The Results
of Implementation of European Requirements in Management of Transfer Pricing
Audit (Experience of Ukraine). Independent Journal of Management &
Production, 11(9), 2417-2434. DOI:
http://dx.doi.org/10.14807/ijmp.v11i9.1412.
Goncharenko, L. P., & Filin, S. A. (2016). Risk management. Moscow: KNORUS.
Ivanova, O. V.
(2010). Methodology for a comprehensive assessment of the
effectiveness of public procurement of the Oryol region. Bulletin of TulSU. Economic and legal sciences, 2, 183-192.
Karlöf,
B. (1996). New age efficiency and demands on organizations. Strategic Change, 5(1), 43–48.
DOI:10.1002/(sici)1099-1697(199601)5:1<43::aid-jsc211>3.0.co;2-a.
Khorunzhak, N., Belova, I., Zavytii, O.,
Tomchuk, V., & Fabiianska, V. (2020). Quality Control of
Auditing: Ukrainian Prospects. Independent Journal of Management & Production, 11(8), 712-726. DOI: dx.doi.org/10.14807/ijmp.v11i8.1229.
Khorunzhak, N., Brukhanskyi, R., &
Ivanyshyn, V. (2019). Logic-statistical information models in control function of accounting. Independent Journal of Management & Production, 10(7), 846-871. DOI: http://dx.doi.org/10.14807/ijmp.v10i7.906.
Kroll, Y.,
& Levy, H. (1980). Sampling Errors and Portfolio Efficient Analysis. Journal of Financial and Quantitative
Analysis, 15(3), 655-688. DOI:10.2307/2330403.
Lambert-Mogiliansky, A., & Sonin, K. (2006).
Collusive Market Sharing and Corruption in Procurement. Journal of Economics Management Strategy, 15(4),
883–908. doi:10.1111/j.1530-9134.2006.00121.x.
Lapin, A. E.,
Kiseleva, O.V., & Kumundzhieva, E. L. (2016). Approaches to assessing the
effectiveness of the contract system in the field of state and municipal
procurement. Business. Education. Right.
Bulletin of the Volgograd Institute of Business, 11(34), 30-35.
MAPS (2018). Methodology for Assessing
Procurement Systems. Available: http://www.mapsinitiative.org/methodology/MAPS-methodology-for-assessing-procurement-systems.pdf. Access: 18th June, 2020.
Mayne, J., & Ontario, O. (2006).
Audit and evaluation in public management: challenges, reforms, and different
roles. The Canadian Journal of Program
Evaluation, 21(1),
11–45.
McCrae, M., &
Vada, H. (1997). Performance Audit Scope and the Independence
of the Australian Commonwealth Auditor‐General. Financial Accountability & Management, 13(3). DOI: https://doi.org/10.1111/1468-0408.00034.
Melnyk, N.,
Trachova, D., Kolesnikova, O., Demchuk, O., & Golub, N. (2020).
Accounting Trends in the Modern World. Independent
Journal of Management & Production,
11(9), 2403-2416. DOI:
http://dx.doi.org/10.14807/ijmp.v11i9.1430.
Methodological
(2008). Methodological recommendations
for assessing the effectiveness and transparency of placing state and municipal
orders. Moscow: Delovoy dvor.
Moncrieffe, J.,
& Luttrell, C. (2005). Аn analytical framework for understanding the political economy of
sectors and policy arenas. Overseas Development
Institute. Availiable:
http://www.odi.org/sites/odi.org.uk/files/
odi-assets/publications-opinion-files/3898.pdf. Access:
1th MarchMay, 2020.
Nesterovich, N. V. (2008). N. 94-FZ: how to find savings. State order: Management, Placement, Provision. 12 (April, June), 32–35.
Pasko, O., Melnychuk, O., & Bilyk T. (2019).
Ownership concentration, investor protection and economic performance in public
agroindustrial companies with the listing on Warsaw stock exchange. Independent Journal of Management &
Production, 10(7), 817-845. DOI: http://dx.doi.org/10.14807/ijmp.v10i7.914.
Perov, K. A.,
Dashkov, S. B., & Abdrakhimov, D. A. (2006). Development of a methodology for calculating the
economic efficiency of placing orders for the supply of goods, performance of
work, provision of services for state and municipal needs: research report. Moskow: LTD Institute of Competitive Technologies.
Pollitt, C.
(2013). The logics of performance management. Evaluation, 19(4),
346–363. DOI: https://doi.org/10.1177/1356389013505040.
Pollitt, C., Girre, X., Lonsdale, J., Mul, R.,
Summa, H., & Waerness, M. (1999). Performance or Compliance? Performance Audit and Public Management
in Five Countries. Oxford: Oxford University Press.
Prozorro
(2020). ProZorro is a
hybrid electronic open source government e-procurement system. Available: https://prozorro.gov.ua/ Access: 5th
June, 2020.
PUBLIC
PROCUREMENT IN THE EU (2016). Public Procurement in the EU: Legislative Framework, Basic
Principles and Institutions. Available: http://www.sigmaweb.org/publications/Public-Procurement-Policy-Brief-1-200117.pdf .
Access: 1th November, 2018.
Pyismenna, M. (2017). Written by MS
State audit and analysis of public procurement: theory, methodology and
practice. Kyiv: Center for Educational Literature.
Rodrigues, P. C. C., & Semenyshena, N.
(2019). Editorial Introduction. Independent
Journal of Management & Production, 10(7), 911-914. DOI: http://dx.doi.org/10.14807/ijmp.v10i7.775.
Rodrigues, P. C. C., & Semenyshena, N.
(2020). Special Edition (Integration System of Education, Science and
Production) Introduction. Independent
Journal of Management & Production, 11(8), 801-806. DOI:
http://dx.doi.org/10.14807/ijmp.v10i7.775
Rodrigues, P. C. C., Simanaviciene, Z., & Semenyshena, N. (2020). Editorial Volume
11, Issue 9. Independent
Journal of Management & Production, 11(9), 2542-2547. DOI:
http://dx.doi.org/10.14807/ ijmp.v11i9.1424
RPP (2015). Recommendation on Public Procurement: Public Procurement is the
cornerstone of strategic governance. Available: http://www.oecd.org/gov/public-procurement/recommendation/OECD-Recommendation-on-Public-Procurement.pdf. Access: 24th April, 2020.
Semenyshena, N., Khorunzhak, N., & Zadorozhnyi, Z.-M.
(2020). The
Institutionalization of accounting: the impact of national standards on the
development of economies. Independent
Journal of Management & Production, 11(8), 695-711. DOI:
https://doi.org/10.14807/ijmp.v11i8.1228.
Semenyshena, N.,
Sysiuk, S., Shevchuk, K., Petruk, I., & Benko, I. (2020). Institutionalism
in Accounting: a Requirement of the Times or a Mechanism of Social Pressure? Independent Journal of Management &
Production, 11(9), 2516-2541.
DOI: http://dx.doi.org/10.14807/ijmp.v11i9.1440.
Shevchuk, V. O. (1998). Control of economic systems in a society in transition (problems of
theory, organization, methodology). Kyiv: Kyiv State University of Trade
and Economics.
Slobodyanik, Y., Kondriuk, L., & Haibura,
Y. (2019). The Strategy of Institutional Reform of the Supreme Audit
Institution: the Case of Ukraine. Independent
Journal of Management & Production, 10(7). DOI: http://dx.doi.org/10.14807/ijmp.v10i7.916.
Sollish,
F., & Semanik, J. (2012). The Procurement and Supply Manager's Desk Reference. Chapter 4 Supplier Selection Criteria Book Editor(s): Wiley. DOI: https://doi.org/10.1002/9781119205098.ch4.
Spiller, P., Reinecke, N., Ungerman, D., &
Teixeira, H. (2013). Procurement 20/20: Supply
Entrepreneurship in a Changing World. In: Chapter 7 The New Economic Drivers: Capturing the
Total Impact of Environmental, Social, and Regulatory Factors, ch. 7. DOI: https://doi.org/10.1002/9781119204985.ch7.
Stupakov, V. S., & Tokarenko, G.S. (2005). Risk management. Moscow:
Finance and Statistics.
Uncitral (2011). Model
Law on Public Procurement. Available: https://uncitral.un.org/en/texts/procurement/modellaw/public_procurement. Access: 7th June, 2020.
Vdovenko, N., Piven,
A., Radchenko, O., Sinenok, I., & Voskobiinyk, S. (2020). Institutional Environment for Financial
Provision of Small Agricultural Business Entities of Ukraine. Independent Journal of Management &
Production, 11(9), 2379-2402.
DOI: http://dx.doi.org/10.14807/ijmp.v11i9.1419.
Waters, D.
(2015). Logistics. Supply chain management. Logistics. An
Introduction to Supply Chain Management. Moscow: UNITY-DANA.
White, G. R.
T., Parfitt, S., Lee, C., & Mason-Jones, R. (2016).
Challenges to the Development of Strategic Procurement: A Meta-Analysis of
Organizations in the Public and Private Sectors. Strategic Change, 25(3), 285–298. DOI:10.1002/jsc.2061.
Zaiets, N. M. (2017). A pilot analysis of the
procurement procedures. Bulletin of ZhDTU. Series: Economics of Science, 1(179), 72-80.
AUDIT
ASSESSMENT OF THE EFFECTIVENESS OF PUBLIC PROCUREMENT PROCEDURES