Синергия предсказательной силы моделей кредитных банковских рисков тема диссертации и автореферата по ВАК РФ 08.00.10, кандидат наук Хромова Элла Павловна

• Хромова Элла Павловна
• кандидат наук
• 2021

Оглавление диссертации кандидат наук Хромова Элла Павловна

Contents

Introduction

1. Review and comparative analysis of academic literature on credit risks

1.1 Underestimation of credit risk by PD models and overcautious rating assignments of CR models

1.2 Non-linear interdependence between CR and PD

1.3 Forecast combination models as a tool to improve predictive power

1.4 Comparison of different methodologies of forecasting credit risk

1.4.1 Binary and Multinomial ordered logit/probit models in detail

1.5 Factors of potential influence on credit risk of a bank

1.6 Research hypotheses

2. Difference in ratings assigned to the same issuer by various rating agencies

2.1 Formation of representative samples from the empirical data

2.1.1 Construction of the first empirical data set of international banks pre-2008 crisis period

2.1.2 Construction of the second empirical data set of international banks post-2008 crisis period

2.1.3 Construction of the third empirical data set of national banks post-2008 crisis period

2.2 Adjustment of ratings to the base scale

2.3 Construction and comparison CR models of various RAs

2.3.1 Construction of international RAs model based on the first empirical data set of international banks pret-2008 crisis period

2.3.2 Construction of international RAs model based on the second empirical data set of international

banks post-2008 crisis period

2.3.3 Construction of international and national RAs model based on the third empirical data set of national banks post-2008 crisis period

2.4 Comparison of deviations of RAs rating grades of the same issuer

3. Construction of synergic model of credit ratings and probability of defaults

3.1 Inclusion information about defaults in the sample of Russian banks

3.2 Construction of empirical models of PD and CR separately

3.2.1 Binomial and Ordered logit regressions of CR and PD models

3.2.2 Principal component analysis

3.2.3 Comparison of in-sample fit of the models

3.3 Calibration of rating scale and probability of default

3.4 Comparison of distributions of forecast errors of PD and CR models

3.5 Construction of the synergic model

3.5.1 Robustness check of the synergic model

3.6 Matching results to the existing literature

4. Dynamic calibration of CR and PD of Russian banks

4.1 Creating a dynamic scale of CR conversion to PD

4.2 Quality comparison of PD forecasts of the PD, CR and Synergic models

Conclusion

References

Appendixes

Введение диссертации (часть автореферата) на тему «Синергия предсказательной силы моделей кредитных банковских рисков»

Introduction

Motivation1

The economic growth and stability of any country depend on the financial environment of its banking system. Given the critical role of banks as financial intermediaries, the estimation of their financial stability is one of the main goals of regulators. The most common ways of assessing the financial performance and controlling the level of credit risk of a bank are an evaluation of its probability of default and a credit rating. The probability of default (PD) is the likelihood of a bank failure over a fixed assessment horizon while a credit rating (CR) determines the class to which a company belongs based on the PD. Although both of these methods have been studied a lot, their forecasting power still has a room of improvement. There are possible biases that may lead to misleading results. PD estimates provided by a model forecast can be underestimated because of an imbalanced structure of datasets containing defaults. Defaults are rare, so a PD model becomes overfitted towards non-default events. Even the classical balancing data methods provided by He and Garcia (2009) and Garcia et al. (2012) do not fully solve this problem and a PD model gives underestimated results (Karminsky and Kostrov, 2017). On the other hand, CR models are not fully reliable either. The main reason is a bad proxy of the dependent variable in the model. Researchers obtain information only about ratings assigned by rating agencies (RA) and they have to assume this information to be absolutely true and objective. However, a CR assessment is a subjective opinion of an agency that depends on its conservatism and methodology. It was proven that recently RAs care a lot about their reputation and try to be overcautious in order not to miss the financial distress of a bank (Park and Lee, 2018; Hu et al., 2019). Therefore, the research question of this thesis was formulated as follows: Can the

1 These sections present amended and extended version of the following research paper: Karminsky A., Khromova E. (2018) Increase of banks' credit risks forecasting power by the usage of the set of alternative models. Russian Journal of Economics 4(2): 155-174.

The relevance of the research1

The relevance of the paper is determined by the ability to compare and interpret different credit risk models and to evaluate the financial stability of banks in a valid and consistent manner. This paper aims to solve one of the most important issues of credit risk: a complete assessment of a bank's financial performance is very time consuming, and a large number of credit institutions remain uncovered by RAs. (Duff and Einig, 2009; Bellotti et al., 2011). Therefore, the possibility of a quick and reliable forecast of a bank's financial performance with the help of the proposed synergic credit risk model will be useful for all a bank's counterparties.

• For investors (depositors) - as a tool for finding the banks with minimal credit risks to be included in an investment portfolio (for placement of funds).

• For regulators - as a tool of permanent control with ability to take adequate measures in advance.

• For banks themselves - as an additional internal management tool in IRB approach.

Literature review2

This research unifies two seemingly separate areas of economic literature. The first area of the literature addresses the issue of the underestimation of credit risk by default models and the second area concerns the overcautious assessments of credit ratings.

All recent research emphasizes the importance of paying attention to the presence of the class imbalance problem in data on defaults and its impact on

2 This section presents amended and extended version of the following research papers: 1) Karminsky A., Khromova E. (2018) Increase of banks' credit risks forecasting power by the usage of the set of alternative models. Russian Journal of Economics 4(2): 155-174; 2) Khromova E. (2020) Dynamic Mapping of Probability of Default and Credit Ratings of Russian Banks. Journal of Corporate Finance Research. 14 (4): 31-46

the estimation procedure and on standard forecasting power indicators. (Esarey and Pierce, 2012; Karminsky and Kostrov, 2017; Lanine and Vennet, 2006). Few default events are usually available to estimate the parameters of the model properly in the training set. The main consequence of the class imbalance problem is the underestimation of the "rare" class, which will deteriorate the forecasting power for bank failures (Florez-Lopez and Ramon-Jeronimo, 2014; Rosch and Scheule, 2014). Garcia et al. (2012) discussed the class imbalance problem and methods to overcome it. Among the most used methods are the random omission of non-defaults, the random inclusion of defaults and the increase in weights of the rare class observations in a log-likelihood function. However, even the balancing data methods provided do not fully solve this problem, and the model gives underestimated results.

A second literature stream examines the discrepancies between the ratings of various RAs. Despite the fact that many of them use similar letter designations, their approaches to financial analysis differ. Many authors studied a consistent difference between the scores of the various rating agencies and the financial stability of issuers (Morgan, 2002; Barton, 2006; Becker and Milbourn, 2010; Livingston et al., 2010; Alsaka et al., 2012; Bolton et al., 2012; Shimizu et al., 2013; White, 2013; Feng et al., 2014). Before the 2008 financial crisis, the activity of rating agencies had little regulation, allowing rating agencies to avoid responsibility for overestimation in assigned ratings, which led to a subsequent conflict between the issuers and users of ratings (Solovjova, 2016; Altman et al., 2005; Behr et al., 2008; Dimitrov et al., 2015; Kose et al., 2003; Ryan J., 2012; Xuefeng et al. 2012). As a result of the massive regulatory improvements the Dodd-Frank law that significantly improved implementation of the basic principles of objectivity, transparency and independence of the rating process was adopted. However, in more recent times it was shown that RAs were very cautious in the estimation of financial stability as their reputation fully depended on it (Becker et al., 2010; Flynn et al., 2018; Skreta et al., 2009; Vasiluk, 2011; Givaykina and Peresetsky, 2017; Karminsky, 2015;

Pomasanov and Hamalinsky, 2012; Steshkin, 2015). A reputation of a RA suffers more when an agency predicts a higher rating grade than it should. Therefore, nowadays RAs tend to react sharply to any bad news of a well-performing bank trying to predict the worst scenario of its performance, so their reputation does not suffer.

Observing the divergence of CR and PD modeling, a research gap arises in the absence of usage a synergic effect of combination of these two methodologies in order to increase the predictive power of the financial stability of a bank. These two approaches give exactly opposite skewnesses of their predictors that makes their combination logical and theoretically justified by forecast combination theory (Clemen, 1989; Timmermann, 2006). One of the first works devoted to this research question in credit risk sphere was a paper by Pompella and Dicanio (2017), who introduced a new approach (PC-Mahalanobis Method), which has parts from PD and CR modeling, for testing the validity of bank ratings assigned by RAs. However, the PC-Mahalanobis Method does not provide a numerical interpretation of the results and allows only to determine whether an observation belongs to either of the two binary groups: healthy or likely-to-fail banks. Therefore, the research gap of combination of PD and CR models for the Russian banks was fulfilled by this paper.

In order to use the combination of PD and CR models the literature streams that study the non-linear dependence between credit ratings and other fundamental risk parameters (PD, LGD and EAD) were analyzed. Different methods of comparing CR and PD were used in the academic papers (Godlewski, 2007; Chan-Lau, 2006; Schuermann and Hanson, 2004). For example, Godlewski in 2007 provided comparison of banks' credit ratings in emerging countries and their corresponding PD. The research showed the CR tends to aggregate bank default risk information into the intermediate-low rating grades and thus proved ratings' partial divergence from the results of a

PD scoring model. The connection between the ratings and loss given default was demonstrated in the articles (Shi et al., 2020; Shi et al., 2019; Rudakova and Ipatyev, 2015; Tasche, 2013; Volk, 2013). Therefore, most of the articles on this topic offer some econometric models, which can be used for interpreting of CRs with the help of PD and vice versa (Karminsky and Khromova, 2018; Zhivaikina and Peresetsky, 2017; Rudakova and Ipatyev, 2015; Tasche, 2013; Volk, 2013; Pomasanov and Vlasov, 2008)). Another literature stream focuses on the calibration of PD and CR to the same scale. The paper by Tasche (2013) compares a variety of calibration approaches and concludes that a 'scaled likelihood ratio' approach is superior to the standard 'scaled PDs' approach. Pomasanov and Vlasov (2008) introduced the model of credit rating calibration on PD for Russian banks. Alternatively, Pomazanov and Hamalinsky (2012) offered models for CR and PD calibration in samples with a small number of bankrupt firms. The method was based on the idea of benchmarking and genetic algorithms. Moreover, the tables with credit ratings and implied PDs are provided by RAs themselves: S&P (2019 Annual Corporate Default and Rating Transition Study), Moody's (Corporate Default and Recovery Rates, 1920-2008) and Fitch (2019 Transition and Default Studies).

Objectives of the research

The aim of this research is to increase the forecasting power of a bank's credit risk by building a synergic model that combines two of the most popular indicators of financial stability (CR and PD). The output of the synergic model is aimed to be transformed into the quantitative measure of credit risk that will allow investors to easily evaluate their potential losses.

In order to achieve the aim of the paper the following objectives were fulfilled: 1. Different methodologies of bank credit risk measurement were systemized (Probability of Default, Loss Given Default, Exposure At Default, Maturity, Ratings).

2. Factors of potential influence on bank credit risks that were used in the previous academic research were structured and summarized.

3. The representative dataset was obtained by filtration methods.

4. Two different models of credit risk (CR and PD) were run on the same dataset.

5. The comparison of the descriptive statistics of the distributions of CR and PD model forecasts, calibrated to a common scale, was conducted in order to develop proposals for the synergy of these models to improve the accuracy of estimates.

6. A synergic model with a higher forecasting power was constructed by assigning optimal weights and monotonic transformations to the PD and CR models.

7. The fitted values of the synergic model were transformed to PD using dynamic historic default frequencies.

The object and the subject of the research

The object of this thesis is Russian banks, while the subject is credit risk measured by probability of a bank's default and its credit rating and the nature of the interdependence between them.

Hypotheses of the research

Based on the analysis of the literature the following hypotheses were formulated.

Hypothesis 1. Some RAs on average tend to consistently assign relatively higher/lower ratings for the same bank.

This hypothesis was formulated alongside the paper (Karminsky and Peresetsky, 2007) that considered Big 3 international RAs and was also extended by the author to comparison of Russian RAs. It implies that there is a consistent difference in ratings assigned by various rating agencies to the same bank: some of them are relatively more conservative, while others are

relatively more liberal. In particular, it was already shown that Moody's tends to overestimate banks' ratings, while Standard and Poor's is the most conservative international rating agency (Karminsky and Peresetsky, 2007). This hypothesis makes it possible to carry out a comparative analysis of the subjectivity of RAs in assessing the credit risk of banks. Moreover, this study will also allow us to evaluate the financial transparency of Russian banks, which is inversely proportional to the difference between the rating grades of various RAs (Morgan, 2002).

Hypothesis 2. There is a significant divergence in the predictions of the CR and PD models: CR models give overestimated results due to reputational effect of RAs, while PD models are overfitted towards credit risk underestimation.

CR and PD models are checked for being overfitted towards credit risk over and underestimation respectively. Due to the presence of this divergency, the paper combines the previously used models of credit risk into a single scale and creates a synergic, more reliable, model of banks' credit risks based on publicly available information. According to the third hypothesis this may improve bank credit risk's forecasting power.

Hypothesis 3. A synergic (combined) model cancels out biases of PD and CR models and has higher credit risk's forecasting power compared to individual PD and CR models.

This hypothesis is checked by combination of the set of alternative models (CR and PD) by assigning optimal weights and monotonic transformations to them.

In order to obtain specific financial measures of credit risk, qualitative CR measures of the synergic model and quantitative PD measures are calibrated. For this purpose, a dynamic transition scale on the basis of the average historic default frequencies is created. The analysis of this scale enables a comparison of different credit risk minimization strategies and hence, the fourth hypothesis is formulated.

According to this hypothesis a bank that survives more than 1 year after low rating assignment has a sharp decrease in PD, while after high rating assignments PD gradually increases over time. The intuition behind this is that speculative banks that survive for a prolonged period are mainly small but stable, while banks with investment rating grades face huge competition and cannot fulfill regulator's requirements for a long period. Therefore, it is supposed that investors should account for not only the current rating grade of a bank, but also how long ago it was assigned.

The novelty of the research

The novelty of the thesis consists of the robust and comprehensive research in the field of a credit risk of a bank.

1. Empirical proof that credit risk overestimation tendency by CR models appears during post-2008 financial crisis period and is sharper for Russian banks compared to international ones.

2. Introduction of an algorithm of derivation of a synergic model based on a set of alternative models that shows superior forecasting power of credit risks.

3. Introduction of methodology of creation a dynamic calibration scale of qualitative CR measures into quantitative PD with superior features. First, it has detailed dynamic nature due to the usage of quarterly PD estimates. Second, it is compatible with all Ras, as it uses a base CR scale of all international and national RAs. Third, it focuses on Russian banks and thus includes country specific tendencies.

Methodology and Contribution

This thesis provides a new algorithm of a PD and CR synergic model construction that consists of several steps. First step is to construct a PD model and a CR model separately on the same dataset using the base rating scale. This part of the research is based on a summary of factors of potential influence on credit risk of a bank. After the predicted values of both models are generated, the calibration of CR and PD is realized in order to bring CR and PD into the single scale. Then the forecasting errors of each model are compared by their statistical distributions (mode, median, skewness). The divergence of both models from the actual data is calculated and the optimal weight coefficients and monotonic transformations for these two models that bring the forecasting error distribution closer to a normal distribution are found. The final version of PD and CR synergic model is further checked for its out-of-sample fit. Then the forecasts generated by the synergic model are transformed to PD using the dynamic transition scale of default frequencies introduced by the author. Using the obtained dynamic transition scale, it is possible to value the credit risk associated with a particular grade, which makes the forecasted grades of the synergic model more comprehensive and intuitive for banks' counteragents and may help them in investment decision making.

The following are the components of the methodology used in the successive stages of the implementation of the proposed algorithm, as well as the author's contribution at each of these stages.

Imbalanced data correction

The empirical part of the research is based on the dataset that was consolidated with the help of Matlab code from two separate databases. The first is the "Banks and Finance" database provided by the informational agency "Mobile",

3 This section presents amended and extended version of the following research papers: 1) Karminsky A., Khromova E. (2018) Increase of banks' credit risks forecasting power by the usage of the set of alternative models. Russian Journal of Economics 4(2): 155-174; 2) Khromova E. (2020) Dynamic Mapping of Probability of Default and Credit Ratings of Russian Banks. Journal of Corporate Finance Research. 14 (4): 31-46

while the second is the database of Central Bank of Russia, which consists of the RAs statements of all Russian licensed banks. The unbalanced panel dataset (as it included defaulted banks that have less time observations) of Russian banks was extracted for the period 2007-2017. The reduction of the sample size appeared mainly due to the fact that only a small share of banks was assigned a CR grade. The historical data of CR changes by national and international RAs was taken from Cbonds.ru and Bankodrom.ru that are the main on-line aggregators of banking statistics. Then the data on banks' defaults were collected from Cbr.ru and Banki.ru. The total number of banks after filtration was 395 (86 of them experienced the default), so the overall number of observations was 11 627 that is enough to make consistent conclusions. In addition, data on macroeconomic factors is taken from Rosstat and World Bank databases.

However, the collected panel database was imbalanced (223 of defaults compared to 11 404 of non-defaults). The nature of imbalanced data is intrinsic (corresponds to the nature of a data set). Furthermore, there is no data available for the "default" class after bank had experienced the default. It leads to embedded rarity and within-class imbalances as well as the failure of generalizing inductive rules by learning algorithms. Therefore, using methodology described by He and Garcia (2009), the combination of random undersampling and oversampling methods was used to obtain the final data for the regression analysis. According to the methodology of He and Garcia, a set of majority class examples was randomly selected and removed from data and then the random set of minority class with new banks' names was added. The selected approach was proved to be useful in eliminating the problems associated with the imbalanced nature of the collected database.

Base scale formation

The credit agencies' assessments were reflected into one base scale for correct comparison and for eliminating the possible methodological differences.

According to the methodology of Karminsky and Sosurko (2011) the best results of mapping scales are obtained by using the class of linear-logarithmic transformations. In this case, the parametrization of mappings implies finding a pair of coefficients for mapping each of the scales into a basic one (free term and coefficient in front of the logarithm of the described rating scale). Moody's was chosen as a dependent variable for the base scale construction. Therefore, the following regression was run in order to fulfill the mapping procedure:

LN(Mt) = ajLNiRt)+ ft + Ei , (1)

where Mt is a Moody's international scale, taken as a base scale and Rt is the scale of CR that should be transformed to a base scale. Therefore, it allows a solution using the least-squares regression. The vector of coefficients (j) for international agencies Moody's, Standard&Poor's, Fitch (both international and national scales) and national Ras such as RAEX, NRA, Rus-Rating and AK&M and Ria-Rating were calculated using methodology of Karminsky and Sosurko (2011). Credit ratings mapping allowed the author to construct a single rating space and to compare the credit ratings of different agencies more accurately.

CR and PD modelling via logistic regressions

As for the modeling methods of this research, binary logit/probit regressions were chosen for PD estimation and Multinomial ordered logit/probit for credit ratings modeling by the classical methodology introduced by Kaplan and Urwitz in 1979. These models continue to be actively used in many modern international studies (Fernando et al., 2019; Darrat et al., 2016; Lin and Yang ,2016; Ciampi, 2015; Karminsky and Kostrov, 2014). Moreover, it was shown (Jiao et al., 2007; Karminsky and Kostrov, 2017; Zan et al., 2004) that the predictions of more complex modeling methods like artificial intelligence models did not outperform the standard binary and ordered multinomial models.

Principal component analysis

Country-specific variables and interaction terms were included in the models for precise credit risk estimation. It was noted that macroeconomic variables and cross terms of financial variables were heavily correlated with each other, which could inevitably lead to multicollinearity problem if no actions were taken. Therefore, due to the fact that the model is constructed primarily to be applied in forecasting, principal component analysis (PCA) is used to eliminate potential problems. PCA methodology, introduced by Hotelling (1933) and Pearson (1901) and actively used on Russian data after Ayvasyan et al (1989) is exploited to reveal the intrinsic structure of the relations between the involved variables and reduce the number of dimensions needed to capture the dispersion. Implementation of this method allowed to capture the biggest amount of information about the data with the minimum number of variables, which also eliminated the problem of overfitting the model.

Calibration of rating scale and PD

CR and PD models were presented in the same scale in order to compare their forecasting power. The research uses methodology by Pomasanov and Vlasov (2008), where the model of credit ratings calibration on PD for Russian banks was introduced. This thesis modifies the calibration scale by extrapolation the CR scale to the continuous scale in order to obtain results for the base scale of 32 dimensions that was used in the research. The non-linear relationship between the rating scale and PD was demonstrated. The correspondence of PD to the highest ratings from ruAAA to ruA+ was estimated by the exponential function and for the middle ratings (from ruA+ to ruCCC+) a convex polynomial quadratic function was used. That proves that PD increases with accelerating pace for these ratings' grades. For the bottom ratings (from ruCCC+ to ruD) a concave polynomial quadratic function proved to be the most appropriate approximation, that shows the decelerating rate of change in PD to rating scale.

Construction of synergic models

Algorithm of construction of a reliable synergic model was introduced by the author (Khromova and Karminsky, 2018). In order to do that as the first step the ratings' grade forecasts by PD and CR models were computed for the same observations. As the second step of the algorithm, the regressions, where the dependent variable was the actual rating and explanatory variables were the fitted values of Rating and Default model, were run on the 3 011 observations. Two different model specifications reached the best forecasting power results. The first of them was the polynomial synergic model that can be used for any banks even without historical ratings:

Yu = PiPDit + foPD3it + PsCRit + uit, (2)

where Yit - is the actual rating, PDit and CRit- are the predicted ratings by PD and CR models respectively and uit ~ a2) is the error component which has a logistic distribution with mean ^ = 0 and standard deviation a =

n

73.

The best forecasting power was reached in the ADL (1,0) synergic model specification, although this model requires the presence of at least half year-old rating assessment:

Yit = PiYi(t-i) + P2Ln(CRit - Yi(t-1)) + P3Ln(PDit - Yi(t-1)) + uit, (3)

where Yit - is the actual rating, Yi(t-1) - is a half a year lag in the actual rating, PDit and CRit- are the predicted ratings by PD and CR models respectively. The ADL (1,0) synergic model that was obtained by the lagged dependent variable amended to CR and PD forecasting errors was found to have the highest predicted power with the smallest deviations.

Construction of dynamic calibration scale of CR to PD

In order to make the forecasted grades of the synergic model more comprehensive and intuitive for investors and help them in investment decision

making, the qualitative CR grades should be transformed to PD. Different methods of comparing the credit ratings and PD were shown in the academic papers (Godlewski, 2007; Chan-Lau, 2006; Schuermann and Hanson, 2004). Moreover, the tables with credit ratings and implied PDs are provided by RAs themselves: S&P (2019 Annual Corporate Default and Rating Transition Study), Moody's (Corporate Default and Recovery Rates, 1920-2008) and Fitch (2019 Transition and Default Studies). However, one faces several limitations in these methodologies while applying these scales to Russian banks:

• First, RAs do not provide the corresponding scales of credit rating conversion for different countries and for different geographic groups. The data used by the RAs to prepare these calculations mostly include their home country (the USA) and the countries of major shareholders (developed countries such as Canada and the UK). Russian banks are not representative in such databases.

• Second, the values of annual historical default frequencies (estimates of default probabilities) for various credit ratings are calculated by each RA empirically (based on default statistics of banks with credit ratings of a particular RA), which leads to an inadequate comparison of the level of creditworthiness for the same rating class in different time periods.

• Third, the scales provided by each RA are not dynamic in nature, i.e. they provide only annual frequencies. An investor may not evaluate the possible losses which can occur in the short run (in a month/quarter after the investment is made).

Taking into account all the above-mentioned limitations, we aim to construct a uniform dynamic scale of the credit rating score calibration to PD using the historic default frequencies of Russian banks. The preparation of dynamic calibration scale included several steps. First of all, the matrix for each credit rating score was calculated. It showed the default frequency for the banks which were assigned a particular credit rating in each of the time periods

presented in the sample. It is important to notice that the quarter of a particular credit rating assignment was not fixed for proper analysis. The period after which a bank went bankrupt was estimated starting from the moment of rating assignment over the entire time horizon. As the third step, the obtained results were summarized for each rating score presented in the obtained sample of Russian banks. The intermediate tables were constructed where default frequencies were presented. They were used to estimate the PD in each of the time periods (quarters) for each credit ratings from the database. Finally, the estimated default frequencies for a set of rating grades were averaged for a particular rating class. This was done for more logical representation of the obtained results and for keeping an approximately equal number of bankperiods in each class.

Main findings

The main findings of this study are as follows:

• It was found that among international rating agencies Moody's is less conservative in assigning ratings to banks than S&P (Hypothesis 1 was not rejected for international RAs). On the historical data set of Russian rating agencies, the CR model of Rus Rating consistently underestimates the ratings of issuers in comparison with other RAs, and RAEX assigns the highest ratings to the same issuers in comparison with competitors (Hypothesis 1 was not rejected for national RAs). In addition, it has been demonstrated that adjusting credit ratings to a base scale helps to increase accuracy of ratings forecasts.

• It was shown that there is a significant divergence in the predictions of CR and PD models: credit ratings' models tend to overestimate a credit risk of a bank, whereas PD models give underestimated results (Hypothesis 2 was not rejected). It was empirically proven that credit risk overestimation tendency by rating models appears after 2008 crisis and becomes even sharper for Russian banks. Therefore, both models

have forecasting bias that decreases the number of correctly predicted forecasts.

• It was estimated that the forecast combination of individual models (CR and PD) improved banks' credit risks forecasting power. The polynomial synergic model reached 41% precise rating estimates and 74% estimated with deviation less than one grade. The ADL (1,0) synergic model that can be used by banks with recent rating history with even higher forecasting power (67% of precise estimates) was also obtained. That is much higher than the separate forecasts of PD and CR models (21% and 32%, respectively). Moreover, both synergic models have shown superior out-of-sample predictive power: polynomial model reached up to 36% of precise estimates in a 32-grades rating scale, while ADL (1,0) model proved to predict correctly up to 63% of estimates (Hypothesis 3 was not rejected). Moreover, the forecast combination puzzle was not supported by this research as a more complex non-linear combination models with weights estimation performed better than forecast combination with equal weights.

• It was shown that investors should account for not only the current rating grade of a bank, but also how long ago it was assigned. Dynamic CR to PD calibration scale showed that banks' probability of default increases after high rating (BB class) assignment and decreases over time after lower rating (B, CCC, C class) assignment (Hypothesis 4 was not rejected). Moreover, it was shown that RAs overestimate credit risk for some banks: the 3-year increment of probability of default 3 years after rating assignment is higher in a rating class CCC (24%) than in a class C (16%) in absolute terms.

100%

+16%

90%

T

83%

20%

40%

70%

60%

50%

80%

30%

10%

0%

BB

B

CCC

C

■ DF in 1 year ■ DF in 2 years ■ DF in 3 years ■ DF in 4 years ■ DF in 5 years

Figure 1 - Distribution of annual cumulative default frequencies by rating classes

(Source: Khromova, 2020)

As a result, the credit risk minimization strategy was formulated. Investment in banks with better credit ratings is less risky right after the rating issue and is efficient to be held for the short run period. However, to minimize credit risk of capital investment in banks with highly speculative rating grades, it is optimal to choose a long run investment 1-2 years after the rating assignment.

Theoretical implications of the study

The theoretical significance of this research consists of possibility to extend the introduced methods for assessing the credit risk of a bank. The methodology for constructing a synergic model of CR and the PD can be subsequently extended by researchers to other models for assessing bank credit risks in order to increase their predictive power. In addition, the work fills in the research gap of comparative analysis of the effectiveness of credit rating models depending on the time period and country of origin of the issuer. Moreover, the proposed improved dynamic calibration scale of CR to PD, based on the relevant data on Russian credit institutions, can significantly simplify future research in this

field.

Practical implications of the study

The results of the study can be used by all bank counterparties to determine the financial stability of Russian banks when distributing funds based on publicly available information. This research helps depositors and investors not only calculate a rating for banks that have not been previously assigned a rating, but also determine the optimal moment and time horizon for placing funds in terms of credit risk minimization. In addition, the proposed mechanism can be used by supervisors and by banks themselves as a tool for constant control with the possibility of taking adequate measures in advance.

Approbation of results

The results of the studies were presented at the following conferences:

1. Analytics for Management and Economics Conference (AMEC) (Online/ St. Petersburg, Russia, 2020). "Empirical Modeling of International Banks' Credit Risk: Assessment and Comparison of Credit Ratings"

2. 32nd Eurasia Business and Economics Society Conference (Online/ Turkey, 2020). "Empirical modeling of international banks' credit risk: assessment and comparison of credit ratings."

3. IX Moscow International Conference on Operations Research (ORM), (Moscow, Russia, 2018). "Increase of Banks' Credit Risks Forecasting Power by the Usage of the Set of Alternative Models."

4. Second World Congress of Comparative Economics (St. Petersburg, Russia, 2017). "Assessment of Banks' Credit Risks Using a Set of Alternative Models."

5. Information Technology and Quantitative Management Conference (Seoul, South Korea, 2016). "Extended Modeling of Banks' Credit Ratings."

6. 17th Eurasia Business and Economics Society Conference (Venice, Italy, 2015). "Modeling Banks' Credit Ratings of International Agencies."

The research results were repeatedly presented within Scientific Research Seminar on Empirical Studies of Banking and deeply discussed with external specialists. Moreover, PD and CR models of this thesis have also found application in teaching activities of the author at Higher School of Economics (double degree Programme in International Relations) in the framework of practical parts on risk management in financial institutions of the "International Finance and Globalization" course.

Furthermore, during all of the research period the author was a part of the Research Group "Innovations in the banking sector, its financial stability and prudential regulation" and Scientific-Educational Group "Formation of a system of models for managing a bank's credit risk in conditions of financial instability" at Higher School of Economics. Therefore, the robustness of the research was tested by the team members.

Publications

JEL: G17, G21, G24, G33, C23, C53

The main results of the following PhD dissertation are published in five papers with a total volume of 6.4 author's sheets with the author's contribution of 5.6 author's sheets:

1. Khromova E. (2020) Dynamic Mapping of Probability of Default and Credit Ratings of Russian Banks. Journal of Corporate Finance Research

14(4): 31-46

doi: 10.17323/j.jcfr.2073-0438.14.4.2020.31 -46

2. Khromova E. (2020) Increase of Banks' Credit Risk Forecasting Power Using the Set of Credit Ratings and Probability of Default Models. Recent Advances of the Russian Operations Research Society 1st Edition: 177-196

ISBN (10): 1-5275-4792-2; ISBN (13): 978-1-5275-4792-6

3. Karminsky A., Khromova E. (2018) Increase of banks' credit risks forecasting power by the usage of the set of alternative models. Russian Journal of Economics 4(2): 155-174

doi: 10.3897/j. ruje. 4.27737

4. Karminsky A., Khromova E. (2016) Extended modeling of banks' credit ratings. Procedia Computer Science 91: 201 - 210

doi: 10.1016/j.procs.2016.07.058

5. Karminsky A., Khromova E. (2016) Modelling banks' credit ratings of international agencies. Eurasian Econ Rev (2016) 6: 341-363

doi: 10.1007/s40822-016-0058-5

Structure of the thesis

The thesis contains of the introduction, four chapters, conclusion, reference list and appendixes. The total size of the thesis includes 151 pages of the main text with 38 pages of reference list and appendixes. The reference list contains 246 papers, 55% of which were published within the last 10 years.

Заключение диссертации по теме «Финансы, денежное обращение и кредит», Хромова Элла Павловна

Conclusion

The paper is aimed to compare divergence of existing models of credit risks and to create a synergic reliable model. For this purpose, it applied Rating model and Probability of Default model to the same dataset and normalised its estimates to the common scale. After thorough analysis of probability density functions of that output, the optimal weights and monotonic transformations were assigned to each model. As a result, the polynomial synergic model (that predicted 41% of precise estimates out of 32-grade scale) and the ADL (1,0) synergic model with even higher forecasting power (67% of precise estimates) were obtained. Polynomial synergic model can be used for banks without rating history, while ADL (1,0) synergic model requires the presence of at least half year-old rating assessment. The resulting symbolic CR grades of the polynomial synergic model were converted into 1-year cumulative PD using a dynamic transition scale. The converted PD estimates outperformed single models in terms of AUC of a ROC curve (AUC equal to 0.891 was achieved with the synergic model).

The following research has stated 4 hypotheses and all of them were not rejected and found some support by this paper.

• It was found that among international rating agencies Moody's is less conservative in assigning ratings to banks than S&P (Hypothesis 1 was not rejected for international RAs). On the historical data set of Russian rating agencies, the CR model of Rus Rating consistently underestimates the ratings of issuers in comparison with other RAs, and RAEX assigns the highest ratings to the same issuers in comparison with competitors (Hypothesis 1 was not rejected for national RAs). In addition, it has been demonstrated that adjusting credit ratings to a base scale helps to increase accuracy of ratings forecasts.

• It was shown that there is a significant divergence in the predictions of CR and PD models: credit ratings' models tend to overestimate a credit

risk of a bank, whereas PD models give underestimated results (Hypothesis 2 was not rejected). It was empirically proven that credit risk overestimation tendency by rating models appears after 2008 crisis and becomes even sharper for Russian banks. Therefore, both models have forecasting bias that decreases the number of correctly predicted forecasts.

• It was estimated that the usage of the set of alternative models (CR and PD) improved banks' credit risks forecasting power. The polynomial synergic model reached 41% precise rating estimates and 74% estimated with deviation less than one grade. The ADL (1,0) synergic model that can be used by banks with recent rating history with even higher forecasting power (67% of precise estimates) was also obtained. That is much higher than the separate forecasts of PD and CR models (21% and 32%, respectively). Moreover, both synergic models have shown superior out-of-sample predictive power: polynomial model reached up to 36% of precise estimates in a 32-grades rating scale, while ADL (1,0) model proved to predict correctly up to 63% of estimates (Hypothesis 3 was not rejected). Moreover, the forecast combination puzzle was not supported by this research as a more complex non-linear combination models with weights estimation performed better than forecast combination with equal weights.

• It was shown that investors should account for not only the current rating grade of a bank, but also how long ago it was assigned. Dynamic CR to PD calibration scale showed that banks' probability of default increases after high rating (BB class) assignment and decreases over time after lower rating (B, CCC, C class) assignment (Hypothesis 4 was not rejected). Moreover, it was shown that RAs overestimate credit risk for some banks: the 3-year increment of probability of default 3 years after rating assignment is higher in a rating class CCC (24%) than in a class C (16%) in absolute terms. As a result, the credit risk minimization

strategy was formulated. Investment in banks with better credit ratings is less risky right after the rating issue and is efficient to be held for the short run period. However, to minimize credit risk of capital investment in banks with highly speculative rating grades, it is optimal to choose a long run investment 1-2 years after the rating assignment.

The novelty of the thesis consists of the robust and comprehensive research in the field of a credit risk of a bank.

1. Empirical proof that credit risk overestimation tendency by CR models appears during post-2008 financial crisis period and is sharper for Russian banks compared to international ones. Forecast errors were shown to be on average 2 (out of 32) grades biased.

2. Introduction of an algorithm of derivation of a synergic model based on a set of alternative models that shows superior forecasting power of credit risks. Polynomial and ADL (1,0) synergy models showed 41% and 67% precise estimates out of 32-grade scale respectively that is at least 23pp higher than in a single CR model.

3. Introduction of methodology of creation a dynamic calibration scale of qualitative CR measures into quantitative PD with superior features. First, it has detailed dynamic nature due to the usage of quarterly PD estimates. Second, it is compatible with all RAs, as it uses a base CR scale of all international and national RAs. Third, it focuses on Russian banks and thus includes country specific tendencies.

4. Introduction of credit risk minimization strategy based on a calibration scale analysis. The better is a bank's credit rating, the shorter investment horizon should be and the closer the date of investment should be to the rating assignment date. The proper moment of investment and investment horizon may help to reduce credit risk (measured as PD) up to two times.

In further research we are going to apply such technique of derivation of a synergic model to other credit risk measurements. Moreover, we are already building a synergic PD and CR model for international banks (based on the data sample described in section 2.1.2) and are going to compare the results with Russian banks. Furthermore, in further research we are going to check other monotonic transformations for the synergic model and try to use specialized tools for optimization of the functional form of a synergic model. In addition, we would like to introduce one more robustness check of the synergic model by replacing an outdated PD to CR calibration scale (Table 3.4) that was used during synergic model construction by a new calibration scale introduced in Chapter 4 (Table 4.1). As previously shown in the paper, rating models overestimate credit risk more after 2008, so we expect that using an updated scale will increase the discrepancy between credit rating and default probability models and thus will enlarge the efficiency of the synergic model. Moreover, more sophisticated methods of balanced dataset formation that are discussed in the papers (He, Garcia, 2009; Hosmer, Lemeshow, 2000) should be also tested. Finally, other statistical or artificial intelligence methodologies can be used in CR and PD modelling in order to increase their and consequently the synergic model forecasting power. Moreover, the last chapter that contains derivation of a dynamic calibration scale of CR to PD will be developed further as a separate research topic. The empirical research will be done in order to study the reasons for the spikes in imminent PD after CR assignment. For that reason, the dynamic scale will be tested for separate data samples of opaque and transparent bank.

Список литературы диссертационного исследования кандидат наук Хромова Элла Павловна, 2021 год

References

1) Aiolfi, M. & Favero, C.A. (2003). Model Uncertainty, Thick Modeling and the Predictability of Stock Returns. Forthcoming in Journal of Forecasting.

2) Aiolfi, M. & Timmermann, A. (2004). Persistence of Forecasting Performance and Combination Strategies. Mimeo, UCSD.

3) Aitchison, J. & Silvey, S. (1957). The generalization of probit analysis to the case of multiple responses. Biometrika, 44(1-2), 131-140.

4) Ali, A. & Durdu, K. (2012). An assessment of strategic importance of credit rating agencies for companies and organizations. Procedia - Social and Behavioral Sciences, 58, 1628-1639.

5) Allen, F., Fulghieri, P. & Mehran, H. (2011).The value of bank capital and the structure of the banking industry.Review of Financial Studies, 24

(4).

6) Almamy, J., Aston, J. & Ngwa, L. (2016). An evaluation of Altman's Z-score using cash flow ratio to predict corporate failure amid the recent financial crisis: Evidence from the UK. Journal of Corporate Finance, 36, 278-285.

7) Alrabadi, D. & Hamarneh, R. (2016). Determinants of Banks' Credit Ratings: Evidence from Jordan. Jordan Journal of Business Administration,12(4), 919-933.

8) Alsakka R., Gwilym O. (2012). The causes and extent of split sovereign credit ratings in emerging markets. Emerging Markets Finance and Trade, 48 (1).

9) Altman, E. & Rijken, H. (2004). How rating agencies achieve rating stability. Journal of Banking & Finance, 28(11), 2679-2714.

10) Altman, E. I. & Saunders, A. (1998). Credit risk measurement: Developments over the last 20 years. Journal of Banking & Finance, 21(11-12), 1721-1742.

11) Altman, E. I. (1968). Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. The Journal of Finance, 23(4), 589609.

12) Altman, E., Haldeman, R. & Narayanan, P. (1977). ZETATM analysis. A new model to identify bankruptcy risk of corporations. Journal of Banking & Finance, 1(1), 29-54.

13) Altman, E.I., Brady, B., Resti, A. & Sironi, A. (2005). The Link between Default and Recovery Rates: Theory, Empirical Evidence, and Implications. The Journal of Business, 78(6), 2203-2228.

14) Amato, J. D. & Furfine, C. H. (2004). Are credit ratings procyclical? Journal of Banking and Finance, 28(11), 2641-2677.

15) Ammar, S., Duncombe, W., Hou, Y., Jump, B. & Wright, R. (2001). Using Fuzzy Rule-Based Systems to Evaluate Overall Financial Performance of Governments: An Enhancement to the Bond Rating Process. Public Budgeting & Finance.Vol.21, 91-110.

16) Anderson, H. M. & Vahid, F. (2001). Predicting the probability of a recession with nonlinear autoregressive leading indicator models. Macroeconomic Dynamics, 5, 482-505.

17) Andrawis, R., Atiya, A.F. & El-Shishiny, H. (2011). Combination of long term and short term forecasts, with application to tourism demand forecasting. International Journal of Forecasting, 27, 870-886.

18) Anginer, D., Demirgu?-Kunt, A. & Zhu, M. (2012). How does deposit insurance affect bank risk? Evidence from the recent crisis. World bank Policy Research Working Paper. 6289.

19) Anzoategui, D., Peria, M.S.M. & Melecky, M. (2012). Bank competition in Russia: An examination at different levels of aggregation. Emerging Markets Review, 13, 42-57.

20) Armstrong, J. S. (2001). Combining forecasts. In J. S. Armstrong (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic Publishers.

21) Ashbaugh-Skaife, H., Collins, D. & LaFond, R. (2006). The effects of corporate governance on firms' credit ratings. Journal of Accounting and Economics, 42, 203-243.

22) Ashraf, B., Zheng, C., Jiang, C. & Qian, N. (2020). Capital regulation, deposit insurance and bank risk: International evidence from normal and crisis periods. Research in International Business and Finance, 52, 101188.

23) Atiya A.F. (2020). Why does forecast combination work so well? International Journal of Forecasting, 36(1), 197-200.

24) Attig, N., Ghoul, S., Guedhami, O. & Suh, J. (2013). Corporate Social Responsibility and Credit Ratings. Journal of Business Ethics, 117(4), 679-694.

25) Ayvasian, S.A., Buhshtaber V.M., Enukov I.S. & Meshalkin L.D. (1989). Applied statistics. Classification and dimension decrease. Finance and statistics — 607 (in Russian)

26) Ayvasian, S.A., Golovan, S.V., Karminsky, A.M. & Peresetsky, A.A. (2011). About the approaches to a comparison of rating scales. Prikladnaia Ekonometrika (Applied Econometrics), 3 (23), 13-40. (in Russian)

27) Baetschmann, G., Staub, K. & Winkelmann, R (2011). Consistent Estimation of the Fixed Effects Ordered Logit Model. IZA Discussion Paper No. 5443. Available at http://ftp.iza.org/dp5443.pdf .

28) Balios, D. & Thomadakis, S. & Tsipouri, L. (2016). Credit rating model development: An ordered analysis based on accounting data. Research in International Business and Finance, Elsevier, 38(C), 122-136.

29) Barth, J.R., Caprio, G. & Levine, R. (2001). Bank regulation and supervision: what works best? Journal of Financial intermediation, 13, 205-248.

30) Barton A. (2006). Split credit ratings and the prediction of bank ratings in the Basel II environment University of Southampton // University of Southampton, doctoral thesis.

31) Batchelor, R. & Dua, P. (1995). Forecaster diversity and the benefits of combining forecasts. Management Science, 41, 68-75.

32) Bates, J.M. & Granger C.W.J. (1969). The Combination of Forecasts. Operations Research Quarterly 20, 451-468.

33) Becker, B. & Milbourn, T. (2010). How Did Increased Competition Affect Credit Ratings? Harvard Business School Finance Working Paper No. 09-051.

34) Behr P., Guttler A. (2008). The informational content of unsolicited ratings. Journal of Banking & Finance 32, 587-599.

35) Bellotti, T., Matousek, R. & Stewart, C. (2011). Are rating agencies' assignments opaque? Evidence from international banks, Expert Systems with Applications, 38, 4206-4214.

36) Bennell, J. A., Crabbe, D., Thomas, S. & Gwilym, O. (2006). Modeling sovereign credit ratings: neural networks versus ordered probit. Expert Systems with Applications, 30(3), 415-425.

37) Berger, A.N., Davies, S.M. & Flannery, M.J. (2000). Comparing market and supervisory assessments of bank performance: who knows what when? Journal of Money, Credit and Bank, 32, 641-667.

38) Bermpei, T., Kalyvas, A. & Nguyenc, T. (2018). Does institutional quality condition the effect of bank regulations and supervision on bank stability? Evidence from emerging and developing economies. International Review of Financial Analysis, 59, 255-275.

39) Betz, F., Oprica, S., Peltonen, T. & Sarlin, P. (2014). Predicting distress in European banks. Journal of Banking & Finance, 45, 225-241.

40) Bissoondoyal-Bheenick E. (2005). An analysis of the determinants of sovereign ratings. Global Finance Journal, No.15.

41) Bolton, P., Freixas, X. & Shapiro, J. (2012). The credit ratings game. The Journal of Finance, 76(1), 85-111.

42) Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3.

43) Buckle, M. & Beccalli, E. (2011). Principles of banking and fmance.University of London.c.206.

44) Budescu, D. V. & Chen, E. (2015). Identifying expertise to extract the wisdom of crowds. Management Science, 61(2), 267-280.

45) Cao, Lijuan, Guan, L.K. & Jingqing, Z. (2006). Bond rating using support vector machine. Intelligent Data Analysis, 10, 285-296.

46) Caporale, G. M., Matousek, R. & Stewart, C. (2012). Ratings assignments: Lessons from international banks. Journal of International Money and Finance, 31, 1593-1606.

47) Chan-Lau, J. (2006). Fundamentals-Based Estimation of Default Probabilities: A Survey. IMF Working Papers, 2006(149).

48) Christoffersen, P. & Diebold, F.X. (1997). Optimal Prediction under Asymmetrical Loss. Econometric Theory 13, 806-817.

49) Ciampi, F. (2015). Corporate governance characteristics and default prediction modeling for small enterprises. An empirical analysis of Italian firms. Journal of Business Research, 68(5), 1012-1025.

50) Claeskens, G., Magnus, J.R., Vasnev, A.L. & Wang, W. The forecast combination puzzle: A simple theoretical explanation. International Journal of Forecasting, 32(3), 754-762.

51) Clarke, G., Cull, R. & Shirley, M. (2005).Bank privatization in developing countries: A summary of lessons and findings. // Journal of Banking & Finance, 29(8/9), 1905-1930.

52) Clemen, R.T. & Winkler, R.L. (1986). Combining Economic Forecasts, Journal of Business andEconomic Statistics, 4, 39-46.

53) Clemen, R.T. (1987). Combining Overlapping Information. Management Science 33(3), 373-380.

54) Clemen, R.T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5(4), 559-583.

55) Clements, M.P., Harvey, D.I. (2011). Combining probability forecasts. International Journal of Forecasting, 27(2), 208-223.

56) Cole, R. & White, L. (2012). Déjà Vu All Over Again: The Causes of U.S. Commercial Bank Failures This Time Around. Journal of Financial Services Research, 42, 5-29.

57) Crane, D.B., Crotty J.R. (1967). A Two-Stage Forecasting Model: Exponential Smoothing and Multiple Regression. Management Science, 13(8), B387-C203.

58) Cucinelli, D., Battista, M., Marchese, M. & Nieri, L. (2018). Credit risk in European banks: The bright side of the internal ratings based approach. Journal of Banking & Finance, 93, 213-229.

59) Darrat A. F., Gray S., Park J.C, Wu Y. (2016). Corporate Governance and Bankruptcy Risk. Journal of Accounting, Auditing and Finance, 31(2), 163-202

60) Dash, M. & Das, A. (2013). Performance Appraisal of Indian Banks Using CAMELS Rating. IUP Journal of Bank Management, 12(2), 31-42.

61) De Menezes, L.M., Bunn, D.W. & Taylor, J.W. (2000). Review of guidelines for the use of combined forecasts. European Journal of Operational Research, 120, 190-204.

62) Diebold, F.X. & Lopez, J. A. (1996). Forecast Evaluation and Combination. In Maddala and Rao (eds.) Handbook of Statistics. Elsevier: Amsterdam.

63) Diebold, F.X. & Rudebusch, G.D. (1989). Scoring the leading indicators. Journal of Business, 62, 369-391.

64) Dimitrov, V., Palia, D. & Tang, L. (2015). Impact of the Dodd-Frank act on credit ratings. Journal of Financial Economics, 115(3), 505-520.

65) Distinguin, I., Hasan, I. & Tarazi, A. (2013). Predicting rating changes for banks: how accurate are accounting and stock market indicators? Annals of Finance, 9(3), 471-500.

66) Driss, H., Massoud, N. & Roberts, G. (2019). Are credit rating agencies still relevant? Evidence on certification from Moody's credit watches. Journal of Corporate Finance, 59,119-141.

67) Duff, A. & Einig, S. (2009).Credit ratings quality: The perceptions of market participants and other interested parties. The British Accounting Review, 41, 141-153.

68) Elliott, G. & Timmermann, A. (2003). Optimal Forecast Combination Weights Under Regime Switching. Forthcoming, International Economic Review.

69) Elliott, R., Siu, T. & Fung, E. (2014). A Double HMM approach to Altman Z-scores and credit ratings. Expert Systems with Applications, 41(4/2), 1553-1560.

70) Esarey, J., & Pierce, A. (2012). Assessing fit quality and testing for misspecification in binary-dependent variable models. Political Analysis, 20 (4), 480-500.

71) Feng Q., Menggang L., Gong D. (2014). Research on the competitiveness of crediting rating industry using PCA method. Journal of Industrial Engineering and Management. JIEM, 2014 N 7(5). PP. 1268-1282.

72) Fiorucci, J.A., Louzada, F. (2020). GROEC: Combination method via Generalized Rolling Origin Evaluation. International Journal of Forecasting, 36(1), 105-109.

73) Flannery, M.J. (2001). The faces of market discipline. Journal of Financial Services Research, 20(2-3), 107-119.

74) Florez-Lopez, R. & Ramon-Jeronimo, J-M. (2014). Modelling credit risk with scarce default data: on the suitability of cooperative bootstrapped strategies for small low-default portfolios. Journal of the Operational Research Society, 65 (3), 416-434.

75) Flynn S., Ghent A. (2018). Competition and credit ratings after the fall. Management Science, N 64(4). PP. 1672-1692.

76) Fungacova , Z., & Weill, L. (2009). How market power influences bank failures: Evidence from Russia. // BOFIT Discussion papers, 12/2009

77) Fungacova, Z. & Solanko, L. (2009). Risk-taking by Russian banks: Do location, ownership and size matter? // BOFIT Discussion papers, 21/2008

78) Gaba, A., Tsetlin, I. & Winkler, R.L. (2017). Combining interval forecasts. Decision Analysis, 14(1), 1-20.

79) Garcia, V., Sanchez, J. & Mollineda, R. (2012). On the effectiveness of preprocessing methods when dealing with different levels of class imbalance. Knowledge-Based Systems, 25 (1), 13-21.

80) Genre, V., Kenny, G., Meyler, A. & Timmermann, A. (2013). Combining expert forecasts: can anything beat the simple average? International Journal of Forecasting, 29, 108-121.

81) Gneiting, T. & Raftery, A.E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

82) Godlewski, C. (2007). Are Ratings Consistent with Default Probabilities?: Empirical Evidence on Banks in Emerging Market Economies. Emerging Markets Finance and Trade, 43(4), 5-23.

83) Gogas, P., Papadimitriou, T. & Agrapetidou, A. (2014). Forecasting bank credit ratings. The Journal of Risk Finance, 15(2), 195-209.

84) Graefe, A., Armstrong, J.S., Jones Jr, R.J. & Cuzan, A.G. (2014). Combining forecasts: An application to elections. International Journal of Forecasting, 30, 43-54.

85) Granger, C.W.J. and Jeon, Y. (2004). Thick Modeling, Economic Modelling, 21, 323-343.

86) Granger, C.W.J. and Ramanathan, R. (1984). Improved Methods of Combining Forecasts. Journal of Forecasting 3, 197-204.

87) Gropp, R., Vesala, J. & Vulpes G. (2006). Equity and Bond Market Signals as Leading Indicators of Bank Fragility. Journal of Money, Credit and Banking, 38, 399-428.

88) Grunert, J., Norden, L. & Weber, M. (2005). The role of non-financial factors in internal credit ratings. Journal of Banking and Finance, 29, 509-531.

89) Grushka-Cockayne, Y. & Jose, V.R.R. (2020). Combining prediction intervals in the M4 competition. International Journal of Forecasting, 36(1), 178-185.

90) Grushka-Cockayne, Y., Jose, V.R.R. & Lichtendahl Jr, K.C. (2017). Ensembles of overfit and overconfident forecasts. Management Science, 63(4), 1110-1130.

91) Gu, X., Kadiyala, P. & Mahaney-Walter, X. (2018). How creditor rights affect the issuance of public debt: The role of credit ratings. Journal of Financial Stability, 39,133-143.

92) Hajek, P. & Michalak, K. (2013). Feature selection in corporate credit rating prediction. Knowledge-Based Systems, 51, 72-84.

93) Hajek, P. & Olej, V. (2010). Credit rating modelling by kernel-based approaches with supervised and semi-supervised learning. Neural Computing and Applications, 20, 761-773.

94) Hajek, P. (2011). Municipal credit rating modelling by neural networks. Decision Support Systems, 51, 108-118.

95) Hanafi, M., Santi, F., & Muazaroh, (2013). The impact of ownership concentration, commissioners on bank risk and profitability: Evidence from Indonesia. // Eurasian Economic Review, 3(2), 183-202.

96) He H. & Garcia E. A., (2009), Learning from Imbalanced Data, IEEE Trans. Knowledge and Data Engineering, 21(9), 1263-1284.

97) Hendry, D.F. & Clements, M.P. (2004). Pooling of forecasts. The Econometrics Journal, 7, 1-31.

98) Hendry, D.F. and Clements, M.P. (2002). Pooling of Forecasts. Econometrics Journal 5, 1-26.

99) Herzog, S.M. & Hertwig, R. (2009). The wisdom of many in one mind: improving individual judgments with dialectical bootstrapping. Psychological Science, 20, 231-237.

100) Hibon, M. & Evgeniou, T. (2005). To combine or not to combine: selecting among forecasts and their combinations. International Journal of Forecasting, 21, 15-24.

101) Hora, S.C. (2004). Probability judgments for continuous quantities: Linear combinations and calibration. Management Science, 50(5), 597604.

102) Hosmer, D.W. & Lemeshow, S. (2000). Interpretation of the fitted logistic regression model.Applied Logistic Regression, Second Edition.

103) Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6), 417441.

104) Hu, X., Huang, H., Pan, Z. & Shi, J. (2019). Information asymmetry and credit rating: A quasi-natural experiment from China. Journal of Banking & Finance, 106, 132-152.

105) Huang, Zan, Chen, H., Hsu, C., Chen, W. & Wu, S. (2004). Credit rating analysis with support vector machines and neural networks: a market comparative study. Decision Support Systems, 37, 543-558.

106) Hwang, R-C., Chung, H. & Chu, C. (2010). Predicting issuer credit ratings using a semiparametric method. Journal of Empirical Finance,17(1),120-137.

107) Jaganathan, S., Prakash, P.K.S. (2020). A combination-based forecasting method for the M4-competition. International Journal of Forecasting, 36(1), 98-104.

108) James, G., Witten, D. & Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning with Application in R. SpringerVerlag New York.

109) Jewell, J. & Livingston, M. (1999).A comparison of bond ratings from Moody's S&P and Fitch IBCA. Financial Markets, Institutions & Instruments, 8(4), 1-45.

110) Jiao, Y., Syau, Y. & Lee, E.S. (2007).Modelling credit rating by fuzzy adaptive network.Mathematical and Computer Modelling, 45, 717-731.

111) Johnstone D.J. (2011). Economic interpretation of probabilities estimated by maximum likelihood or score. Management Science, 57, 308-314.

112) Jose, V.R.R. & Winkler, R.L. (2008). Simple robust averages of forecasts: some empirical results. International Journal of Forecasting, 24, 163-169.

113) Jose, V.R.R., Grushka-Cockayne, Y. & Lichtendahl Jr., K.C. (2014). Trimmed opinion pools and the crowd's calibration problem. Management Science, 60(2), 463-475.

114) Kaiser, H. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20,141-151.

115) Kamstra, M. & Kennedy, P. (1998). Combining qualitative forecasts using logit. International Journal of Forecasting, 14, 83-93.

116) Kaplan, R. & Urwitz, G. (1979). Statistical Models of Bond Ratings: A Methodological Inquiry. The Journal of Business, 52(2), 231-61.

117) Karas, A., Schoors, K. & Weill, L. (2010). Economics of Transition.BOFIT Discussion Papers 3/2008.

118) Karminsky A. M. & Kostrov A. (2017). The back side of banking in Russia: forecasting bank failures with negative capital. International Journal of Computational Economics and Econometrics, 7(1/2), 170-209.

119) Karminsky A. M. & Kostrov A., (2014). The probability of default in Russian banking. Eurasian Economic Review, 4, 81-98.

120) Karminsky A. M. & Khromova E. (2018). Increase of banks' credit risks forecasting power by the usage of the set of alternative models. Russian Journal of Economics 4(2): 155-174 (doi: 10.3897/j.ruje.4.27737)

121) Karminsky A. M. & Khromova E., (2016). Extended Modelling of Banks' Credit Ratings, Procedia Computer Science, (91), 201-210

122) Karminsky A. M. & Khromova E., (2016). Modelling banks' credit ratings of international agencies, Eurasian Economic Review, (6), 341363

123) Karminsky A., Khromova E., Kudrov R. (2021-exp). Empirical modeling of international banks' credit risk: assessment and comparison of credit ratings. Eurasian Economic Review. - in a publishing process

124) Karminsky, A. & Petrov, A. (2004). Ratings of dynamic financial stability for banks and enterprises, Analytical Banking Journal, 12, 74-78. (in Russian)

125) Karminsky, A. M. (2015). Credit ratings and their modeling. National Research University Higher School of Economics.-Moscow: HSE Publishing House. (in Russian)

126) Karminsky, A. M., Hainsworth, R. N. & Solodkov, V. M. (2013). Arm's Length Method for Comparing Rating Scales. Eurasian Economic Review, 3(2), 114-135.

127) Karminsky, A. M., Morgunov, A.V., Bogdanov, P.M. (2015). The Assessment of Default Probability for the Project Finance Transactions. Journal of the New Economic Association, 2(26), 99-122.

128) Karminsky, A.M. & Peresetsky, A.A. (2007). Models of ratings of international rating agencies.PrikladnaiaEkonometrika (Applied Econometrics), 1, 3-19. (in Russian)

129) Karminsky, A.M. & Sosurko, V.V. (2010). Features of modeling the international ratings of banks. Upravleniefinansovimiriskami (Financial risk management), 4, 292-305. (in Russian)

130) Karminsky, A.M. & Sosurko, V.V. (2011). The unified rating mapping: a step from the myth to reality. Bankovskoedelo (Banking subject), 6, 5863. (in Russian)

131) Kashyap, A. & Kovrijnykh, N. (2016). Who should pay for credit ratings and how? The Review of Financial Stidies, 29(2), 420-456.

132) Kaufmann, D., Kraay, A. & Mastruzzi, M. (2010). The Worldwide Governance Indicators. Policy Research working paper; no. WPS 5430. World Bank.

133) Khromova E. (2020). Dynamic Mapping of Probability of Default and Credit Ratings of Russian Banks. Journal of Corporate Finance Research.

14 (4): 31-46

134) Khromova E. (2020). Increase of Banks' Credit Risk Forecasting Power Using the Set of Credit Ratings and Probability of Default Models. Recent Advances of the Russian Operations Research Society 1st Edition: 177-196

135) Kim, K. S. (2005). Predicting bond ratings using publicly available information. Expert Systems with Applications, 29, 75-81.

136) Kisgen, D. (2006). Credit ratings and capital structure, Journal of Finance, 61, 1035-1072.

137) Ko, Y., Fujita, H. & Li, T. (2017). An evidential analysis of Altman Z-score for financial predictions: Case study on solar energy companies. Applied Soft Computing, 52, 748-759.

138) Kose, M., Otrok, C. & Whiteman, C. (2003). International Business Cycles: World, Region, and Country-Specific Factors. American Economic Review, 93 (4), 1216-1239.

139) Kourentzes, N. & Petropoulos, F. (2016). Forecasting with multivariate temporal aggregation: The case of promotional modelling. International Journal of Production Economics, 181, 145-153.

140) Kourentzes, N., Petropoulos, F. & Trapero, J.R. (2014). Improving forecasting by estimating time series structural components across

multiple frequencies. International Journal of Forecasting, 30(2), 291302.

141) Krüger, S. & Rösch, D. (2017). Downturn LGD modeling using quantile regression. Journal of Banking & Finance, 79, Pages 42-56.

142) Kumar, K. & Bhattacharya, S. (2006). Artificial neural network vs linear discriminant analysis in credit ratings forecast: a comparative study of prediction performances. Review of Accounting and Finance, 5, 216227.

143) Kvamme, H., Sellereite, N., Aas, K. & Sjursen, S. (2018). Predicting mortgage default using convolutional neural networks. Expert Systems with Applications, 102, 207-2017.

144) Kyoung-jae K. & Hyunchul A. (2012). A corporate credit rating model using multi-class support vector machines with an ordinal pairwise partitioning approach. Computers & OR, 39(8), 1800-1811.

145) La Porta, L. (2000). Government Ownership of Banks.The Journal of Finance.Vol.57, No.1, 265-301.

146) Lanine G., & Vennet, R. (2006). Failure prediction in the Russian bank sector with logit and trait recognition models. Expert Systems with Applications, 30 (3), 463-478.

147) Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science, 52(1), 111-127.

148) Le, H. & Viviani, J. (2018). Predicting bank failure: An improvement by implementing a machine-learning approach to classical financial ratios. Research in International Business and Finance, 44, 16-25.

149) Lee, Y. C. (2007). Application of support vector machines to corporate credit rating prediction. Expert Systems with Applications, 33, 67-74.

150) Leow, M. & Mues, C. (2012). Predicting loss given default (LGD) for residential mortgage loans: A two-stage model and empirical evidence for UK bank data. International Journal of Forecasting, 28(1), 183-195.

151) Lichtendahl Jr, K.C. & Winkler, R.L. (2020). Why do some combinations perform better than others? International Journal of Forecasting, 36(1), 142-149.

152) Lin, C-C. & Yang, S-L. (2016). Bank fundamentals, economic conditions, and bank failures in East Asian countries. Economic modelling, 52(B), 960-966.

153) Lin, T.-H.(2009). A cross model study of corporate financial distress prediction in Taiwan: Multiple discriminant analysis, logit, probit and neural networks models. Neurocomputing, 72, 3507-3516.

154) Liss H. & Fons J. (2006). Mapping Moody's national scale ratings to global scale ratings. Moody's Rating Methodology, December. [Link:http://www.moodys.com.br/brasil/pdf/Mapeamento_Dez_06e.pdf.]

155) Livingston M., Wei J. & Zhou L., (2010). Moody's and STANDARD & POOR'S Ratings: Are They Equivalent? Conservative Ratings and Split Rated Bond Yields.Journal of Money, Credit and Banking, 42 (7).

156) Lopez, J.A. (2001). Evaluating the predictive accuracy of volatility models. Journal of Forecasting, 20, 87-109.

157) Makridakis, S. & Hibon, M. (2000). The M3-competition: results, conclusions and implications. International Journal of Forecasting, 16(4), 451-476.

158) Makridakis, S. (1989). "Why Combining Works?," International Journal of Forecasting, 5, 601-603.

159) Mânnasoo, K. & Mayes, D. (2009). Explaining bank distress in Eastern European transition economies. Journal of Banking & Finance, 33, 244253.

160) Mannes, A.E., Soll, J.B. & Larrick, R.P. (2014). The wisdom of select crowds. Journal of Personality and Social Psychology, 107(2), 276-299.

161) Martin, D. (1977). Early warning of bank failure: A logit regression approach. Journal of Banking & Finance, 1(3), 249-276.

162) Micco, A., Panizza, U., & Yanez, M. (2007). Bank ownership and performance. Does politics matter? Journal of Banking & Finance, 31(1), 219-241.

163) Mizen, P. & Tsoukas, S. (2012). Forecasting US bond default ratings allowing for previous and initial state dependence in an ordered probit model. International Journal of Forecasting, 28 (1), 273-287.

164) Modigliani, F. & Miller, M. (1958). The Cost of Capital, Corporation Finance and the Theory of Investment. American Economic Review, 48 (3), 261-295.

165) Morgan, D.P. (2002) Rating banks: risk and uncertainty in an opaque industry, American Economic Review 92, 874-889.

166) Newbold, P. & Granger, C.W. (1974). Experience with forecasting univariate time series and the combination of forecasts. Journal of the Royal Statistical Society. Series A (General), 137, 131-165.

167) Offerman, T., Sonnemans, J., Van de Kuilen, G. & Wakker, P.P. (2009). A truth serum for non-Bayesians: correcting proper scoring rules for risk attitudes. Review of Economic Studies, 76, 1461-1489.

168) Ögüt, H., Doganay, M. M., Ceylan, N. B. & Akta§, R. (2012). Prediction of bank financial strength ratings: The case of Turkey. Economic Modelling, 29, 632-640.

169) Onyiriuba, L. (2016). Bank Risk Management in Developing Economies. 1st edition. Academic press.

170) Pacelli V. & Azzollini M. (2010). An Artificial Neural Network Approach for Credit Risk Management, Journal of Intelligent Learning Systems and Applications.

171) Pagratis, S. & Stringa, M. (2007). Modelling bank credit ratings: A structural approach to Moody's credit risk assessment. Working paper. Bank of England, London.

172) Palm, F.C. & Zellner, A. (1992). To Combine or not to Combine? Issues of Combining Forecasts. Journal of Forecasting, 11, 687-701.

173) Park, G. & Lee, H. (2018). Opportunistic behaviors of credit rating agencies and bond issuers. Pacific-Basin Finance Journal,47, 39-59.

174) Pasiouras, F., Gaganis, C. & Doumpos, M. (2007). A multicriteria discrimination approach for the credit rating of Asian banks. Annals of Finance,3(3),351-367.

175) Pauwels, L. & Vasnev, A.L. (2011). Forecast combination for discrete choice models: predicting FOMC monetary policy decisions.Working paper: University of Sydney.

176) Pauwels, L.L. & Vasnev, A.L. (2012). Forecast Combination for Discrete Choice Models: Predicting FOMC Monetary Policy Decisions. Available at SSRN: https://ssrn.com/abstract=2203549 or http://dx.doi.org/10.2139/ssrn.2203549.

177) Pauwels, L.L. & Vasnev, A.L. (2014). Forecast combination for US recessions with real-time data. The North American Journal of Economics and Finance, 28, 138-148.

178) Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2, 559-572.

179) Peresetsky, A. (2009). Measuring the external support components of Moody's ratings. Applied econometrics, 2, 3-23. (in Russian)

180) Peresetsky, A. (2012). Econometric approach to off-site analysis of Russian banks. Publ. House of Higher School of Economics.-235,[5]p. (in Russian)

181) Peresetsky, A., Karminsky, A. & Golovan, S. (2004). Probability of default models of Russian banks.Bank of Finland BOFIT Discussion paper 21/2004.

182) Peresetsky, A.A. & Karminsky, A.M. (2008). Models for Moody's Bank Ratings.Bank of Finland, BOFIT Discussion Papers, 17.

183) Peresetsky, A.A. & Karminsky, A.M. (2011). Models for Moody's Bank Ratings. Frontiers in Finance and Economics, 8(1), 88-110.

184) Pesaran, M.H., Timmermann, A. (2007). Selection of estimation window in the presence of breaks. Journal of Econometrics, 137, 134-161.

185) Petropoulos, A., Siakoulis, V., Stavroulakis, E. & Vlachogiannakis, N. (2020). Predicting bank insolvencies using machine learning techniques. International Journal of Forecasting, In Press.

https://doi. org/10.1016/j.ijforecast.2019.11.005.

186) Petropoulos, F. & Svetunkov, I. (2020). A simple combination of univariate models. International Journal of Forecasting, 36(1), 110-115.

187) Pomasanov, M. & Hamalinsky, A. (2012). Rating Model Calibration for Sectors with a Low Number of Defaults. Financial risk management, 02(30), 82-94.

188) Pomasanov, M. & Vlasov, A. (2008).Calibration of national rating systems.Rynokcennichbumag (Security market), 74-79 (in Russian)

189) Pompella, M., & Dicanio, A. (2017). Ratings based Inference and credit risk: Detecting likely- to-fail banks with the PC-Mahalanobis method. Economic Modelling, 67, 34-44.

190) Rosch, D., & Scheule, H. (2014). Forecasting probabilities of default and loss rates given default in the presence of selection. Journal of the Operational Research Society, 65 (3), 393-407.

191) Rudakova, O. & Ipatyev, K. (2015). Some Approaches to the Calibration of Internal Rating Models. Review of European Studies, 7(10).

192) Ryan, J. (2012). The negative impact of credit rating agencies and proposal for better regulation, SWP, Berlin. Working paper FG 1, 2012/Nr.01.

193) Sahut, J-M. & Mili, M. (2011). Determinants of Banking Distress and Merger as Strategic Policy to Resolve Distress. Economic modelling, 28(1/2), 138-146.

194) Santoni, A. & Arbia, B. (2013). Assessing Rating Agencies' Ability to Predict Bank Bankruptcy - The Lace Financial Case. Economic notes, 42(2), 203-2015.

195) Schaeck, K. & Cihak, M. (2012).Banking Competition and Capital Ratios.European Financial Management. 18(5), 836-866.

196) Schuermann, T. & Hanson, S. (2004). Estimating Probabilitites of Default. Federal Reserve Bank of New York Staff Reports, 190.

197) Shaub, D. (2020). Fast and accurate yearly time series forecasting with forecast combinations. International Journal of Forecasting, 36(1), 116120.

198) Shi, B., Chi, G. & Li, W. (2020). Exploring the mismatch between credit ratings and loss-given-default: A credit risk approach. Economic Modelling, 85, 420-428.

199) Shi, B., Zhao, X., Wu, B. & Dong, Y. (2019). Credit rating and microfinance lending decisions based on loss given default (LGD). Finance Research Letters, 30, 124-129.

200) Shimizu, Y., Lee, J., & Takei, H. (2013). Analysis of Determinants of Split Ratings and Rating Conservativeness between Japanese and US Credit Rating Agencies. International Journal Of Accounting And Financial Reporting, 3(1), 182-188.

201) Skreta V., Veldkamp L. (2009). Ratings shopping and asset complexity: A theory of ratings inflation. Journal of Monetary Economics, Elsevier, Vol. 56(5). PP.678-695.

202) Smith, J. & Wallis, K.F. (2009). A simple explanation of the forecast combination puzzle. Oxford Bulletin of Economics and Statistics, 71, 331-355.

203) Snell, E. (1964). A Scaling Procedure for Ordered Categorical Data. Biometrics, 20(3),592-607.

204) Soll, J.B. & Larrick, R.P. (2009). Strategies for revising judgment: how (and how well) people use others' opinions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 780-805.

205) Solovjova, I. (2016). New approaches to regulating the activities of rating agencies: A comparative analysis. Procedia—Social and Behavioral Sciences, 229, 115-125.

206) Spuchl'akova, E. & Cug, J. (2015). Credit Risk and LGD Modelling. Procedia Economics and Finance, 23, 439-444.

207) Steshkin, A. (2015). Methods of Rating System Validation in the Framework of the Internal Ratings Approach to Assessing Banks ' Credit Risk. Financial analytics: Science and Experience, 32, 29-41.

208) Stock, J.H. & Watson, M.W. (1993). A simple estimator of cointegrating vectors in higher order integrated systems. Econometrica, 61(4), 783-820.

209) Stock, J.H. & Watson, M.W. (2003). Combination Forecasts of Output Growth in a Seven-Country Data Set. Forthcoming in Journal of Forecasting.

210) Stock, J.H. & Watson, M.W. (2004). Combination forecasts of output growth in a seven-country data set. Journal of Forecasting, 23, 405-430.

211) Stock, J.H., Watson, M.W. (1999). A comparison of Linear and Nonlinear Univariate Models for Forecasting Macroeconomic Time Series. In: Engle R, White H Cointegration, Causality and Forecasting: A Festschrift for Clive W.J. Granger. Oxford: Oxford University Press, 144.

212) Tanoue, Y., Kawada, A. & Yamashita, S. (2017). Forecasting loss given default of bank loans with multi-stage model. International Journal of Forecasting, 33(2), 513-522.

213) Tasche, D. (2013). The art of probability-of-default curve calibration. Journal of Credit Risk, 9(4), 63-103.

214) Teker, D., Pala, A. & Kent, O. (2013). Determination of Sovereign Rating: Factor Based Ordered Probit Models for Panel Data Analysis Modelling Framework. International Journal of Economics and Financial Issues, 3(1), 122-132.

215) Thomson, J. (1991). Predicting Bank Failures in the 1980s. Economic Review, Federal Reserve Bank of Cleveland, 27(1),9-20.

216) Timmermann, A. (2006). Forecast combinations. Handbook of economic forecasting, 1, 135-196.

217) Topaloglou, N. (2015). Minimizing bank liquidity risk: evidence from the Lehman crisis. Eurasian Business Review, 5, 23-44.

218) Vasilyuk, A.A. & Karminsky, A.M. (2011). Modeling credit ratings of domestic banks based on Russian accounting standards. Upravleniefinansovimiriskami (Financial risk management), 3, 194-205.

219) Vasnev, A.L., Skirtun, M. & Pauwels, L. (2013). Forecasting monetary policy decisions in Australia: A forecast combinations approach. Journal of Forecasting, 32(2), 151-166.

220) Vernikov, A. & Bobkov, V. (2008). Quality of governance and bank valuation in Russia: an empirical study. Journal of Corporate Finance. 416.

221) Vernikov, A. (2015). Russian bank data: Breaking down the sample of banks by ownership. SSRN Working Paper series No. 2600738, 1-21

222) Volk, M. (2013). Estimating Probability of Default and Comparing It to Credit Rating Classification by Banks. Economic and Business Review, 14(4).

223) Vul, E. & Pashler, H. (2008). Measuring the crowd within: probabilistic representations within individuals. Psychological Science, 19, 645-647.

224) Westgaard, S. & Wijst, N. (2001). Default probabilities in a corporate bank portfolio: A logistic model approach, 135(2), 338-349.

225) White L. (2013). Credit rating agencies: An overview. Annual Review of Financial Economics N5. PP. 93-122.

226) Winkler, R.L. & Clemen, R.T. (1992). Sensitivity of weights in combining forecasts. Operations Research, 40(3), 609-614.

227) Winkler, R.L. & Clemen, R.T. (2004). Multiple experts vs. multiple methods: combining correlation assessments. Decision Analysis, 1, 167176.

228) Winkler, R.L. (1989). Combining Forecasts: A Philosophical Basis and Some Current Issues, International Journal of Forecasting, 5, 605-609.

229) Winkler, R.L. and Makridakis, S. (1983). The Combination of Forecasts, Journal of the Royal Statistical Society Series A, 146, 150-57.

230) Winkler, R.L., Grushka-Cockayne, Y., Lichtendahl Jr., K.C. & Jose, V.R.R. (2019). Probability forecasts and their combination: A research perspective. In Decision Analysis, (in press).

231) Wu, T.-C. & Hsu, M.-F.(2012). Credit risk assessment and decision making by a fusion approach. Knowledge-Based Systems, 35, 102-110.

232) Xuefeng J., Stanford M.H., Xie Y. (2012). Does it matter who pays for bond ratings? Historical evidence. Journal of Financial Economics N 105. PP. 607-621.

233) Yeddou, N. & Pourroy, M. (2020). Bank liquidity creation: Does ownership structure matter? The Quarterly Review of Economics and Finance. Available 7 June, 2020 at

[https://proxylibrary.hse.ru:2055/science/article/pii/S106297692030003X ]

234) Zan, H., Chen, H., Hsu, C.-J., Chen, W.-H. & Wu, S. (2004). Credit rating analysis with support vector machines and neural networks: A market comparative study. Decision Support Systems, 37, 543-558.

235) Zellner, A. (1986). Biased Predictors, rationality and the evaluation of forecasts. Economics Letters 21, 45-48.

236) Zhang, J., He, L. & An, Y. (2020). Measuring banks' liquidity risk: An option-pricing approach. Journal of Banking & Finance,111, 105703.

237) Zhivaikina, A. & Peresetsky, A. (2017). Credit ratings of Russian banks and reviews of banking licenses 2012-2016 yy. Journal of the New economic Association, 4(36), 49-80.

238) Banking Account and Ratio Definitions. (2011). Moody's Investors Service, February 2011.

[Link:https://www.moodys.com/sites/products/ProductAttachments/Bank ing%20Account%20and%20Ratio%20Definitions.pdf]

239) Banki.ru website. The database of Russian defaulted banks. [L ink: http: //www. banki. ru/banks/memory/?PAGEN_1=3]

240) Bankodrom website. The list of government-owned banks [Link:http://www.bankodrom.ru/spisok-gosudarstvennyh-bankov-rossii/]

241) Bureau van Dijk website [Link: https://www.bvdinfo.com/en-gb/home]

242) Cbonds website. The historical ratings of Russian banks [Link: http: //ru. cbonds. info/ratings/]

243) CBR website. Default database. [Link:

http: //cbr.ru/credit/likvidbase/LikvidBase. aspx]

244) Incorporation of Joint-Default Analysis into Moody's Bank Ratings:A Refined Methodology. (2007). Moody's Investors Service. March 2007. [Link:https://notendur.hi.is/ajonsson/kennsla2006/JDA%20revised%20m ethofology%20-%2030%20March%202007.pdf]

245) International Convergence of Capital Measurement and Capital Standards.A revised framework. (2005). BIS, Basel Committee on Banking Supervision.[ Link: http://www.bis.org/publ/bcbs118.pdf].

246) Macroeconomic database [Link: http://data.worldbank.org/country]