We estimated two alternative models of the government's first stage acceptance
decision, i.e., the first government decision node in Figure 4.1: (a) a binary logit
model, and (b) a ``marginal probability''
model.
The marginal probability model is
similar to models estimated in Lahiri et al. (1995) and Hu et al. (1997)
which compute the probability of being accepted or denied benefits at each stage
of the five-stage sequential DDS process outlined in Figure 2.1. Unlike these
previous papers, we do not have access to administrative data that would allow
us to observe the stage of this process at which an applicant was accepted or
denied. Hence, we compute the marginal probability of acceptance or denial for
all stages simultaneously. Due to the similarity of the last two stages of
Figure 2.1 we combined them into stage 4 of the estimated model below.
Let be vectors representing information used to
determine whether an application is accepted or denied at stages 
of
the sequential process, respectively, and let
be conformable parameter vectors. Let
be the conditional probability of being denied at
the first stage, i.e., as having evidence of ability to engage in significant gainful
activity (SGA). Let 
be the conditional probability of
being ``passed on'' to the third stage, i.e., as having a severe impairment.
Let 
be the probability of being accepted at the third
stage, i.e., having a listed impairment. As explained above, in the fourth, and last
stage, 
denotes the probability of being determined as
incapable of performing past or other work. Thus, the marginal probability of
being awarded benefits is given by
The maximum likelihood estimates for the two models are given in
Table 4.3. Note that in the marginal probability model no variable appears in
more than one step. The allocation of the variables across the four steps was
done according to
the SSA classification of the variables which are used in their eligibility
determination. This is also in line with the classification used by Lahiri et
al. (1995). Evidently, some of the most obvious variables do
not seem to be good predictors of the approval probability under either model. In
the set of variables listed under the first step, most variables are insignificant,
although, in general, have the expected sign. Among all variables, the average
number of hours worked per week in the four months following the application
for benefits seems to be the most important. The earnings and wealth variables,
which are the key ingredients in determining one's ability to engage in SGA, and
meeting the income and asset eligibility thresholds for SSI, seem to be very poor
predictors of the approval probability. This may reflect self-selection on the part
of individuals since very few people apply for SSI if they know they exceed the
income and asset thresholds or earn in access of the SGA limit. The set of
variables listed under the second step seems to contain more information about
one's chances of getting benefits. In particular, the self-reported health variable
7 and the number of
hospitalizations in the year prior to the application (variable 8) have
strong effects on the award probability, with robust positive coefficients in all
the alternative specifications we tried.
The other variables included seem to have low predictive power, or even have the ``wrong'' sign such as functional limitation indicators 10 through 12. The results indicate that only the more serious problems increase one's award probability. The same results also apply for the group of variables listed under the third step. Some of the variables such as having back problems have counterintuitive signs. One possible explanation might be that SSA may have found that many applicants use false excuses such as ``back problems'', so this variable might actually be a signal that a given applicant is an impostor. Reassuringly, we find that the more serious health problems such as cancer or stroke significantly increase the probability of being awarded benefits.
There are a number of variables that seem to be good predictors of the fourth stage of the DDS's determination procedure, which evaluates a candidate's capability to engage in past work or other work. The age at application (variables 23 and 24) significantly increases the odds of acceptance, reflecting the fact that disabled older individuals have fewer alternative job opportunities and capacity to engage in their past work. On the other hand, vocational training and a college degree have negative, although statistically insignificant, effects on the approval probability. The negative coefficient can be interpreted as an indication that the DDS regards more highly educated individuals as having higher levels of human capital and higher capacity to do alternative kinds of work than those with lower education levels. Being married or divorced reduces the probability of acceptance relative to those who were never married, possibly reflecting the fact that the SSA regards these people as likely to have other means of support within their extended family units.
{
Table 4.3: First Stage Decision by the Disability
Determination Services
}
The year dummies in Table 4.3 were included to capture policy changes is SSA's disability award policy over the period of our sample. Note that there is tremendous variation in the magnitude and significance of these dummy variables under both specifications. Since the SSA procedures and rules have not changed dramatically during the period from 1990 to 1995, the variation in these coefficients may be spurious, capturing chance variation in acceptance rates for cohorts of applicants in various years. For example a relatively small number of individuals were observed to apply in 1995 and by chance most of these applications happened to be accepted. This seems to be plausible explanation for the large positive estimate of variable 36 since we have no independent evidence of a sudden increase in DDS leniency in awarding benefits in 1995.
The log-likelihood values indicate that the binary logit specification fits the data slightly better than the supposedly ``correctly specified'' marginal probability model. Note that since both models have the same number of parameters, any model selection criteria (such as the Bayesian Information criterion or the Akaike Information criterion) reduce to comparison of the likelihood values for the two models. On that basis the logit model should be chosen over the marginal likelihood model. However, Cox's (1961,1962) nonnested hypothesis test rejects the null hypothesis that the marginal likelihood is the true model in favor of the logit alternative only at the 5% significance level.
In order to compare the predicted acceptance probabilities under the two specifications, Figure 4.3 plots the estimated acceptance probabilities for each observation, sorted from low to high. We note that for most of the observations the predicted probabilities of the logit and the marginal probability model are reasonably close. The marginal probability model predicts slightly higher probabilities of acceptance for the 75% of the observations with the lowest probabilities of acceptance. However, the marginal probability model predicts much lower probabilities of acceptance for most of the remaining 25% of observations with the highest probability of acceptance. We do not have a clear idea which model provides a better approximation to the ``truth'' in this case. It is interesting to note that both specifications predict a very wide range of predicted acceptance probabilities, ranging from 10% to nearly 100% over our sample. Approximately 50% of the observations have a predicted acceptance probability of less than 50%, and the average acceptance probability is about 50% in both specifications.
{
}
The remaining four curves in Figure 4.3 plot the estimated probabilities
of the four stages in the DDS's sequential disability
determination process. Among these probabilities , the probability of
satisfying the SGA test, is the highest, indicating that few applicants are rejected
at this stage. It is probably an indication that most DI applicants are sufficiently
familiar with the DDS screening procedure that very few make the mistake of
engaging in SGA subsequent to their claimed date of disability onset. Our
estimate of 
, the probability of being judged to have a severe impairment,
is significantly less than for most observations, indicating that many more
applications are denied at this stage. Around 50% of the observations are
predicted to have a stage 2 rejection probability of at least 20%. The estimate of
, the probability of having a listing impairment, is the lowest. Even if an
individual is being determined to have severe impairment there is still quite a high
probability that individuals will not be awarded benefits due to the fact that it
may not be a sufficiently severe disability to be included as one of those that
constitute the listed impairments in the SSA's ``blue book''. The final probability
, capacity for past/other work, lies between and for all
observations. This curve indicates that around 50% of the applicants who are
determined not to have a listing impairment are judged to have a capacity to do
either their past work or some other type of work, with probability of at least
50%. Overall, the estimated probabilities are intuitively plausible, reflecting
SSA's stated goal of designing the DDS screening process in stages so as to
weed out the ``easy'' cases first so that scarce administrative resources can be
focused on judging the more difficult or ambiguous cases that reach the last
stage of the sequential decision process.
The predicted probabilities from our marginal likelihood model are quite comparable to those obtained by Lahiri, et al. (1995). They report that the approval rate at the very first step was very close to 100% (which is the reason they estimated only the last four stages). The average approval rate estimated by our model is 97.2%. For the second stage the average approval rate based on our model is 74.7%, while that obtained by Lahiri et al. is 81.9%. The approval rates for the third stage are 33.9% and 35.7%, respectively. The average approval rates for the last two stages (aggregated in our model) are estimated to be 49.2% and 32.9%, respectively. The average acceptance probability for the full DDS determination process was estimated to be 46.5% in the Lahiri et al. study, quite close to our estimate of 49.3%. Given that the sample used here is quite different from that used by Lahiri et al., and given the sample variability in the approval rate estimates, the two sets of estimates are surprisingly close. This is very encouraging, and indicates that even without administrative data on the exact stage at which an applicant was qualified or disqualified we can still accurately estimate the overall acceptance probability, contrary to the claims of Lahiri et al. (1995).
Table 4.4 presents the results for the government's appeal decision. In principle, the same variables affecting the initial decision are those based on which the appeal decision is being made. Some additional information may be provided, though, by the individual during the appeal process. However, given the relatively smaller number of observations on appeals, the number of variables included in the estimation of the second stage decision is somewhat smaller than the number of variables included in the first stage. In particular, there are some variables for which there is not enough variation in the data to be able to get any useful results. Furthermore, in the later stages of the process the SSA simply decides whether or not to accept the appeal, but does not have to go through the multi-step procedure as in the first stage decision. Therefore, for this stage we only estimate a binary logit model.
{
Table 4.4: Second Stage Award decision for Appealled Cases
}
As can be seen from Table 4.4, the variables which are the most important for predicting the initial DDS decision are also important for predicting the appeal decision with the exception that the vocational training dummy emerges as the strongest predictor (next to the time dummies) and has a counterintuitive positive sign. Among the health variables, the most important predictor is HLIMPW (variable 6). Difficulty in walking around the room, sitting, and reading a map also have strong prediction power. Other variables, such as the age at application and the variables providing information on recent employment history seem to have only limited power in explaining the probability of acceptance. As previously noted, the wide variation in the year dummies is probably a reflection of overfitting with a small number of observations rather than an indication of substantial year to year variations in SSA's stringency in judging appeals.