Comment on the Calder Center Florida Study: Competitive Effects of Means Tested School Vouchers Tested

posted Jan 25, 2012, 7:30 AM by Bill Duncan   [ updated Feb 4, 2012, 5:54 AM ]
Summary

Link to the study (and attached )

Voucher supporters used to assert that students got a better education in private schools but the data showed that that did not happen.  So advocates have turned to this new argument - the benefits will be in the impact of competition on the public schools.  This is  ideologically attractive but it has not turned out to be true either.

In support of the competition argument, advocates cite "the study done by Florida's own Department of Education that proves that competition from an education tax credit program like the one proposed for New Hampshire improves the public schools."

There is no Florida Department of Education study of the competitive effects of Florida's Education Tax Credit program on Florida public schools.  The study advocates are referring to does record modest improvements in the public schools in the year between when vouchers were announced in Florida and when the program actually started.  But a reviewer calls those effects "tiny" and the author says that his results would not apply to New Hampshire.  Here is a useful Florida newspaper report on the study.

There is no study from any source that demonstrates that competition from voucher or education tax credit programs materially improves the public schools.

-----------------------------------------------------
Detail

The study itself (two versions are attached below) is a Calder Center "working paper" meaning that it has not been peer reviewed.  Our comments on it are below.

While the paper was written recently, it uses old data.  It seems to be a replication of a study done over 10 years ago by Jay Greene, a well known voucher advocate.  Greene's conclusion that public schools improved in response to voucher competition was discredited when Duke University replicated the Greene study in North Carolina.  Here is the Duke University policy brief based on that study.  

The authors caution that their study may have very little relevance to any other state.  Even a quick comparison makes clear that it is not relevant to New Hampshire.

For the period studied, the Florida program was different from the proposed New Hampshire program in many ways.  

Florida had large scholarships targeted to very poor children coming strictly from public schools rated "F" by the state of Florida for two years in a row.  The scholarship was automatically renewed for a long as the student was in school.  Average family incomes were very low.  Even now, family incomes average only $24,000.  Participating schools were academically accountable.  Florida's schools were ranked among the lowest in the nation.

New Hampshire's proposed program has small scholarship not targeted to low income families (that might change to a modest level of targeting but incomes will be higher than in Florida) coming from any private or public school.  New Hampshire has no school rating system.  The scholarship will not be automatically renewed.  Participating schools would not be academically accountable.  New Hampshire's schools are ranked among the highest in the nation.

Both programs are funded by tax credits but beyond that, they have little in common.  Florida is a mostly urban state 14 times the size of New Hampshire, with a large school system known for poor performance.  

The paper itself analyses the year between the announcement of the voucher program and when the first voucher students leave public schools to attend private schools. Most importantly, the study did not address the impact of other concurrent policies on those schools.  Duke University showed that receiving an "F" grade from the state's school rating system is really what led to improvement in the schools, whether or not a voucher system existed.

The authors did not interview the public school administrators and teachers involved.  The benefit of face-to-face interviews is illustrated  here, with the Indiana school principle who is losing kids to the Indiana voucher program. It is clear that he is on the defensive from the losses, not on the verge of academic gains in response to the new competitive pressures.

The author himself makes the case against trying to apply his conclusions elsewhere:

It is possible that the dynamics between competitive pressures
and public-school students’ test scores are systematically
different in Florida than they are in the rest of the nation. In
particular, more than 90 percent of Florida’s students live in
the state’s top 20 most populous metropolitan areas. In states
with a greater share of the population in rural areas, the effects
of a scholarship program may not exert the same degree of
competitive pressure on public schools. It may also be the
case that Florida’s diverse range of private school options and
accompanying greater competition among the private schools
limit the study’s generalizability.



---------------------
Notes:
  • The Florida program studied is entirely different from the proposed NH program
    • one of the largest in the US in terms of the number of low-income students served
    • 185% of federal poverty
    • student must have spent the entire previous year in a public school to qualify for a voucher
    • the voucher was up to $4,100, about 90% of the tuition and fees at a typical Florida religious private school
    • "The policy change was very large for the target population - the number of scholarships available was larger than half the size of the low income population using private schools on the eve of the policy's introduction.  For low-income Floridians, the voucher program represented a major demand shock for private schooling."
    • The Florida student will get follow-on voucher grants.  Most New Hampshire students will not.
    • Excludes home schooling
    • Requires 90% utilization to grow, rather than 80%
  • The schools lose about $6,600 per departing student and state government assumes that, because incomes are so low, 90% of the students would not have gone to private schools without the voucher.  Family incomes of voucher recipients average.less than $25,000
  • This is a "new" paper but is uses old data, replicating the Jay Greene study from 10 years ago.
  • The authors' thesis rests on a close proximity of the competing private schools to the public schools affected.  It's an odd thesis to being with.  In most cases, there would not be a comparable density of private schools in New Hampshire.  .
  • The authors attempt to control for so many complex interacting factors that it is hard to trust the results.
  • The paper does not establish cause and effect, just correlation of some very subtle and derivative variables.  That's why it gets caught flat footed on the A-F rating issue.
  • Florida's schools rank far lower than New Hampshire's.  Based on 4th grade math in 2011, for instance, New Hampshire was second in the country behind Massachusetts and Florida was 31st.
  • The authors do not account for the other concurrent policy interventions that could have contributed to the changes.  
    • For instance, (SP.8) the state of Florida's school accountability system gave schools grades.  Repeated "Fs" triggered voucher eligibility (and, probably, other policy interventions unmentioned in this paper).  Studies cited find that the F grade itself leads to "improved student outcomes."  This is said to have affected other studies by not this one (SP 9.)
  • They do not talk with administrators - just run the statistics from a distance.
  • They measure the effect in the year before the students left public school for private school, so the premise is that these competitive effects took place immediately upon announcement of the voucher program. (p.12 of pdf, p.4 of study)
  • Study page 6 lists all the studies that found that vouchers had little or no competitive impact.
  • Study page 7: why the Milwaukee study is not reliable.
  • In the Calder report, Figlio said his report was valid because he tested only the first year  after the program announcement, before any students had left.  In the follow-up report, he analyses all years.


“Direct certification list” means the certified list of children who qualify for the food assistance program, the Temporary Assistance to Needy Families Program, or the Food Distribution Program on Indian Reservations provided to the Department of Education by the Department of Children and Family Services.

(j) “Unweighted FTE funding amount” means the statewide average total funds per unweighted full-time equivalent funding amount that is incorporated by reference in the General Appropriations Act, or any subsequent special appropriations act, for the applicable state fiscal year.

1. A student is eligible for a Florida tax credit scholarship under this section if the student qualifies for free or reduced-price school lunches under the National School Lunch Act or is on the direct certification list and:
a. Was counted as a full-time equivalent student during the previous state fiscal year for purposes of state per-student funding;
b. Received a scholarship from an eligible nonprofit scholarship-funding organization or from the State of Florida during the previous school year;
c. Is eligible to enter kindergarten or first grade; or
d. Is currently placed, or during the previous state fiscal year was placed, in foster care as defined in s. 39.01.
2. A student may continue in the scholarship program as long as the student’s household income level does not exceed 230 percent of the federal poverty level.
3. A sibling of a student who is continuing in the scholarship program and who resides in the same household as the student shall also be eligible as a first-time tax credit scholarship recipient if the sibling meets one or more of the criteria specified in subparagraph 1. and as long as the student’s and sibling’s household income level does not exceed 230 percent of the federal poverty level.

.....
A scholarship organization must:
(d) Must provide scholarships, from eligible contributions, to eligible students for the cost of:
1. Tuition and fees for an eligible private school; or
2. Transportation to a Florida public school that is located outside the district in which the student resides or to a lab school as defined in s. 1002.32.
(e) Must give priority to eligible students who received a scholarship from an eligible nonprofit scholarship-funding organization or from the State of Florida during the previous school year.
(f) Must provide a scholarship to an eligible student on a first-come, first-served basis unless the student qualifies for priority pursuant to paragraph (e).
(g) May not restrict or reserve scholarships for use at a particular private school or provide scholarships to a child of an owner or operator.
(h) Must allow an eligible student to attend any eligible private school and must allow a parent to transfer a scholarship during a school year to any other eligible private school of the parent’s choice.

.........

(e) The parent shall ensure that the student participating in the scholarship program takes the norm-referenced assessment offered by the private school. The parent may also choose to have the student participate in the statewide assessments pursuant to s. 1008.22. If the parent requests that the student participating in the scholarship program take statewide assessments pursuant to s.1008.22, the parent is responsible for transporting the student to the assessment site designated by the school district.

..............

(8) PRIVATE SCHOOL ELIGIBILITY AND OBLIGATIONS.An eligible private school may be sectarian or nonsectarian and must:
.........

(c) Be academically accountable to the parent for meeting the educational needs of the student by:
1. At a minimum, annually providing to the parent a written explanation of the student’s progress.
2. Annually administering or making provision for students participating in the scholarship program in grades 3 through 10 to take one of the nationally norm-referenced tests identified by the Department of Education. Students with disabilities for whom standardized testing is not appropriate are exempt from this requirement. A participating private school must report a student’s scores to the parent and to the independent research organization selected by the Department of Education as described in paragraph (9)(j).
3. Cooperating with the scholarship student whose parent chooses to have the student participate in the statewide assessments pursuant to s. 1008.22.
(d) Employ or contract with teachers who have regular and direct contact with each student receiving a scholarship under this section at the school’s physical location.
...............

(9) DEPARTMENT OF EDUCATION OBLIGATIONS.The Department of Education shall:
.............

(i) Maintain a list of nationally norm-referenced tests identified for purposes of satisfying the testing requirement in subparagraph (8)(c)2. The tests must meet industry standards of quality in accordance with State Board of Education rule.
(j) Select an independent research organization, which may be a public or private entity or university, to which participating private schools must report the scores of participating students on the nationally norm-referenced tests administered by the private school in grades 3 through 10.
1. The independent research organization must annually report to the Department of Education on the year-to-year learning gains of participating students:
a. On a statewide basis. The report shall also include, to the extent possible, a comparison of these learning gains to the statewide learning gains of public school students with socioeconomic backgrounds similar to those of students participating in the scholarship program. To minimize costs and reduce time required for the independent research organization’s analysis and evaluation, the Department of Education shall conduct analyses of matched students from public school assessment data and calculate control group learning gains using an agreed-upon methodology outlined in the contract with the independent research organization; and
b. According to each participating private school in which there are at least 30 participating students who have scores for tests administered during or after the 2009-2010 school year for 2 consecutive years at that private school.
2. The sharing and reporting of student learning gain data under this paragraph must be in accordance with requirements of 20 U.S.C. s. 1232g, the Family Educational Rights and Privacy Act, and shall be for the sole purpose of creating the annual report required by subparagraph 1. All parties must preserve the confidentiality of such information as required by law. The annual report must not disaggregate data to a level that will identify individual participating schools, except as required under sub-subparagraph 1.b., or disclose the academic level of individual students.
....
(n)1. Conduct random site visits to private schools participating in the Florida Tax Credit Scholarship Program. The purpose of the site visits is solely to verify the information reported by the schools concerning the enrollment and attendance of students, the credentials of teachers, background screening of teachers, and teachers’ fingerprinting results. The Department of Education may not make more than seven random site visits each year and may not make more than one random site visit each year to the same private school.

......

(12) SCHOLARSHIP AMOUNT AND PAYMENT.
(a)1. Except as provided in subparagraph 2., the amount of a scholarship provided to any student for any single school year by an eligible nonprofit scholarship-funding organization from eligible contributions shall be for total costs authorized under paragraph (6)(d), not to exceed annual limits, which shall be determined as follows:
a. For a scholarship awarded to a student enrolled in an eligible private school:
(I) For the 2009-2010 state fiscal year, the limit shall be $3,950.
(II) For the 2010-2011 state fiscal year, the limit shall be 60 percent of the unweighted FTE funding amount for that year. [DNHPE NOTE: Florida's state aid per student is approximately $6,600 in 2012]
(III) For the 2011-2012 state fiscal year and thereafter, the limit shall be determined by multiplying the unweighted FTE funding amount in that state fiscal year by the percentage used to determine the limit in the prior state fiscal year. However, in each state fiscal year that the tax credit cap amount increases pursuant to subparagraph (5)(a)2., the prior year percentage shall be increased by 4 percentage points and the increased percentage shall be used to determine the limit for that state fiscal year. If the percentage so calculated reaches 80 percent in a state fiscal year, no further increase in the percentage is allowed and the limit shall be 80 percent of the unweighted FTE funding amount for that state fiscal year and thereafter.
b. For a scholarship awarded to a student enrolled in a Florida public school that is located outside the district in which the student resides or in a lab school as defined in s. 1002.32, the limit shall be $500.
2. The annual limit for a scholarship under sub-subparagraph 1.a. shall be reduced by:
a. Twenty-five percent if the student’s household income level is equal to or greater than 200 percent, but less than 215 percent, of the federal poverty level.
b. Fifty percent if the student’s household income level is equal to or greater than 215 percent, but equal to or less than 230 percent, of the federal poverty level.

Ċ
Bill Duncan,
Jan 30, 2012, 5:23 AM
Ċ
Bill Duncan,
Jan 25, 2012, 7:30 AM