Roland Fryer: Are high-quality schools enough to close the achievement gap?

[ Silence ]
^M00:00:21
>> Good afternoon.  Welcome to a [noise]CLOSUP Ford School speaker series from the Education Policy Lecture Series we have here.  Today is my privilege and pleasure to introduce Roland Fryer who is a colleague and a friend, someone who I knew way back when he was less famous, but I'm glad he has agreed to come and talk about some really interesting work he's doing on the Harlem Children's Zone.  Before I forget, I'd like to thank some of the folks that helped organize this.  First thanks to the Ford School for cosponsoring this.  Thanks to Beth Rebar [phonetic], Tom Ivacko, Jill and Katie in the Communications Office for helping put all to get--put together this.  And to Bonnie Roberts as well.  And so, Roland is an economist, the--as the Robert Beren, Professor of Economics and CEO of the Education Innovation Laboratory at the Harvard University.  He's had original training and work.  For those of you who may know him, was kind of more an economic theory which has lots of fun Greek letters and mathematical expressions.  He has moved to some extent of completely overtime the Applied Microeconomics side, looking at a host of interesting education policy programs.  He--in prior lives, he served as the Chief Equality Officer at the New York City Department of Education in '07-'08 and he has a number of awards that are listed here.  You can read including the recipient of Presidential Early Career Award for Scientists and Engineers, the 2009 Time 100--Time Magazine's annual list of the world's most influential people, and most importantly for our case here, Roland is a incredibly bright, an articulate and engaging speakers so I'll forever more enjoy the talk.  The standard is that Roland you can decide how many questions to accept as you go.  I think we--it goes to 5:30 so maybe if you'd leave a few minutes at the end for questions.  And with that, I'll let go and take it away.
>> All right, thank you.
^M00:02:46
[ Applause ]
^M00:02:54
Brian is just being modest.  When I first got in education research, I asked 1 of my advisers I said, "You know, I think about going to education.  I think it's interesting."  He says, "Yeah, you really ought to be like Brian Jacob if you want to be good at this."  [Laughter] So it is so good to be here.  Thanks for coming out.  I'm incredibly excited today to talk about some work we've been doing with the Harlem Children's Zone and you guys--Michigan time is 10 minutes late.  Do I get to go to 5:40?  [Laughter]  Or is it--
>> As long as people will stay.
>> Yeah.  Oh, good.  I've got some time.  I've got to leave for my flight but other than that, we can stay in and hang out.  So look, I think the format I like the best is that you should ask me questions as we go along.  I really don't like it when I'm in the--your chair and someone is talking and I'm confused.  And there's a burning question in my head and I can't ask it until 45 minutes ago and then at my age, I forget it.  So, you know, and so like this.  [Laughter]  You know--just, you know, yell it out whatever.  We'll get to it and--but feel free to ask me as we go along.  But let me go ahead and get started.  Alright so here's a quick outline of what I want to accomplish between now and 5:30.  I'll give you some motivation behind why I think the 97 blocks in Harlem are an incredibly rich laboratory for social science.  Social science is to understand some deep questions that we've been thinking about for some time.  I'll briefly and very briefly go through the related literature.  I'll give you a brief history of the Harlem Children's Zone.  For some of you who don't know what Geoff Canada and his staff are doing, then I'll tell you about all the data that we've collected in our econometric framework and our results.  And then I'd like to draw a bright line after the results, okay?  I'm going to give--I want to be very clear about a few things, like I'm going to give you results that I believe in.  And then we're going to talk about stuff that I have no clue about, alright?  [Laughter]  And that's kind of a fun stuff but I want to be very crystal-clear about what I know and what I don't know, the results that come from our identification strategy I believe in, what mechanisms are involved, how you break those down, how do you move forward, how do you take these in scale?  I have no clue, alright?  Well, I got a little more than a clue but not much and so I want to make sure that you understand that is a very speculate discussion and I'm happy to have it.  I'm not shying away from it but I believe much more in my members than I do in what's next.  Alright, so here's the bottom line because I'm sure at some point, I have this great quality that tend to piss people off, that you're going to leave.  The Harlem Children's Zone is enormously successful at increasing the academic achievement of the poorest minority students in Harlem and I don't really have much of a clue why, okay?  It feels like in part of the analogy, maybe it's extreme but I feel like, you know, we've found a cure for a disease that's been plaguing us and that the cure is something like 1 apple slice, 4 nights a week, for 2 and a half weeks, broccoli every night and man, that's a really dynamic doctor and nurses that seem to care about you, alright?  So--
[Inaudible Remark]
Yeah, I really feel like there's something special going on in the Harlem Children's Zone, but I don't really know what and the problem with people like Geoff Canada is he likes to do a lot of things to help people and that's great but not for statistics, okay?  [Laughter]  And--
[Inaudible Remark]
--so, you know, I think we have a result here and I want to show it to you but at the end of the day, I don't know why they are having such amazing games.  In the classic, academic way, I kind of know what it may but it's not, but I don't know what it is, alright?  So here's the motivation.  The racial achievement gap is really, really big, alright?  It's about the gaps arise at age 2.  You don't really see them earlier than that.  That could be because they're not there or it could be because we're not good at testing 9-month-old kids.  I think we're pretty good at testing 9-month-old kids.  I think there are gaps there but that's a different subject.  I'm happy to talk about that later.  In about 0.64 standard deviations in math, the gaps are--and 0.40 standard deviations at school entry.  Now Brian told me this was a mixed audience.  Mixed in the sense that some folks know what a standard deviation is and some folks don't know what a standard deviation is.  No problem.  You should think of it this way.  The average kid gains in the average American public school about 0.08 standard deviations per month, okay?  So if I do my math, right.  In math, they are black kids are about 8 months behind in math, about 5 months behind in English language arts when they enter school, alright?  You probably know this from the NATE but the average black 17-year-old reads at the proficiency level of the average white 13-year-old in our urban centers--this is nationally representative data.  In our urban centers, it's much worse.  You know, I do a lot of work in DC and, you know, if you go to DC if you look at the NATE, 12 percent of the kids are performing math at grade level, 8 percent are reading at grade level in Washington DC, okay?  You know, it's--this business is so hard trying to, you know, help increase achievement and every now and then, you get a big bonus and big, you know, boost and you feel great about yourself and then you'd get kicked right down.  It happened to me this fall.  I was in DC with 1 of our schools because I like to be in our schools where we're doing this financial incentive experiments and I like to be there the first time they give the checks because we take so much flacks from all the adults, at least the kids really like it and so I like to be there with the kids when they get their first check.  [Laughter]  So I was there with the kids when they got their first check and the kids were going crazy.  They were so happy about the money and da da da.  And they had earned so much and I was so excited.  I said, "Man, well I think we're really doing something here."  And a fight broke out, next to me while we were given our checks.  [Laughter]  "Hey man, you got to separate that."  I said, "What are you guys fighting over?"  He said, "Mister, tell him that 38 is bigger than 42."
^M00:09:19
[ Laughter ]
^M00:09:26
I didn't know what to do.  I don't know.  There have been many attempts to close the achievement gap.  [Laughter]  'Cause the kid who thought 38 was bigger than 42 was bigger than me.  I didn't know.  [Laughter]  There have been many attempts to close the achievement gap.  There had been early childhood programs, lots of stuffs, small schools and smaller classrooms.  There had been school choice, voucher staff you know, people try to find systematic ways to make better public schools.  We've had neighborhood scuffled.  We've moved people out of there, you know, poor neighborhoods and move them to less poor neighborhoods.  Also it's a des--you guys, I want to know this.
^M00:10:03
We've tried a whole lot of things in the past 3 decades and I don't think we've had big results.  In fact, there's a great book called, So Much Reform, So Little Res--I think So Little Success, I think it's a typo.  But anyway, it goes through these things.  I think this lack of success was played into a pretty rancorous debate about whether schools alone can actually close the achievement gap.  Okay and when the--in the roundup to the presidential election, there were 2 kind of groups that were--there wasn't that much distance between them but they basically were kind of staking out their claims on this dimension.  Some folks said, "Schools alone are enough," that was the equal opportunity, equal--education equality project.  And others said, "You need a lot more than schools.  You need a lot of social supports, et cetera."  It turns out Jeff Canon signed both petitions.  I don't think we can get data from that but it shows that he's a smooth character.  [Laughter]  What I'm interested in is trying to understand whether or not schools alone can close the achievement gap or whether or not we do need these community investments, okay?  Let me be very clear.  I am not that interested in the Harlem Children's Zone per se.  I'm not interested in evaluating a particular charter school or a set of charter schools, right?  You know, since the results have come out, I've gotten lots of calls and people saying, "Can you give me some results like Jeff?"  [Laughter]  You think I'm lying?  So I'm not interested in that per se.  What we're interested in is the bigger question which is, you got community investment, should you invest in the community programs that we all know a lot about?  Should you totally invest in schools if you want to close the achievement gap, right?  And it turns out that these 97 blocks in Harlem are pretty good laboratory of at least trying to start to think about whether or not schools alone are enough or whether or not you need community investments.  So when I grew up my neighborhood, the easiest way to piss someone off is to step on their new shoes.  In academia, the easiest way to piss someone off is not have them on your related literature slide.
^M00:12:21
[ Laughter ]
^M00:12:27
It's true and it's likely because all the academics I know have bad shoe, so I don't do that.  I don't put anyone up on the related literature slide.  I know a lot of people in this room have written incredibly important papers in this literature.  But I don't do that so here is my very brief related literature, okay?  There is a huge literature on the racial achievement gap.  Lots of folks have contributed to that.  There early childhood literature, there had been bunch of stuff on school input charter schools, class size, teacher quality, et cetera.  There's a bunch of stuff on neighborhoods, peers.  This really--the Harlem Children's Zone are like, putting all of these stuff together and it's very related to a lot of these literatures that you have contributed to.  Okay, here's a brief history of the Harlem Children's Zone.  Just in case you haven't seen Jeff on Oprah promoted it.  It started in 1970 as New York City's first truancy-prevention program.  It was called Rheedlen Center for Children and Families.  Until the late 1990s, basically Rheedlen was an amalgam of after-school programs that in just words, helped a handful of kids escape the neighborhood's cycle of violence and poverty but allowed many more to slip to the cracks.  So his exact quote is, "I was helping them by the tens and losing them by the thousands,' okay?  So Geoff decided to create a new organization to focus on changing the whole neighborhood, alright?  And that's why he dubbed it the Harlem Children's Zone.  The idea was to address all the problems that poor kids were facing from bad apartments to failing schools, violent crime, chronic health problems, with the kind of cohesive web of services from birth to college, okay?  HCZ started as a 24-block area.  It expanded to 64 blocks in 2004.  And now, it's a 97-block area in Central Harlem, okay?  Now I know there're a lot of my colleagues here who were superstar statisticians much better than I am.  This is not the good part of Harlem, okay?  Don't worry about that.  This is the Central Harlem, right?  This just Harlem.  Now here are all the programs they have.  They have their charter schools which admit by lottery which we're going to use.  They have a bunch of early childhood programs so they got Baby College, right?  It's not for the babies, it's for the parents.  So Baby College is a 9-week parenting program where the parents come and they learn things about how to be a parent, like putting up--whatever, you know--Brian's got 3 kids.  He's my like idol on this stuff.  I have zero kids that I know about.  [Laughter]  So I just like to be honest.  So there's a socket plug thing that you cover the plugs with, right?  Okay, so they teach you to do that stuff.  My grandmother didn't know anything about that.  She just thought survival of the fittest.  If you're dumb enough to electrocute yourself--[laughter] that's just you.  They also teach about discipline, 1 of the most famous series of classes in the Har--in the Baby College is about not spanking your kid and using alternative forms of discipline, something my grandma also didn't know about.  [Laughter]  After you go from Baby College, at 3 years old--there's a 3-year-old journey which is a very similar parenting program for parents with 3-year-olds.  Then there's the Harlem Gems which is like Head Start on speed.  It has a 4:1 student and teacher ratio.  Kids learn 3 languages in the Harlem Gems.  It's basically a preschool program.  Just 1 thing of note, Baby College, your parents can go there as soon as--you know, when they're expecting parents, okay?  And 1 thing that is pretty unique about the Harlem Children's Zone is you don't just go sign up for these programs.  They--you're actively recruited for it.  So he has a whole list of full-time employees and volunteers that go door-to-door in project buildings, in laundry mats, at the check-cashing place on their corner and when they see someone with a small child they say, "You need to be in Baby College.  It's free, da da da," alright?  So in elementary schools, got a bunch of after-school programs, same thing with middle school programs, high school, college.  You see, he's got a bunch of stuff here, okay?  And what's interesting--what's nice about it--oh, that's not good.  Okay, let's hope that didn't happen again.  That was a map for the Harlem Children's Zone.  And what--there're borders there.  And let's go back to it, you can imagine.  Okay, there are borders here in this map and what's interesting about the Harlem Children's Zone is if you live inside the borders, you're actively recruited for that list of programs I just showed you.  However, there are also 3 to 4 charter schools in there and those admit by lottery, right?  So you can see where I'm going with this.  There are some kid--people signed up for the lottery, most of them live in the zone.  Some of them get to schools, others don't.  Everyone gets the community programs, okay?  That's the type of variation I'm going to be using to try to identify schools versus communities, okay?  In the end, I'll be able to tell you that it ain't communities alone.  I don't know if it's schools or school interacting with the communities that are the most important.  Although I'm going to make an argument and it's not just schools but it's an argument.  It's--I don't have the data to be able to support this.
>> You have to put the name in the lottery?
>> You do have to put your name in the lottery.  Again, you're actively recruited to do so and there're no stipulations for the lottery.  It's not like some charter schools that say, "Everyone's invited as long as they sign a parent pledge," or, you know, "You can do it if you satisfy the following criteria."  And in fact, I'm going to show you data, at least an observable that these kids look just like every other kid in Central Harlem.
>> How many kids--
>> Their parents [inaudible]
>> The parents--absolutely.  I mean--any lottery--I'm going to do 2 things.  Let me get to my--love question.  You're right.  I've thought about it.  I'm not sure I'm that satisfying, okay?
>> What percentage of your kids usually get in that signed up for it?
>> Half, because about 200 come in.  They only have 100 slots.  Okay, here's the data.  We combined 2 data sets, 1 from the Harlem Children's Zone Administrative File, so we went and--to the zone and helped them digitize their administrative files that actually have the lottery winners and losers, okay?  We also have data from Baby College and Harlem Gems of all the kids who have actually gone in.  Subsequent to the time I made this slide, we also have data on every single community program, time in and time out for all the kids who have gone through that.  As you might imagine, it's taken a few trusty dusty undergrads to get that data into a working format, but we're working on it.  We merge that data with the Department of Education and New York data where we have achievement data, attendance, et cetera, all the administrative data from '03-'04 to '07-'08, okay?  And we have the admin files from the actual lottery winners and losers to the achievement data so as long as you're still inside New York City Public Schools, that's Manhattan and the other boroughs, then you're on our data set, okay?  Test scores are only available for grades 3 through 8.  One second.  Attendance and promotion data are available for all years.
^M00:20:00
Yes ma'am.
>> What national data set did you use from the Department of Education?
>> This has been the Department Education in New York City.
>> I know, okay.  But what particular demographic did you use?  Did you use that set of data from the Department of Ed to compare with the Children's Zone?
>> Yeah, so of in--
>> What--
>> In New York City, with the New York City DOE, they keep data on every single kid and--
>> So it's every child.
>> --every single child, so we have 10 years where the--it's 5 years of data.  But we have 10 years of the data now and all 1.1 million kids for every year.
>> So state thing step rather than a national?
>> It's a city--New York City Department--
>> Just the city.
>> Yeah.  But it's all 5 boroughs.  So if the kids move to Buffalo, we don't have it.
>> And how many children then?
>> We have about 10 million and--it's 1.1 million in the New York City schools.  Yeah, it's great for experimentation.  But--yeah.  You know, it's funny 'cause I was there as the Chief of Equality Officer.  I hate that tittle so bad, Chief Equality Officer.  [Laughter]  But--it's a long story why that that title came out.  But it was not my idea but, you know, New York City is so big, like I'll remember sitting in my little cube next to Joe Klein and I'd say, "Hey, how many schools do you have?"  And they say, "You know that's interesting question."  I was like, "No, it's not."  [Laughter]  No wonder that kid don't know if 38 bigger than 42.  [Laughter].  Alright, so that--that's what we have.  We have New York City Department of Ed there and we have our Harlem Children's Zone stuff and we're going to merge those stuff.  Alright, so here's how we did that?  We've matched the data used in the file algorithm so we took your last name, your first name, and date of birth, okay?  And matched you to the New York City Department of Education with various abbreviations and alternatives, spelling, et cetera, okay?  Here's the quality of the matches we got.  For the Harlem Gems, we got 91.2 percent of the kids who went to Harlem Gems.  We found them in the New York City data at some point.  In the kindergarten treatment, we got 92.5.  In the control, we got 89.2.  In the middle school treatment, we got 90.6 and the control we got 85.4 percent of the kids.  Again, if you move outside the city, we don't have it or if you--you know, if your name is--you know, Caitlin with a Q in between and we didn't get the Q then we don't find you either.  My former colleague, Caroline Hoxby, one of our students estimate that 1 can expect about 90 percent based on natural attrition, we're in that ballpark.  And the nice thing about the results I'm going to show you is that they're so large that like even if we took the control kids and said they--the ones we can't find were really terrible, were--I mean the balance on these estimates will still--even in a lower bound estimate, you'll see big effects.  Okay, so here's the econometric framework in to your question.  We're going to do 2 statistical strategies, 1 we're going to use the lottery, the other we're going to use I, okay?  So here're the lotteries.  New York State dictates that oversubscribed charter schools allocated enrollment offers via random lottery.  Lottery winners will form a treatment group and lottery losers will form a control group.  So the ITT, the intent to treat estimate is the effect of being offered admissions into the Harlem Children's Schools, okay?  So a lot--the stuff I'm going to show you is going to be ITT effects, the intent to treat.  It's kind of like lower bound.  So this means, right, the effect of being offered admissions, that means you--Brian wins the lottery, he doesn't go, I'm still going to count him as a Harlem Children's Zone kid, okay?  Or if he goes for 2 days and leaves, I'm still going to count him as a Harlem Children's Zone kid, that's the intent to treat, alright?  And we're going to estimate that by having the outcome on the left hand side for each individual I, we're going to control for some basic demographic stuff and we're going to include a variable for whether or not you've been treated, that's the ZI, okay?  So gamma is going to estimate the treatment effect.  Now, the treatment on the treated which technically not an upper bound but it's--answered a different question is the effect of actually attending the Harlem Children's Zone Charter Schools, this estimate is going to be obtained by basically instrumenting for whether or not you went with your original assignments.  So we need something that's correlative with whether or not you go to Harlem Children's Zone, okay?  And that's going to be--what's correlated with that your actual original lottery assignment, okay?  And you want--you notice 2 things, I don't want to get bugged down to technical details unless you have questions but 1 estimate is the effect of being offered admission, that's the ITT, the other is the effect of actually going treatment on the treated.
>> Okay, Roland, my question was how did the students get in to the pool that was subject to lottery?
>> So they were actively recruited by the staff of the Harlem Children's Zone.
>> That's not a random process.
>> No, no.  They--well, it's not a random process but it's an exhaustive process.  They go out in these 97 blocks and they knock on the doors, they go to the laundry mats, et cetera, and they try to get people to sign up.  Now, who signs up is not a random process that's for sure.
>> That's what I'm saying.
>> And okay--so 2 things.  One, I will show you an observables that they look very similar.  And 2, we do IVs so that we don't have to actually--so this will help us with the external--
[Simultaneous Talking]
>> Well, what percentage of the illegible students have really signed up for the lottery.  That would be an interesting--
>> I don't know the denominator because I don't know how many people they contact.
>>No, just so the students in the region that are eligible to be in the lottery.  What percentage--
[Simultaneous Talking]
>> I don't know on the top of my head 'cause I don't know how many kids are in the 97 blocks.  I think--I don't know.  I don't even want to guess.  If you send me an e-mail when I get home tonight, I can find the number out for you.  Again, I want to underscore that this isn't a problem with any lottery-based analysis.  And 2, what I like about the Harlem Children's Zone is that they actually actively recruit people who wouldn't sign upon on their own.  So with--the 2 important issues which you just gotten 1 of them with lotteries.  One, the kindergarten lotteries which were not sufficiently oversubscribed when we have the data, and 2, lotteries are not necessarily externally valid, okay?  Who signs up for the lottery could be very different from the treatment effect that would happen for the kids who actually didn't sign up.  So we're going to complement the lotteries with an instrumental variable specification so we're going to play very simple IV strategy using the interaction between a student's address and their cohort, okay?  So let me just explain this in other words.  The identification is going to be driven by 2 forces, 1, a comparison of kids within the zone who were of eligible age in that year relative to others who were not, okay.  So the Harlem Children's Zone schools opened in 2004.  They only opened if--and they were on--you were only eligible if you were in kindergarten and in 6th grade.  So 1 of our basic counterfactuals in our IV estimate is to say, "Okay, let's compare the achievement of the kids who were in 6th grade and eligible that year versus the kids who were in 5th grade and were ineligible in the zone, just--I'm sorry, 5th grade is the bad one 'cause they'll be eligible the next year, 7th grade in the zone and who would never be ineligible for the Harlem Children's Zone, okay?  And the second piece of this is the comparison of kids who were eligible age who were close to the zone versus those that are not, okay?  It's the interaction of those 2--Jesus, are all my slides this way?  This is not--okay.  Okay, so what are the threats to this IV strategy?  Where it could go wrong?  Well, if the instruments are correlated with unobservable determinants of outcomes, then we're in trouble.  For example, if high achieving people move into the zone, when they have kids who were eligible for the lottery, then we're in trouble.  Did that happen?  I don't know but I can tell you that you're eligible for the lottery no matter where you live.  So moving would not have helped you in any way get your kid into the school, okay?  Two, address specific shocks.  Now, so if individuals inside the zone re--receive some sort of positive shock that is not received by other kids inside the zone, right--oh, sorry, other kids outside the zone or any other cohorts inside the zone, what do I mean by that?  If I'm given this talk and Brian says, "Oh Roland, I forgot to tell you.  In 2004, I went to the Harlem Children's Zone and I gave lectures about how cool it was to be an economist.  And that if you really study hard in school, you too can be an economist."  I would be screwed, okay?  Well, I mean, I'm sure that has no treatment effect but it is.  [Laughter].  So if you imagine--[laughter].  The biggest assumption I'll make all day is the fact that Brian has a treatment effect on this.  So--but if you imagined that Brian had a program, had a big treatment effect and he did it in the precise year that we're measuring and he did it only inside the zone and kicked out kids who were outside the zone, then the IV strategy would be invalid.
>> Just to clarify this, what you're talking about now is just how you're going to estimate this depth of the attendance to the charter school itself--
>> No.
>> As opposed to the effect of the community services and so forth?
>> Yes, this is--one of them now is how I'm going to measure the effect of the impact to the charter schools and I want to compare these IV estimates with the lottery estimates.
>> Okay.  And you're going to--and the--and you think that--I'm asking because the inside the zone and outside the zone obviously is going to influence how likely they were to receive and recruited for services--
^M00:30:06
>> Exactly.  That's the whole point, right?  And what I was going to show you with this quick time decomposer thing is that if you look--if you take a line and it's in the paper if you have it, if you go towards the border of the zone and you look at the probability that you going to sign up, that probability is pretty flat and then as you get close to the zone, it starts to go up, even outside the zone, okay?  So, you know, in same way people have used distances and instrument, we're using distance times cohort as the instrument, okay?  I hope the rest of these slides are here.  Let me show you some results.  This is the problem because I actually don't have the--this is a real--I did this on the Mac and this is not like working.
>> Yes, that happens.
[Laughter]
>> Thanks for telling me.  So how can--how should we do this?  I have Mac too.  Can someone have a--if you have a card I will PDF it on my Mac and then we can put it on these thing as PDF, right?  Sorry for the technical glitch here.  Okay, probably better now, okay cool.  I just want to show you I have a map  I know you think I'm lying but we got a map.  [Laughter]  So this is 116th Street, the 125th Street, and Madison and so this is were the zone's middle schools are, alright?  By the 134th Street--yeah and 131st Street in over here, that--that's where the elementary schools are.  Te elementary schools is away from the middle schools, et cetera and I think that has some interesting hypotheses about what's going on there but that's that map, got everything for you now.  Okay, here's the thing I promised before.  So here's--now I have no pointer, that's okay.  [Laughter]  So here is where the identification is coming from, okay, so if you are in the previous cohort, so if you were a year older, if you were in 1st grade when the lottery was for kindergarteners, you can't get in so you see that the black line is being percent enrolled a hundred meters--a thousand meters from the zone, 500 meters from the zone, et cetera, this is going towards the border which is a little counterintuitive but going towards the border, getting closer and closer to the border of the Harlem Children's Zone.  What we did was we mapped all the kids in a thousand meter away from the zone using RGIS, using different points from the zone.  And here the test scores in ELA and math, okay?  So as you can see, as you get closer to the zone's border actually test course are falling.  Now, here is the 2004 cohort, okay.  So these are kids who actually are eligible.  You see that the percent enrolled is very close to zero until you get close to the zone and it actually starts to go up.
>> Pointer.
>> Oh, what does this thing do?
>> It's a little thingy.  Put it the USB and it's--
>> Oh, okay.  It's like you're going to give me something that's broken.  [Laughter]  Alright, let me see if I can do that.  I'm sure--let's go--all of us go on break.
[Inaudible Remark]
Alright, I'm doing a talk about whether or not communities are important.  Everyone's having to help me with my own talk.
[Laughter]
>> It's working?  Probably working?
>> Alright, I want to keep talking and see if we can get this thing to work.  So the identification is coming from this, the difference between these 2 sides, okay?  Various cohorts, kids who were eligible, kids who were not, and distance.  Distance interact with cohorts, just want to show you that, okay?  So here are the results.
>> I try.
[Laughter]
[Inaudible Remark]
>> She knew that before.
[Laughter]
Let's see if I can spend up some more this time.  Okay, so the--[laughter] I had to call you.  So here you go.  Here's 5th grade when the kids--so April 14th of their 5th grade years when this lottery happened.  So I'm showing you their test result in 4th grade and in 5th grade.  No treatment has happened yet, okay.  This is for math scores.  Alright, the dot--the dash black line is the average black kid in New York City public schools.  Alright, the dash gray line is the average white kid in New York City public schools, okay?  The red line are the lottery losers, the blue line is the lottery winners.  Okay.  So, now to your point, I still don't have it about whether or not they have motivated parents.  I don't have that here, point taken.  If you were to do this in KIPP or some other charter school in New York City, they'd be a quarter standard deviation above the city average before they ever start.  Okay?  At least that's not the case here.  So, you see what's happening.  Now these are the ITTs.  This is just--did you actually win the lottery?  Okay.  So this--so for kids who went for 2 days and then left, they're still in here.  This is the ITT, this is the lower bound, alright?  There are no controls here.  I don't have any free lunch stub, I ain't control them, or what kind of car they drive.  I have nothing like that.  Okay?  So these are just raw treatment effects.  Oops, no way.  Here is if you look at the actual people who comply with the lottery versus the control [inaudible] mean.  What that means is these are people who actually were admitted and went.  So this gets closer to the treatment on the treated, the effect of actually going to the Harlem Children's Zone.  Okay, again I have zero controls here.  So what you see is the same thing, very similar, increase in 6th grade, they increase in 7th grade, and a big bump in 7th--in 8th grade, okay?  Now we have this data, this is the--which cohort is this?  These are 2005 cohort.  We have the data for the other cohorts and it's very, very similar.  So this is not a one year thing that's happening here, okay?  Now, I just want to pause and look at this for a little bit because at least in my work, I'm always constantly staring in like 0.2 treatment effects.  Okay, and you saw--you sit there and you say it, "This has two stars versus one star, this is 0.2 and this, you know, this is great."  This is like 1.2, it's very different, okay?  And so I remember when the first time that like, you know, you have this data program, this data spits out--this [inaudible] the laser thing there.  This laser works.
^M00:37:05
[ Laughter ]
^M00:37:10
Next time I will study the racial gap in laser.
[Laughter]
[Inaudible Remark]
Okay, got it.  Thank you.  Go and hurt me I can say.  So, first of all I saw this, at first I thought it was a mistake.  But then once we really looked at the numbers and realized that there is actually something here, I had a great time.  Because I have, you know, a few full time [inaudible] and they're like, you know, former Harvard undergrad which is I said,--well, not to him and I said, "You know, we really got to think about this gap here because there's like a white-black gap going on [laughter] and I'm not sure why you guys can actually achieve."  [Laughter]  I would start with your culture. 
^M00:37:56
[ Laughter ]
^M00:38:01
I said that, they were like, "Well, we went to Harvard."  "Like just calm down."  [Laughter]  Just relax.  Okay, so that's math, and that's great.  On ELA, things are more muted.
>> [Inaudible] please go back one slide there.
>> Yeah sure.
>> So even though there are [inaudible], these huge treatment effects going on--
>> Yeah.
>> --grade 7 and 8?
>> Yeah.
>> If the average for all bachelors in New York City is kind of staying flat.
>> Yes.
>> So is that true?  We cut--and it's mechanically true because people largely and [inaudible] actually doing worst?
>> Yeah, but this is like 200 kids, so--and there's like, you know, 500,000 black kids so even the--even if they weren't--even if they were doing a little bit better, you wouldn't expect the mean in the line. 
>> Even the red line, it's going down.
>> Yeah.
>> So there are actually the ones who, you know, were complying and actually didn't get in or doing worse.
>> Yeah.  What I should do here, this is the average in New York City.  What I should do is put the average in Harlem here.
>> Right.
>> And you know I can do that with--I don't know what it looks like but it's not clear to me that, you know, we know middle schools are a place where gaps really start over, but not and it's not clear to me that if you looked at just the poor place, I mean, did--you know, did the--this is all black kids and the treatment and the control are just basically Harlem kids, okay?  That's a fair point, totally fair point and I should put it here, okay?  Now, ELA is not as dramatic, however, if I'd shown you this first, you would have left happy.  You see the same thing here, all of them--they kind of go down and there's a little bit of a difference here and basically this difference is about whatever 0.2, 0.3 standard deviation difference is, you see a little bit more in the control comply mean, okay?  So the ELA results are not as impressive at all as the math results.  Why is that true?  I don't know.  One thing I do find interesting though if you look at the elementary school results that I'm going to show you in a minute, those math and ELA are both going in lock step together, they're both just as big.  So there are obviously these theories out there that say by the time they're in 6th grade it's very tough to move language relative to math, skills, et cetera.  This is consistent with that, whether or not that's true, I don't know.
^M00:40:10
Okay, so let me show you the distribution 'cause you might be thinking, well, Harlem Children's Zone is probably just good at like taking up the kids who are already motivated and the rest of them really didn't happen.  So here you just see the distributions between the winners and the losers, again this is back to the ITT which is just a distribution of 5th grade, it's very similar, okay?  In 7th grade what you see is that, and this is--this is math, this is math.  The lower kids you see a push here, okay?  By 7th grade, you see the distribution starts to separate.  By 8th grade, you just see the--basically they're pulling apart of these distributions, okay?  I always like it when you can get a paper and pictures instead of like saying and, you know, if you squeeze real hard, so that's great.  In ELA, you basically find nothing that looks like that.  Very similar in 5th grade, 6th grade very similar, 7th grade incredibly similar, and 8th grade you find a little push out, okay?  Why do you find that push out in 8th grade?  I don't know.  But when I was going over the results with Jeff [inaudible] I said, this is curious, man, you got nothing here in this little push out, why amongst these kids?  And if you known New York City public schools they rate things in scales of 4, 1 and 2 aren't proficient than 3 and 4, right?  And if you look at it like that, if you discretize it, you realize all of it is coming from 2 through 3.  All of it is coming from 2 through 3.  I said, "Yeah. I don't really--" I said, "I don't really know a model to predict that."  He's saying, "I do."  I say "What's your model?"  He said.  "I told the kids if you went from a 2 to a 3, you got a trip to Disneyland."  [Laughter]  So, I don't--I have no idea if that's what happened but he is unapologetic with those kids who went to Disneyland.  Okay, so let me show you something a little bit different which is, you might be thinking, "Yeah, [inaudible] where those are affects from the state test scores in New York City and they concentrate on the test, they teach to the test and I would argue that with you till the--to ignite with all that but that's fine.  But let me just show you something that's not even like for public consumption in some sense.  Let me show you something that's just for HCZ kind of internal purposes.  So we found out that HCZ gives the Iowa Tests of Basic Skills, okay?  Just for their kind of internal--one thing internal test they give kids.  Some people give chapter test, they do that also, they also give the Iowa Tests of Basic Skills, okay.  So here a straight line is normal growth, so if you're going up, you're doing more than average growth.  The dotted line is kind of obviously the 50th percentile, okay?  I don't have treatment effects here for you because the control kids were not in schools that gave the Iowa Tests of Basic Skills so all I can show you is how Harlem Children's Zone kids are doing by themselves.  You can make up your own mind whether or not this is big or small or I don't know, but I think it's something interesting and I want to show you all the data we got.  So you see in math scores they're gaining, I mean that they--on another test not the actual state test itself.  A very similar, this is the 2005 and 2006 cohort here, similar thing with a different cohort.  In reading scores, they're going up in 2006 cohort a little bit, this is going up and then it's flat.  They're not near the 50th percentile so that's, you know, that's something one should consider that.  Whether or not the treatment effects are huge I don't know again because I don't have the control group but I want to show you this.  This is--some summer statistics for the lottery, winners versus losers, the point is there are no P values that are small.  Here are the middle school lottery results.  Let me just put some numbers into the pictures that I gave you, because we all have the kind of, you know, you reduce class size from 24 to 16, you get 0.22 standard deviations.  You have your teach for American teachers, it should gives you 0.15 standard deviations in math, 0.03 in reading.  You know, we you can go on about basically 0.2, alright.  So let's just look at what happens so we take the 2005 cohort.  Okay.  You have about 0.2 in the ITT.  It's about 0.279 on the Treatment-on-Treated okay, for 6th grade math score, 7th grade math score it goes up even more, by 8th grade math score these are big things so these are standard deviation units.  I haven't divided by anything, I didn't pull one on you, okay?  This a 0.733 standard deviations in the ITT, 1.112 in the Treatment-on-Treated, okay?  ELA score is not as big but then if you get to the end is by 0.239 and 0.363, okay.  You can look at the paper.  I want to get the more things here but 2006 cohort, this is an old table, we now have these results but you can see the pool samples, et cetera.  It's robust across years.  That's what I want to be able to show you.  Now, let me show you some IV results.  Yes?
>> I have a question here.  It looks like that based on that, you know, only about two-thirds or three quarters of the people who won the lottery ended up attending, what's the difference just from going from ITT to the treated Treatment-on-Treated?
>> Not quite because the Treatment-on-Treated is like, you know, yeah, no, actually you're right.-
>> Something like.
>> Yeah, somewhat--
>> Some not completely trivial fraction--
>> Don't even go.
>> So I'm just curious, you have any sense where they were going and it kind of get set like the treatment effects.  What is the difference between the Harlem Children Zone and--
>> Yeah.
>> Is it the neighborhood school--
>> Yeah.
>> --at some other great charger outside the zone?  Is it?
>> For the vast majority it's just the neighborhood school's problem, the vast majority.  Alright, so let me show the IV results.  Okay, now we have to decide what inside versus the outside of the zone means and I don't have any good theory on that, so I've shown you multiple ways where, you know, you take 800 meters outside the zone, you take 1600 meters outside the zone and 2400 meters outside the zone, et cetera, okay.  And as you might expect, as you get 2400 meters outside the zone, you know, you're over in Columbia University and the more fancy places and so, you know, the comparison groups are a little different.  
>> Yeah, is there kind to teach African and African-American history in this schools?
>> Yes, but not more so than, you know, a typical charter does--I mean a typical school does.  I think the difference is that--where the difference potentially could be is that there are a tremendous amount of positive male role models et cetera who were there teaching the kids on a day to day basis, so.
>> Were they didn't cooperate again to the regular curriculum, do they have separate classes? 
>> I don't know the answer to that.  That's a great question, I don't know the answer.  Okay, so you see the lottery estimates in general are big, of the IV estimates in general are bigger, okay?  So now by 8th grade you're looking up, you know, 1.3, 1.5, 1.3 depending on how you actually define this and ELA you're still around 0.2, 0.3, okay?  So what's kind--here's what I take away, it's a little over a standard deviation treatment effect in math about 0.3 in ELA, okay.  So we do the results by subsample, by girls and boys, for example, and you don't really see big differences except for the 8th grade math score here okay, and you see it on boys.  Again, that's something that we haven't--has not been consistent in the literature, you know.  My reading of the literature is if you look at the intervention projects, the ones that are--the ones that work are more likely to work for girls than the boys.  Harlem Children's Zone is a little different in that, okay.  The elementary school IV results, okay, here I don't have nice pictures to show you because they were just in 3rd grade.  So it would just be a dot.  Here you see the math scores are about 1.9 to 2.  Now you're saying, "Why the elementary school things larger?"  Well they had an extra year, okay?  So the average basically is about, you know, 0.4 a year so and these people--these kids had an extra year because they went kindergarten through 4th grade, right?  These other ones are just 6th grades so they have an extra year, it's a larger treatment effect.  What's interesting here is that the ELA scores mimic the math scores in the elementary school.  Yes ma'am.
>> I have a question here.  The reason I have--
>> Oh sorry, sorry.
>> I can't see here because of that.
>> Oh I'm sorry.  No, I would--I'm sorry I'm just ignoring it for now.  [Laughter] What do you want? [Laughter]
>> I actually ignored that question, I'm going to go on to the different question, okay.
>> Oh [laughs].
>> The--reading and writing, English language are it's very--this is a very different subject, this happen in 3rd grade than it is in middle school
>> Yes.
>> Do you have any breakdown of ELA scores for the 3rd graders' reading fluency, comprehension and vocabulary, anything like that? 
>> Great question.  We can, we just got the data.  So we just got about maybe three weeks to a month ago, we just got individual question level data for 10 million children in New York City.  And then--
[Inaudible Remark]
^M00:50:03
No, undergraduates girl, that's what you do.  [Laughter] So we have that data and so I'm not sure I'll be any good in sub grouping the questions but I think the state test actually puts them in sub-groups.  Yes, I think while I'm talking in and so--yes we can do that, we ought to that, very fantastic question and because of my age could you e-mail me that question?  [Laughter] Oh good.  Okay, we present lottery results but as I said the lotteries won't really over subscribed but I just want to show you everything I know.  These treatment effects are still big but the standard there is enormous.  Here's the Iowa Test of Basic Skills of the elementary school 2004 cohort, 2005 cohort.  What's interesting about these is now you're starting to get above the national median, you know, and so again I don't have treatment effects for these but these are some pretty, well a little wild, but pretty on average, pretty big gain is coming even in the Iowa Test of Basic Skills, okay.  Now I want to keep plowing through here.  That's all the data I have that I can do IV or lottery estimates on them.  Now, I really want to show some results with Baby College and with Harlem Gems even though I can't do it justice.  They don't admit in Baby College in a way that would allow you--they don't do it by lottery.  Everyone, they want everyone to be in Baby College, right?  I know good for people, bad for me or Harlem Gems, they don't do it either.  So what I'm going to do now is show you some really, really simple ELAs of just what happens to kids who come into Baby College versus those who go to Harlem Gems, and which way the selection goes?  I don't know.  Okay, I don't know if people select in who were worst, I don't know if people selected who are more motivated.  All I'm going to be able to be able to tell you is basically there's very low evidence of a big treatment effect on Baby College, okay?  So here, what I have here is Harlem Gems and Baby Colleges of.  Now, I'm looking at the math scores in 3rd grade, so I've traced people who have gone through Baby College and are now in 3rd grade, and what I'm doing is looking at your 3rd grade scores on the left hand side, controls on the right-hand side plus an indicator of whether or not you're in Baby College.  I'm reporting that indicator here, okay?  I'm also doing the same thing for Harlem Gems, I've also done the same thing for Promise Academy in an all ELAs ways you can kind of see, ELA versus ELA, okay?  Promise Academy still looks big, Harlem Gems.  You know, the problem with this is this is point 2, which is basically like the effect of head start point 2.  But it's measured with such error, I can't tell you much, okay?  So it could be, I don't know.  What I'm more confident in is that Baby College doesn't seem to have a big effect on test scores.  Kids are coming to school more, they have less--sorry, that's the ELA score.  The total absences and so total absences is--this is point 4, it's incredibly time and relative to standard error.  So I would say on these dimensions of achievement I don't see any evidence that Baby College had any effect, okay?  Now, what I told Jeff, I said, "Your Baby College ain't nothing."  He says, 'cause that's the way I talk, he says, "Oh, but you got different outcomes.  You think Baby College is to increase test scores and total absences," Baby College of his vision was to get past stop cursing up the kids.  So he says, "I don't know if that has treatment effect or not, we have different views, different outcomes," okay?  So one of the things I want to do going forward is to collect more data on what's actually happening when people go through Baby College, et cetera.  All I can show you is what I have, alright?
[Inaudible Remark]
>> Yes.  There is a long list.  Do you want me--so you know like, you know, the stress level is I want to set the interaction the parents are having with the kids having come to Baby College.  I would like to figure out a way to understand even it's just a survey data, how the discipline is gone, I want to observe parent-child interactions to figure out if the, whatever they call it, the cultured cultivation that theory that they're teaching Baby College the action stick in, so.  Bill Wilson, Orlando Patterson, Rob Samson, myself would put in together a qualitative team who's actually going into the Harlem Children's Zone to actually observe these other outcomes that it's hard to get it through administrative data but I don't have that yet.  Okay.
>> These are only students who ended up in the charter school or--
>> No, these are in New York City public schools.
>> So who's the, in some of the people, there's some subset who went to Baby College.
>> Yes.
>> But then there's, you know, a million kids in New York City who didn't go to Baby College.
>> Yes.
>> But the number of observations here is 109.  Who's the--
>> These are people who signed up for the lottery.
>> Okay.  So it's only who signed up in the lottery.
>> Yes, I was trying to take out some of the effects that you're probably thinking about.  Okay?
>> So these are people who signed up for the lottery,
>>Yes.
>> Didn't get in then went to a public school.
>> Went to a public school, but also--so these people went to Baby College?
>> Went the Baby College.
>> Yup.
>> But they're not in the charter school?
>> Not all of them, so that's why we control who went to the charter school.
>> Have you done this for the students that went in to the chatter schools?
>> Yes.
>> Look at the effect of the Baby College?
>> Yes, yes.  The results, the samples are small, too small to actually show you anything.
>> Too small.
>> But let me show you the following, okay?  Here's the best evidence I got.  Forget the regressions.  This is just a, did you go to Harlem Gems and not on your own, your IOWA test of basic skills?  Those lines couldn't be more similar.  Okay.  Again this is, I want to show everything we have.  I'm much less confident obviously in these results because there's no identification strategy than are on the others.  But I will say if someone asks me, is Baby College knocking it out of the box?  I would say, I doubt, okay?  And so again, you just see that these things are following a very similar whether or not you went to Harlem Gems, okay?
>> Okay, this is what I'm thinking.  Going back to your, this is just the speculative point of it?
>> Oh, yeah, let's call it that.
>> No, this is, I want to honest.  I mean, I don't know.  I mean, I'm confident that these regressions are correct.  What they measure is a different story.
>> The graph you have up there on performance in lottery
>> Yes.
>> In school, not in school but in the lottery.  With the performance of the kids who did not get into the school but who were in the lottery was flat or declining, but the performance of the schools of the kids who got into the school was increasing, you got that very dramatic results for the map.  That's we're [inaudible] graph right.
>> Yes, that's true.
>> Okay, so all of those kids, those 2 samples, the ones who've performed well, the ones who did not.  On average, their parents are the same because it's just a random draw, right?  In that school, if that school is doing something for those parents who want to do something and is working, you might be able to pick that up with the--whether or not they actually went to the Baby College.  Last statement, if I got you right.
>> No, 'cause I think what you want me to do is you want me to run an interaction term between Baby College and like the treatment, whether or not you, so, Baby College in the treatment.  So what you really want.  Like I have clicker that works is that, you want this.  You want you want a sub sample to be Baby College about Baby College? 
>> That's right.
>> If I have the data, I can do it in minutes.  I don't know if I had the data but I'll check.  It's great question, great point.  I'll see.  You know, the problem with this obviously is, you know, even we got 486, so even if we're splitting by gender, we're on the end, so we have to be a little careful, okay?  And you know more years, more data.  I mean, I think this is the first project not the last project in terms of this treatment process.  Okay.  So let me keep me rolling here.  Okay, what have we learned?  And then I want to open it up obviously for discussion and you can beat all this up more than you already have.  I think what we've learned is the Harlem Children Zone, at least the Promise Academy Schools or norms is successful in boosting math achievement, and ELA achievement is not as good--ELA and math in elementary school and math achievement middle school.  Now what else can you squeeze from the data?  Okay.  Now, here is where we're speculating.  Is of community versus our schools?  So let me show you what investments you get if your community versus our schools.  Reiterating what we talked about before.  The community bundle, you get to early childhood programs, the public elementary school programs, et cetera, just in the community bundle, you get the after school stuff, et cetera.  There's also kind of student-family bundle, maybe it's a bad way of thinking about it but if you're a sibling of a kid who gets into the lottery, and you're in that family, you're going to get some sort of investment, right?  Because Geoff Canada, he sends fruits and vegetables home, okay?  He sends pre-made meals home to the kids.  He gives as much materials support, advice the parents, you also have greater knowledge about the community programs if you have a sibling who's coming home saying, "I'm going to go win the All Night Basketball games happening at the Rex center they told me about today in school."  Alright?  That's what I think of the student-family bundle.  Then there's just a pure school bundle.  Now what is Jeff Kennedy doing?  On several trips to the Harlem Children's Zone, I tried to figure this out because, you know, it's like a little mystical when you first see it.
^M01:00:03
And so we went to the Harlem Children's Zone and we just interviewed the principals, the teachers, the kids, and that kind of staff.  Honestly it was like a bad episode of Law and Order because the principal--I would say that the principal, wow, look at this effects in Math.  How did you--what things happened?  He said, "Strong principles."  [Laughter].  And then you got all the teachers he said, "How do you think you can explain this?"  And you'll say, "Just good teaching, boy."  So I am not sure we got much out of that but what we did get out of it was that, the kind of list of things that they're doing in that school.  So they give health and dental services, okay?  So if you're sick, you're not feeling well, you go right in this--in a middle school the health and dental services right inside the building.  They're not if you're in elementary school and I actually think that's interesting.  They have social workers inside the school, so I remember when I was a kid and you look out and see if it's sunny outside, you hit someone outside in the head, you could go home.  Here you can't go home, they like talk to you about your feelings.  I mean that's terrible. [Laughter] So they got like social workers.  So if you get in the fight here, you get pulled away, you know, why did you do that?  Do you feel regret?  And all that stuff [laughter] and then you go back into the classroom, okay?  So they also have date oriented instruction.  So, you know, it's not like the term which is used a lot education reform.  What they do is they test kids every 6 weeks.  They break the tests down by skill so they know whether or not Mel understands linear equations with one unknown and then he gets 2 hours after school everyday to practice on the things that he's the weakest, right?  Okay, you got more time in school, these kids, they stay in school from 8 to 6 and they only get 27 days out in summer.  In the winter, when they go home for Christmas break just kind of it pays 50 dollars a day to come back, okay?  So they actually calculate it.  They spent twice as much, literally twice as much time in school relative to kids in normal public school.  Every time I want to borrow money from Joe, I tell him, "I tell you to your elementary school, it's not normal to be in school 'til 6."  He says, "Don't do it, don't do it!"  This may seem silly but they're real.  You know, like I've been in a lot of schools and the environment in the school is pretty incredible, like the teachers seem and they sound silly but they really care about these kids.  They have skill teachers, they have student incentives, they have nutritious school lunches.  So it's got like the whole food basically there like, you know, you go in, you get like whole grain and like, you know, broccoli, I've never eat there, it's ridiculous.  [Laughter] They have coordinated after school programs so if not like your normal after school program which I participated as a kid, where you break through school, you go like 10 minutes then you go after school people, you just show out, right?  No, these after school people will come into the school in like the last few periods and it's a seamless space of the after school.  The other teachers you notice they're not there, you're working on your stuff.  It's very, very different.  Now, this is cool and all but it makes statistics impossible, okay?  So communities or schools?  Well, I don't really know, okay?  But let me tell you 4 pieces of evidence that suggest the change in communities alone won't do the trick.  Number one, our IV strategy compares kids relative to other cohorts in the zone that were not eligible for the lottery.  So in some sense, our IV strategy purges much of the community effect, okay?  Second, I'm going to show you these results in a minute I think.  Siblings of the zone kids who had access to all the community programs but for random luck, we're not actually in the schools show no gains, okay?  So if my brother gets in Harlem Children Zone versus I don't really--into the actual schools and I don't.  I'm still in the community programs, I know I had pretty good knowledge of the community programs 'cause it's my brother, my test scores don't move.  My access has go down and that's probably because my brother is in school all day and my mother is like you need to go to school too but my achievement doesn't actually go up, okay?  When we do analysis of sub samples, like the boys versus the girls, we notice the kids inside the zone don't have any difference in effect of the actual charter school relative to kids who are actually outside the zone.  Forth, if you look at the MTO experiment?  You know when they move neighborhoods?  I think if you designed an experiment, you said, "Roland, we should design an experiment where we're going to change neighborhoods but not change schools?  MTO would probably be like really, really good.  I mean that's like to a first order approximation, that's basically what they did, okay?  I mean not on purpose but folks move neighborhood but they're going to be very, very similar schools and what you see is that the girl's achievement doesn't change, the boy's achievement actually goes down a little bit.  And the last thing which is pure speculative and should be a separate point here is if you talk to Geoff, you say, why did you start charter school?"  He say, "because of my community investments.  I didn't see any return in terms of the actual achievement of the kids going out, okay?  So four pieces of evidence, one anecdote.  Here the sibling results, you see the math score, it's point 2 but it's measure was set error, ELA score is this, the absences were actually statistically significance or nonetheless absences.  Being on-time, the grade level is nothing more so these are looking at just the set of siblings in the Harlem Children Zone.  One kid goes to the charter school, the other kid does not, you don't see differences in achievement during that day.  Now here's a very speculative discussion and I'll probably end it here which is what's driving the school bundle, okay.  You got all these stuff, health, mental health, longer day, the chef in the school, all this stuff.  What can you parse out from this?  Well, literally impossible to disentangle what the heck they're doing.  I mean we got like 12 things, right?  And just a few schools was all of but you're doing all 12 things.  I want to run this experiment where you test all the different possible combinations in the bundle but 2 of the 12 experiments and I'm too tired for that.  To tell you, I don't think just based on other people's research, not my own research, it's purely speculative.  I don't think it's teachers [inaudible] value added alone.  Right, 'cause there's other folks who are doing, teacher's incentive tied to value added, they're not getting any results to look anything like this.  I don't think a social worker is alone.  I mean we have programs like turn around for schools et cetera that put the social workers inside the schools, they're not getting any effects or anything close to this.  I don't think it's due to the incentives alone.  I've been through a lot of those.  I can tell you I don't have any pretty graphs like this to show.  I got some interesting results but not of the magnitude of these results are.  And you know, I don't think as longer school day alone because again, they're been other people who have done things like lengthening the school day and--sorry and not getting really great results.  So what could it be?  I though they were cheating.  I really did, it turns out we got the questions level stuff that I talked about before and we looked in like excess variation then we ran with Brian and Jacobs and Steve Levitt [phonetic] algorithm for how to catch cheating teachers and it turns they're not cheating or at least cheating very well.  [Laughter] You can't tell the difference between cheating very well and not cheating.  So it could be amazing teacher though.  They have some very, very, very, very, very good teachers, okay.  So 8-grade math there's this guy name Mr. Petite [phonetic], he's an ex marine.  He's a 6-foot-3 black guy who you know does like math as a combat sport, okay?  And he is unbelievable, right? And he's a very interesting guy.  What I think it is and what economists typically have kind of troubles thinking about is interactions between elements of the battle.  Okay.  And so, you know, at least when I was in graduate school the cool thing to do is to estimate a partial derivative.  I held everything else constant and I increased the school day.  So I'd know exactly the school day you know, so you know, 'cause it wasn't cool like have like four things.  All of which could be interacting in ways you didn't know but the combined effect was large.  That wasn't cool.  What was cool was to estimate a partial derivative, right?  And my sense is, I'm not saying anything deep here, is that the total derivative may be far more interesting, right?  If you have better teachers with a longer school day with something else, maybe that total, right, you remember the chain rule.  Maybe that total derivative is actually a lot higher.  Okay, so, I think try and understand elements of the bundle, I mean, interactions of the bundle.  It's really important.  Let me tell you about what we're doing next and then I'm happy to stay as long as you want for questions.  We're looking at longer term right now in non-educational outcomes.  Things like teen pregnancy, crime and so on, okay.  And before I make a stronger statement about communities versus schools, I want to look at other outcomes where you think the communities will have much more of an effect than actually just test score achievement exam.  The other thing we're doing is we've been working with the Obama administration who plans to roll out 20 of these across the country in a way that will, you know, putting them together in a way that will maximize learning, right so, you know, I'm a nerd and I called up folks in the administration.  I say, hey, you're rolling out this thing.  How you're rolling it out and the way where we're going to be able to learn which kind of is what's driving luck 'cause you can roll in out in a way and they said, that's a good question.  No, it's not.  You're spending a billion dollars, you ought to be able to tell me, you know, how to do that and I have concrete ideas about how to do that and we're going to talk about that.  Last thing and then I'll shut up is one of the things I'm so interested in is, of all these, is trying to figure out, okay, let me read this.  What the pill looks like.  Okay, I want to figure out what the magic bullet might look like.  You know, he's doing 12 things.  Could we figure out four or five of those things to actually try in a random school-based trial and regular public schools 'cause personally, I'm not that interested in just, you know, this set of charters that are all closing the achievement gap, that's great.  Question is how can we take that stuff and actually close the gap in public schools and you know if, you know, we asked Geoff, so I did.  I said, hey Geoff, I want to do an experiment.  We'll take four things you're doing and what should those four things do--be.  He would say human capital, the human capital piece, you got to have good teachers, you got to be able to pay them in a way that's going to attract the right people.  Longer school day, data-oriented instruction and the culture piece.  You got to make sure that everybody in the hallway, from the teachers to the janitors, thinks that 100 percent of these kids will go to college.  If you don't have that you're going to fail.  So, I would like to figure out, I know that's complicated how you do an experiment like that and all that, I get it.  But the goal would be to try to figure out are there three or four things that we should take from this or keep or an achievement first.  Put it to a school-based randomized trial where we get 100 schools to try--to sign up, 30 of them get this four element treatment, the others don't, to see if we can actually get gains over three years [inaudible], right.  So, last statement there, I promise I'll shut up.  Here's why I'm so excited about this.  I think before we conducted this analysis, at least for me, I was sitting in my computer saying, oh my God, nothing works, nothing works, nothing works.  Contributing to fatigue, people saying nothing can work.  This all--you're just wasting your money, nothing can work, nothing can work.  Now I feel like we're in a spot where there is something out there that is actually has parts of the tide.  It's actually working.  The question is, how can we boil that down to pill form so you could transport in other places.  But for me that's a much, much, much, much better place to be in.  I know there's something.  I just don't what it is versus maybe [inaudible].  Okay.  Yes ma'am.
>> A quick question about the [inaudible] places.  Obviously, person needs to be part of the field somehow [inaudible] leadership.
>> I totally disagree with that in the best way, with all due respect.  I just--
[Inaudible Remark]
Okay, the question is, she says, and I get this question all the time is that one of most important things is going to be to clone Geoff Canada.
[Inaudible Remark]
Okay, his type of leadership.  Okay.
>> Okay, that's a broad concept.
>> Okay, that's a broad concept.  Now I'm more on your board--more on board such that he is a good manager, okay.  He does, and that's it.  He's a good manager and we--the supply of good managers I think is large.  A lot of people take he's a mystical figure who somehow miraculous can change schools because he's so smooth.  I say no, he's like my uncle.  He just got a job, that's the only difference.  And look--and he's--so, I mean, he's good and he says he can raise money but he's a good manager, he has good performance management.  He sticks to goals, has timelines and I think we got a lot of folks out there like that and we certainly got to one.
^M01:13:36
[ Inaudible Remark ]
^M01:13:59
Yeah, I can add it to my list of things I don't know but yeah, I think that--I think you may be right.  I mean it's part of what's going on.  I mean, yeah, I think you could be right.
>> Yeah.  Are you familiar with that experiment they did back in '84 out in North Carolina State where it had college students participate in the apocentric [phonetic] classes versus the none and the ones who were in apocentric classes had GPAs improved by 1.2?
>> No, I haven't seen that.
>>Okay.  But I'm just saying that--
>> Well you send it to me.  I would like to read it.
>> Okay.
>> Please do, please do
[Inaudible Question]
>> Yes and no.  I have a cost per kid--I have cost per kid in the zone.  I don't know what denominator kids take up the program, okay, so that's 5,000 dollars per kid.  Okay, you raise these 50 million dollars a year.  He's got 10,066 kids in the zone that he treats for one service or the other.  And you know, so that's the 50 million.  How much of that gets allocated to the schools versus not, I don't know.  But it's in the 3 to 5,000 dollar over and beyond what normal New York City public schools get per kid.
>> I have a question about your definition of schools and communities.  It's been interesting to hear schools discussed in contrast to that communities.
>> Yeah.
>> Schools are important part of communities.
>> Yeah.
>> And part of building an effective school community is making the connections with that community.  So, I'm just wondering about--maybe it's a language [inaudible] I'm just wondering about how the [inaudible].
>> I will admit I really don't understand your question but I'm happy to answer it anyway.  [Laughter]  Yes, okay.  The slight of hand I did on you today was I've redefined what a school is.  Okay, that's the slight of hand I did which is I've got, you know, I've sat at schools but I've got schools that you know, he's got a school that's got you know social workers inside the school, you know, bababa, right?  The reason I did that is 'cause I think there's a lot of public schools who are doing that.  Like the number of New York City public schools I'm gone in and seen a dentist in the next room from the ELA teacher is amazing.  And so I think we can get there in terms of our definition of school.  So I think that--I'm using the broadest definition of what school is.  And I just really want to contrast that with purely community programs that are after school, et cetera, in terms of driving achievement.  Now whether or not they affect teen pregnancy and those others, I don't know that yet, but that's coming, these kids really aren't old enough, right.  All these kids in the zone right now are in 10th grade.  So they'll be in 11th grade soon and then they will start doing the stuff I did 11th grade, we'll get some good data, okay.  But and--so that's just the reason, just a time constraint, we're waiting.  Yes.
^M01:17:15
[ Inaudible Remark ]
^M01:17:25
>> Oh, it's in the paper in the tables, I don't know.  It is lower than--it's rough--I think it's roughly the same.  I think it's a little lower than the public schools in Harlem but I think we reported in the paper.  It doesn't really matter for our statistical analysis because as soon as you are in--we're fine.  But I don't--we reported in the paper just, you know, age.  I don't know but it's in the paper.  Yes.
>> I don't think we need to clone more Geoffrey Canada but I think we need to talk a little bit more about what he did.
>> Yeah.
>> 'Cause there's not very [inaudible] people in our country that are doing what he did which is he carved out his own community and he identify what he thought were overtime?
>> Yeah.
>> And he was brave enough to say I may raise enough to start my own rule.
>> Yeah.
>> So, I'm not going to have to play by the rules of the game, he has to stay or--
>> Yes.  
>> City or district or [inaudible].
>> Yes, yes.
>> Contra dictates and that's a huge thing
>> Yeah.
>> Because as long as you're a leader that has to lead by the terms of how you get your money.
>> Yeah
>> You can never set it up to [inaudible] be attracted by what you thing is adoptive.
>> Okay, I was with you until your last sentence which is his still has to live by the terms of which he got his money.  There's no such thing as pre money, I noticed that.
>> Yeah.
>> So, [inaudible]--I have lots of fun just doing [inaudible] like we got ideas, I'm going to be okay.
[Laughter]
So, [inaudible]--but I take your point, you got a lot more flexibility than the average person who's over school but not the average person is over a charter school.  So, two points as in, you know, the flexibility to be able to--to develop solutions that you think are important locally is potentially an important thing.  Second which I didn't mention yet in the talk is the--you know, if you look at Caroline's work on charter school as you realize that there's a distribution of treatment effects in the right tale of that distribution, others schools they don't have any in this community programs were getting, you know,  exactly they are the same results but very similar.  If you look at the work that Josh [inaudible] and Tom Cane [phonetic] are doing using charters schools in Boston, you know, they're finding very similar results again, they don't have the community element or the [inaudible] a part of it.  But what they do share is this kind of, you know, a few elements, you know, they had longer school days, they basically--I hate this no excuses thing 'cause--whatever.  But your [inaudible]--they basically educate kids with whatever it takes and so it has a lot of flexibility that you added but again I just why it hates it when I say this I told your men is not all that, she's like you don't know.
[Laughter]
So I [inaudible]--'cause I hate [inaudible] on that, I think it's really important because even people who are his funders think, ow, 'cause just because we got miracle Jeff and this can never be replicated.  I'm like, yeah, but there's lots of--there are other schools that are getting similar results.  And so, I think its too convenient to say this is a miracle.  I think the best thing we can do as researchers is collect data and try to demystify actually what's going on and make it more formulate.
[Inaudible Remark]
>> So, we're going to--that's going to be some of our outcomes that we do.  So let me just say, you know, is--I don't know it is a very complicated question because it--I think I don't actually trust people who go around and say, "Do you want to go to college?"  "Oh yeah."  You know, I--I don't know if I trust them.  Right, but I can tell you there is no expectations they are between--they are no big group differences and expectations.  I think some of the group differences come in and how you actually achieve those expectations.  It's under like the systematic ways, you know, I want to be a scientist but I don't like science, that kind thing.  But I don't think if you look at expectations and at least the date I know about, look at ECLS--the GS--GSS et cetera.  If you look at those you know and if you ask is your kid going to be what degree are they going to get, right?  And an enormous fraction of mothers in the highest poverty places saying their kids are going to be [inaudible], okay?  So, I--I mean, again I don't know what that measures but it is certainly to [inaudible] measures real expectations then that seems to be an expectation.  What I don't know is how the treatment effect of the charter alters those expectations, if it makes them more real, you know, in terms of like giving them on the path edge, you get this, let me get--
[Inaudible Remark]
So, we should talk about this offline but [inaudible] one of the things that I learn a lot and in the last couple of years in terms of trying to get involved in education realize how hard it was and its like one of the things I've learned is like design a survey question.
[Inaudible Remark]
Okay, so like even your question I'm not trying to pick on you but like there is no way that kids are going to be like concrete, I don't know, concrete, what?  
[Laughter]
I really had to designed that, I'm happy like offline after this I talk to you about precisely how you work with that but yet the answer is yes, people try to get it down.  I don't know whether or not the guy you have do your satisfaction.
>> [Inaudible] with that, I'd like to thank Roland for joining us and--
[Applause]