Sunday, January 13, 2013

Since using student test scores to evaluate teachers is back in the news

Lately I've been reading a lot about using student test scores to evaluate teachers.  This has been a long standing problem for me.  Not that it is not appropriate, but that the skills to do it don't exist in K12 education.

There is a belief that you can do simple things, like the fraction of students that pass a certain standard from one year to the next, and be done with it.  That would be true, if what you were dealing with was a well-defined experiment with random assignment to treatment groups  -- but we don't have that.

Analysis of treatment effects and program evaluation works on a continuum.  When you have strong experimental controls, the statistics are easy, mean of this vs mean of that with a t-test.  The more you deviate from strong experimental controls, blind evaluation and random assignment, the harder the stats get.  K12 standardized tests are very far away from strong experimental controls.

Last year, the Parkrose board of education sent the letter below against the adoption of ORA 581-022-172, 1725 on teacher evaluation, not because we thought the idea was bad, but because we thought K12 going to screw it up.

This is not a training issue.  I can't teach someone that thinks a chi-squared test is pushing the envelop how to do this in a week.  This is a job for professionals, not education professionals, statistical and program evaluation professions, not an a education professional that knows stats, a real statistictician and program evaluator.

I spent years trying to explain to ODE that they were understating the uncertainty on OAKS test scores by a minimum of a factor of four because they had violated an assumption of the estimator they were using but it fell on very deaf ears.

Here is the letter.

The Parkrose Board of Education would like to comment on the proposed OAR 581-022-1723, 1725 Teacher and administrator evaluation and support.  We believe that a change may help in understanding true student achievement, especially, the view that state assessments make up the some of the core measures of student achievement.

The proposed rules are almost entirely permissive and can be made part of teacher and administrator evaluation when properly performed.  The list includes many items that are extremely difficult to perform well.  In particular, we suggest excluding the use of student assessments from 581-022-1723 (1) (2) (a).

Student data, like many tools, can be very useful in the hands of a trained expert.  Too often student performance data is treated like the end result of a randomized control trial with one school or one teacher having a randomly assigned collection of students.  Students are not randomly assigned.  Parents choose neighborhoods.  Students are sorted into classrooms based on need, and students move from school to school over the course of a year.  This confounding self-selection means that you can not simply look at the data, but must make adjustments for those factors.

Those familiar with program evaluation and quasi-experimental design know that it is possible to use student data to evaluate the efficacy of teachers.  The problem is that the statistical expertise does not exist in individual school districts, Educational Service Districts or at the Oregon Department of Education.

Until the practitioners understand, at a minimum, when they should use propensity score matching, or multinomial logit for classroom selection, or linear hierarchical models for evaluating both a school and teacher simultaneously, we do not recommend using student data.  Without understanding these basics, teachers and administrators may be improperly identified as excellent or failures.

Our hope would be that we could find the best possible measures, using the best statistical models available to us for the good of all students in the state of Oregon.


James Woods PhD
Parkrose Vice Chair

Ed Grassel
Parkrose Chair

Friday, December 21, 2012

Grant Incentives in Higher Education

About five years ago I went through a tooth pulling exercise with my dean over adding incentives to my contract that rewarded me with bringing in grant money.  My argument was that I had no incentive to get grants since all I got out of it was extra work without extra time to do it or extra money to compensate me.  The kind of grants I get don't allow me to get equipment  I could use on other projects or anything like that.

There was a lot of tooth pulling.  I eventually got it in the contract but the extra incentive was so small it was hardly worth it.  That said, the school picked up on the idea and it is now in its second year.

The email below is from the research office.

In 2011 Vice President Jon Fink created the "PI Incentive" program to
encourage faculty to engage in sponsored research.  The program returns a
portion of the F&A earned to Principal Investigators, as long as they
earned at least $5,000 for PSU.  Incentive amounts were distributed last
fiscal year based on the amounts earned in FY 2010 and FY 2011.

We are pleased to announce that RSP has funded this program for the third
consecutive year.  Money was transferred to PIs individual "PIN" index
codes in November based on F&A earned last fiscal year.  Please contact
your department chair for details.

Monday, December 17, 2012

Report to Oregon Legislature on Textbook Costs

Here is the final product of the HB 4058 Work Group on textbook costs.  We took a very different approach than other states, which decided to make things available and to encourage other things.  We have two current initiatives.  The first is to make sure that faculty have instructional material assignments in on time so students and book stores can shop for the best prices.  The second is a grant (bribe) fund to encourage faculty to reduce instructional materials costs with the help of an expert.

I've shopped this with the Portland State University Provost and will head out to other institutions as soon as I get my head above water.

Tuesday, November 13, 2012

OSBA Conference: Decisions With Data

This one should have been titled, "A Love Letter to the Hermiston School District".  Over the years the Hermiston school district has generated more good ideas and better implementations of good ideas than any other school district in Oregon.  Hats Off.

How should I put it.  They created a way of looking at student data that is an absolute dream.  It is everything that I said was needed and what I was told was impossible.  We should be sending teachers and administrators to Hermiston to take a look at what they do, how they do it, and figure out how their culture makes this all possible.

Notes and audio below.

  1. Dashboard point of view
  2. Question:  How can you do custom hypotheses tests.  Outside of fancy programming, data pulls for analysis.  It is complicated a lot of times.  Requires an expert.  So if you are a small district, you can get the reports and maybe filter access.
  3. So the mobility is transactional system...that which leads into the warehouse.
  4. Do you create an anonymous micro data sample for research purposes?  No they don’t create this.
  5. Says the it can handle proficiency based.
  6. Their dashboard can do gender things too...and very good dashboard.
  7. Robin DeLoach is the one that made this all work.  She is at Willamette ESD.  Get her connected.
  8. Pearson power is for the info to parent.

Monday, November 12, 2012

OSBA Conference: ORVED

This was about the Oregon Virtual Education Charter School.  I wanted to see if this was ready for Parkrose kids and I had high hopes.  They have not done anything about teacher and course evaluation.  I asked about this last year when they made a presentation to the OSBA board.

This is not ready for our kids.

Notes and audio.

  1. Can enroll on any date.
  2. The rest looks like the same as the other presentation
  3. Cooperative agreement is that the student can still come back to the school to take classes at the base brick and mortar.
  4. Teachers, part time, decide how many they want to teach and which classes.  Up to 59 students.
  5. Platform I have not heard of
  6. Revenue sharing model.
    1. Fee for course
    2. Enroll is the alternative  cost 3600, Sponsor gets 292 and the remainder goes back to district where the student lives.
  7. Questions
    1. Teacher contract
      1. $10.42 per week per student
    2. Not much with evaluation part
    3. Religious based none, but that may be a demand.
    4. Long discussion on this.  There may be a problem.

OSBA Conference: Foundations Q&A

The second seminar was on foundations.  Parkrose has an educational foundation, but it is small.  A long time ago I was involved in getting that restarted after it was abandoned for a few years.  It has grown a lot since those days but we have not hit the point were it has a paid employee and where we are getting the corporate donations.   I want to know how to get there.

The big takeaway was that the foundation needs to move away from funding teacher projects to funding their own initiatives.  Maybe it has to partner with All Hands Raised to get that done?  I really think that is the way it will happen.

Here are the notes and audio.

  1. Karina Intro
    1. 68K to  1.5M with 240 employees.
    2. Alignment with district plans
    3. Doing more than just teacher grants
    4. They operate the after school programs.
    5. They have a limited base, high poverty and not a lot of corporate because of seat of government.
    6. they now manage volunteer services
    7. They raise money for professional development.
  2. My question.  Specific strat for 3,500 district mostly residential with FRL near 80% from 60K budget to next level and no employees.  Added difficulty neighbor district foundation tends to dominate the area with, “all students in all neighborhoods’ line.
  3. Make sure sup and school board members are not involved with ask to vendors.
  4. They take a cut of pass through then 10% or so.  SK takes 15%.
  5. Best
    1. Sustainable
    2. Credible
    3. Transparent
  6. Five Pieces
    1. Infrastructure -- treat it like a corporation
    2. Board -- pick a team not people that are breathing.
    3. Image -- hire to make you look credible.  Spend money on the marketing material
    4. Stories
    5. Development
  7. Two models to go to paid
    1. District pays for exec director but has problems with staff after that
    2. No relationship to district
    3. District lets contract for foundation to provide some services.
    4. exec is district employee and foundation reimburses part

OSBA Conference: Blended Learning Notes

The next few posts are just a compilation of notes from seminars at the Oregon School Boards Association Conference that was held on November 9th through the 11th.  I had to do some introductions as being an OSBA board member but was able to go to a few that I had interest in.

What I got from this is where the state of the art is in K12.  I was a little taken aback that they were waiting for new computers with webcams when you can buy webcams for $20 and that they were not making their own brief instructional videos.  Making those brief videos was the first thing I did and I discovered the most valuable thing you can do for student success.

What follows is just a public dump of my notes and an audio file.

  1. Background 10AM Friday.
    1. Clackamas online is a charter.  Web academy
    2. They have a middle college too
    3. They are OSBA members.
    4. Appointed board
    5. All staff members are employed by North Clackamas school district.
    6. Adhere to limits on funding.
    7. And serve a note to district that was incurred earlier.
    8. Odessyware is the online curriculum development platform.
  2. Blended
    1. Online  platforms and on site.
    2. They have classrooms in commercial space.  Always been curious why we don’t do this more.
    3. bimonthly meetings in students home.
    4. They also have onsite tutors.  Are tutors online too?
    5. They have the councilors go with the students.  The Cs don’t stay in place.
  3. Work on math
    1. they are hooking videos to the courses
    2. Why are you taking a final?
    3. You can have online proctoring if you need.
    4. Sounds like they were stopped by not having a $20 piece of equipment.
  4. Home school mom
    1. Saves her time grading
    2. Still allows her to help with education.
    3. Also saved her money. She does not have to purchase it anymore
    4. OH, they provide a laptop for each student.
    5. Weekly progress report including how much time they spend on each assignment.
  5. The student view
    1. The oldest child was the real go getter.  BTW Mom was on the board.
  6. What about students that can be home schooled at home for whatever reason.
    1. Answer focused on special education.

Saturday, November 3, 2012

Portland State University: New Academic Titles

Portland State is going round and round about new academic titles.  They are adding all sorts of new ranks for people like me, non-tenure line faculty, that now that we do most of the work.  The OARs that define the new titles are here.  There are now instructors, lecturers and clinical faculty, and, well the list goes on for a lot of letters.

The Portland State University senate will be discussing this at the meeting on November 5th.  You can find the meeting materials here. I would speak on this issue, since I feel like I may be screwed in the process, but I have a Parkrose board meeting to attend.  Thankfully, I know someone in the senate that I can pass letter to -- Bob Liebman.

I passed him these two letters and told him that he could use one or both of them.  The first is earnest the second is satirical.  While I could put 'Sincerely' at the end of the first, the best I could manage was 'Very Respectfully' on the second.  That is what you use when addressing a superior.  I hope they get that jab.

Letter One

I've read the OAR and it allows the institutions to use these ranks as appropriate to hiring and retention of their faculty members as it relates to their institutional mission.  (3)(5) allows us to keep our current system of academic titles if we wish.

I urge us not to make changes to our current system.

This new taxonomy takes a very simplistic view of both what fixed-term faculty do and why some people are on a tenure line while other are not.

Some very good people are not in tenure-line positions simply because of the market was bad when they finished degree, or because they have limitations on were they can move.

Many tenure-line positions are held by those that have retired in place at an early age, or by those that have ignored one of the three components of their jobs  -- either research, teaching, or service.

Being tenured or on a tenure-line is not a perfect indicator of quality. These new ranks would make fixed-term status a much more permanent state, like a negative tenure.  They will leave a mark on a resume that can't be erased.

Moving from an academic title and then being, apparently, demoted to lecturer will lead to questions about why.  Was there an ethical violation?  Did it involve a student?

This ties the individual to the school, making it harder to get a tenure-line appointment at another school or even industry appointment.  That is my primary concern, it takes temporary second-class status and makes it permanent across all institutions.

My second concern is that the taxonomy views fixed-term faculty as much more specialized into teaching, research and service than we often are.

My contract is written as being the same as a tenure line faculty member with the all the same research, teaching and service requirements.  I do have a significantly lower salary and a higher teaching load, but tenure is the only difference.

It feels like this is a ploy to pay me for one of functions and get the benefits of me doing the others for free.
If I am a lecturer and I publish, do I put PSU down as my institutional affiliation or do a create a non-profit to hold that honor? If I get a grant, will the indirect costs be forwarded to an institution of my choice or will PSU still collect those?  I think PSU is very likely to claim the credit and the cash even though those two things are not part of my job description.

I would be just uncomfortable if PSU had declared that they would only pay me for every third minute even though I would clearly work during the intervening two.

Under this new system of ranks, it is unclear where I would fit. Will I be transferred from the instructor to lecturer ranks depending on course load?  Will an excess of grants or publications suddenly turn me into research faculty?  I have dual appointments in Economics and Urban studies, will I be a lecturer in one and research faculty in the other?

Finally, I would have to leave the school before I was demoted to one of these new titles.  I don't want to sit through an interview and have to explain that I didn't sexually harass a student and that they new title was an administrative decision that save the school a little money.

James Woods

Letter 2

The OAR allows the institutions to use these ranks as appropriate to hiring and retention of their faculty members as it relates to their institutional mission.

I believe these changes don't go far enough, and I encourage the administration to press for further modifications to OAR 580-020-005. The distinctions made in the non-tenure track faculty titles should be applied to tenured faculty members.

This is clearly necessary given market conditions. There is a surfeit of Ph.D. in all fields. PSU has traditionally restricted itself to hiring tenure line faculty that had strong research potential and could shape their disciples. We can now hire candidates that can excel at teaching, service and research.

This is a clear benefit to students. Placing the careers of our faculty above those of our students is wrong. We should be granting tenure only to those that can show excellence in all three areas and not to those that show excellence in research and only adequacy in teaching or service.

The tenure review remains a noisy signal of quality. After tenure some faculty members have shown themselves to be deficient in one or more areas. New titles are necessary to clarify the relationship between these individual and the institution.

The traditional academic ranks should be reserved for those tenured faculty members that have very high current research output, strong current service commitments, and some of the highest graduate and undergraduate teaching commitments in their departments.

Tenured faculty that rarely teach graduate course and do minimal research would fall into the tenured instructor sequence. A current Full Professor would have the title 'Tenured Senior Instructor II' should their research output drop below an acceptable level. Those that continue to teach at the graduate level would of course be a 'Tenured Senior Lecturer II'.

Tenured faculty that have devoted themselves almost purely to research, buying out several courses, would enter the ranks of the Tenured Research Faculty. An Associate Professor that bought out three classes through grants would then be a 'Tenured Research Associate Professor'.

Tenured faculty that focus on service, e.g., our department chairs and senate committee members, with below average teaching or research output would be placed in the Tenured Professor of Practice ranks.

These new title would enable the institution to make contractual distinctions between the post tenure performance of faculty members and compensate them appropriately. This should help alleviate some of the tension in some departments and would go a long way toward identifying the true causes of the disparity and improve the educational experience for ourselves and, most importantly our students.

I urge this body to press for these changes.

Very Respectfully,
James Woods

Friday, October 26, 2012

What the HECC: The Costs of Textbooks

I know that some people think I just sit around doing nothing, but really, I'm busy.  One of my side projects was to head a work group (HB 4058 Work Group) on textbook costs and make a report to the legislature on how to reduce textbook costs to college students.

That project is in mid swing.  The Higher Education Coordinating Commission (HECC) will be voting on November 1st to send the executive summery on to the legislature.  This is still in draft mode but it is definitely a public document.

The draft of the report is here.