Syllabus

Instructor

Course details

  •   Tue/Thu
  •   September 25–December 15, 2023
  •   2:00–3:20 PM
  •   Anteater Learning Pavilion 1100
  •   Slack
Please read the syllabus!

This page has important information about the course, including structure, assignments, grading, and student hours, among many others.

Even Snoop Dogg wants you to read the syllabus!

Course objectives

In this course, you will learn how to evaluate the effects of educational programs. You will develop three broad skill sets:

  1. Applying theory to develop and assess causal statements.
  2. Using modern tools for measuring causal effects.
  3. Being able to judge, use, and learn with evaluations in practice.

At the end of the course, you will have developed a plan to evaluate an education program of your choice.

Specifically, at the end of the course, you should be able to do the following.

  • Describe how a program expects to achieve its outcomes (its “Theory of Change”).
  • Develop a measurement plan along with that Theory of Change.
  • Explain what is needed to make causal statements.
  • Identify ethical issues and limits of evaluations.
  • Understand the main analytical tools used to measure causal effects.
  • Analyze data using these tools.
  • Assess the quality of program evaluations and aggregate their findings.
  • Know what makes evaluations valuable for education practitioners.
  • Become curious and confident in consuming and producing evaluations.

Expectations

In this course, you will be expected to

  • participate in class,
  • do all readings, and discuss them outside of class,
  • write four data-analytic memos,
  • take one in-class exam, and
  • complete three milestones, which lead up to one final evaluation plan.

Connections to other classes

EDUC 10 (Educational Research Design) and EDUC 15 (Statistics for Education Research) are prerequisites for this course. That said, we will also dedicate a class to brush up on everyone’s foundational statistics skills. The course does not assume any prior knowledge of statistical software.

While we will learn about a mix of quantitative and qualitative approaches, the course emphasizes quantitative methods more strongly. In contrast, EDUC 156 (Introduction to Field Methods in Education) provides a greater focus on qualitative methods.

Finally, in the future, I will also teach EDUC 159 (Experimental Research Methods) at UCI. There will be some overlap between our class and EDUC 159 but, as the course title suggests, the latter will focus on experimental methods only.

My promise

I promise you can succeed in this course.

This course is all about practical application. You will notice “How will I ever use this?” is a question I consider for every class.

Now, you will have to tool up. You will hear about weird new concepts, use some unfamiliar maths, learn new software, and approach practical questions from a new perspective. Initially, some of this might pose challenges. Most everything will be new to you. But, it’s perfectly normal not to grasp concepts immediately and make mistakes during your initial encounters. This is how you acquire new skills and build a strong toolkit.

Many parallels come to mind. Learning to ride a bike. Learning a new language. Learning how to use a hammer (and when to pick a different tool). Building muscle through training. All of these demand time, grit, and—oftentimes—productive failure. All of them are easier with friends, through apprenticeship, and by working with a coach. Similarly, if you find yourself struggling with some of the course content, that’s a good sign. When you do, take a breather, talk to classmates, ask questions, or contact me for assistance.

Course structure

This syllabus reflects a plan for the quarter. Deviations may become necessary as the quarter progresses.

Flow of the course

The course will follow a basic flow each week, with small differences if an assignment is due or not.

  • Monday: Complete reading/watching course material for Tuesday, submit your reactions by 11:55 PM. Weekly assignment posted.
  • Tuesday: Course meets in-person.
  • Wednesday: Complete reading/watching course material for Thursday, submit your reactions by 11:55 PM.
  • Thursday: Course meets in-person.
  • Sunday: Submit your assignment by 11:55 PM (if due)

Aside from our class meetings on Tuesdays and Thursdays, there are no other lab meetings or sections.

Canvas

Our Canvas page is here. There are three main purposes for Canvas:

  1. Assignments and grades. I post assignments. You submit your solutions. I post your grades.
  2. Additional materials. I post PDF copies of additional reading materials. I post PDFs of my slides.
  3. Announcements. I may send out reminders or make announcements.

Everything else is on this website here and on Slack, not on Canvas.

It is very important you update the Canvas settings and notifications to ensure you receive important messages and announcements. Click on “setting” to add another email address and/or a cell phone number to receive notifications. Click on “notification” to configure how you receive Canvas notifications.

Slack

Our dedicated Slack channel is here. Watch out for an email invitation from me. You will submit your reading reactions on Canvas, not on Slack. However, feel free to ask additional questions over Slack.

Course materials

Books

There are three official textbooks for the class. All three are available digitally.

Articles, book chapters, and other materials

There will also occasionally be additional articles and videos to read and watch. When this happens, links to these other resources will be included on the reading page for that week. For example, we will review three chapters from the following book.

R and RStudio

This course will expose you to R—one of the most popular, sought-after, and in-demand statistical programming languages. Armed with the foundation of R skills you’ll learn in this class, you’ll know enough to be able to find how to analyze any sort of data-based question in the future.

You will use RStudio as the main program to access R. Think of R as an engine and RStudio as a car dashboard—R handles all the calculations and the actual statistics, while RStudio provides a nice interface for running R code.

R is free, but it can sometimes be a pain to install and configure. To make life easier, you can (and should!) use the free RStudio.cloud service, which lets you run a full instance of RStudio in your web browser. This means you won’t have to install anything on your computer to get started with R! RStudio.cloud is all you need for our course.

RStudio.cloud is convenient, but it can be slow and it is not designed to be able to handle larger datasets or more complicated analysis. Over the course of the quarter, you’ll probably want to get around to installing R, RStudio, and other R packages on your computer and wean yourself off of RStudio.cloud. This isn’t necessary, but it’s helpful. You can find instructions for installing R, RStudio, and all the tidyverse packages here.

There are tons of online resources to help you with R and RStudio. Two of the most important are StackOverflow (a Q&A site with hundreds of thousands of answers to all sorts of programming questions) and RStudio Community. Also, ChatGPT will sometimes fool you but, as it was likely trained on and (is quickly replacing) StackOverflow, it is quite good at writing R code.

An the resources page, you will find a long list of additional resources. All of this is way too much information, but certainly good to have as a backup for any of your future work after our course. And, if all of this sounds overwhelming, don’t worry. Once we get to it, I will point you to all the materials you do need.

Assignments and grades

Participation

You should actively participate in class, do all the readings, and discuss them outside of class. To that end, you should submit reactions to at least 14 of our readings on Canvas. Also, you may (and should) engage with questions you see posted on Slack. However, it’ll be up to me to decide whether a Slack comment engages with the assigned reading and adds to the discussion, meaningfully.1

Data-analytic memos

Over the quarter, you will write four “data-analytic memos” (DAMs). Each DAM will task you to use your newly-acquired evaluation skills. Many of the questions will be similar to what you may be asked to do on a future job (whether as an education practitioner, analyst, or researcher). You will also work with real-life education datasets.

You will complete these memos at home, individually. You may (and should!) work together on the DAMs, but you must turn in your own answers. You cannot work in groups of more than three people, and you must note who participated in the group in your assignment.

Plus-delta

Twice during the quarter, I will ask you to complete a survey. At the beginning of the course, your answers will help me better understand your personal background and interests. Towards the middle of the course, I will ask you for feedback on how to improve the course and make changes if needed. Finally, at the end of the quarter, please leave a course evaluation on EEE.

Personally, I like the plus-delta approach. Start by telling me what you like. These are things I should maintain and build on. Then, provide opportunities for improvements. These should be action-oriented suggestions as to how I can provide a more effective course.

This feedback is very important to me, and I appreciate your input.

Exam

You will also take one in-class exam, which serves as an important mid-quarter check of your understanding. This exam will be closed book and without access to the internet. I will provide more information about the exam as it approaches.

Your evaluation project

For the final course project, you will develop an evalution plan for a new education program that excites you. That could be any new educational program a school district, state, charter organization, or NGO has established recently (or plans to establish shortly). No matter the topic, you will formulate a key research question and design a state-of-the-art evaluation plan to measure the program’s effects.

Throughout the quarter, there will be three milestones related to the evaluation plan. During the final exam period you will finalize the plan and submit your document to me. You are not asked to execute the evaluation (but you may after the course).2

You will work in small groups of two or three (at the beginning of the course, I will randomly assign you to a group). This structure will be most similar to the professional work environments to which you will soon be transitioning. Evaluations are also far easier to design and implement when you have the benefit of a team with whom you can generate and critique ideas. The goal is to develop a professional project that you can use to showcase the skills that you have gained to potential employers.

Calendar

You can find a detailed class calendar on the schedule page.

Here, I am providing a brief summary of the assignment dates.

Category Release date Due date
Survey 1 Mon, Sep 25 09:00AM Sun, Oct 01 11:55PM
Milestone 1 Sun, Oct 08 11:55PM
Milestone 2 Sun, Oct 15 11:55PM
DAM 1 Mon, Oct 16 09:00AM Sun, Oct 22 11:55PM
DAM 2 Mon, Oct 23 09:00AM Sun, Oct 29 11:55PM
Survey 2 Mon, Oct 30 09:00AM Sun, Nov 05 11:55PM
DAM 3 Mon, Nov 06 09:00AM Sun, Nov 12 11:55PM
DAM 4 Mon, Nov 13 09:00AM Sun, Nov 19 11:55PM
Exam Tue, Nov 21 02:00PM Tue, Nov 21 03:20PM
Milestone 3 Sun, Dec 03 11:55PM
Evaluation plan Sun, Dec 15 11:55PM
Course evaluation Sun, Dec 15 11:55PM

All reading reactions are due by 11:55PM on the day before class. All times are in Pacific time.

Writing

Your assignments will involve a fair amount of writing. You should not take this writing lightly; an important part of your undergraduate training is to convey arguments clearly. You are expected to review your assignments for typos and grammatical errors before submitting them. Aim for simple and precise language in active voice. As you do, feel free to use related online services (such as Grammarly or the Hemingway App). You may also take full advantage of the various on-campus resources to help improve your writing, including the Writing Center.

Grading

You (the student) and I (the instructor) should care the most about what you learn, not what numerical/letter summary of that learning you get at the end of the quarter. So I would love to not have grades at all, but unfortunately we humans are very good at procrastinating on our good intentions when there is no incentive not to. Thus, we have grades to help solve this commitment problem and to encourage you to put effort into learning the course material.

Here is how each of your various course contributions and assignments contribute to your overall grade. For group assignments, all group members will receive the same grade. All other contributions are graded individually.

Category Percent of final grade
Participation, reading reactions 14%
Data-analytic memos (DAMs) 30%
Exam 25%
Milestones 1-3 2% each
Your evaluation plan 25%
Grace policy

When calculating the final DAM portion of the overall grade, I will drop the lowest score and use the remaining scores. Thus, if you have an emergency that forces you to miss one submission, your grade will not be severely affected.

All grades will be posted through Canvas.

Bump-up policy: I reserve the right to “bump up” the grades of students who have made valuable contributions to the course in the in-person course and discussions/Slack. This also applies to students who show tremendous progress and growth over the quarter.

Course policies

Be nice. Be honest. Don’t cheat.

We will also follow UCI’s Code of Conduct.

Attendance and participation

Attendance and participation are important to your success in this course. You are expected to come to all classes prepared—having read the material—and ready to discuss the content.

Missing a class has the potential to put you significantly behind. Still, life happens and, therefore, you may miss one class without contacting me (except for the midterm exam).

If you test positive for COVID-19, hunker down and focus on getting better! Please do not come to class. If you have an absence due to such an unavoidable health or emergency situation, you should message me before class starts. Similarly, if you have to miss a class due to religious reasons, please remember to message me before the affected class starts.

You are expected to arrive before the start of each class to allow us to begin on time. Unexcused absences and late arrivals will be considered in your course participation score.

If you have to miss the midterm exam, you will need to take a similar make-up exam during UCI’s final exam period (I would assign a date and time). If at all possible, try to avoid this. There at least three reasons. First, I deliberately scheduled the class’ only exam during a period in which you do not have to worry about other exams. Second, the make-up exam may also include the additional content we will have covered after the midterm exam. Third, the midterm exam is meant to prepare you for your final project—you would miss an important opportunity to check your knowledge before finalizing your evaluation plan.

Staying focused during class

Evidence from multiple randomized experiments indicates that students who take notes on their laptops or tablets learn less than those who take notes using pen/pencil and paper. They are also more likely to adversely affect their peers’ learning and grades (see Prof. Susan M. Dynarski’s summary of the evidence). Therefore, using a laptop or tablet is prohibited during class.

There are only three exceptions:

  1. I may prompt you to take out your laptop (e.g., to complete a survey).
  2. You may use a tablet that lays flat on your table, with a stylus pen (i.e., without typing on a keyboard).
  3. You have an accommodation plan requiring a tablet or laptop.

Similarly, cell phone use is prohibited during class (unless I ask you to take out your phone). Please stow your phone away.

Finally, no eating is allowed during class. If you eat during class, you are likely to disrupt the classroom dynamic. You may bring a water bottle or coffee/tea in a covered container.

Let’s stay focused!

Late assignments

You should budget enough time to submit all course assignments well ahead of each deadline. Late assignments will not be accepted, regardless of how late they are. No extensions will be granted due to religious holidays, sickness, or any other reasons. Instead, I will disregard your assignment with the lowest score. I suggest using this option cautiously, just in case something happens, you foresee needing time to observe a religious holiday, or any other issue that may keep you from submitting your assignment on time.

Academic honesty

Violation of the UCI Academic Senate Policy on Academic Integrity will result in an F in the course and possible disciplinary action.3 Violating the Academic Integrity Policy is unacceptable, devaluing the teaching and learning experience for the entire community. All violations will be formally reported.

When it comes to academic integrity and generative artificial intelligence (AI), I follow the approach of UCI’s Composition Program. That is, I assume that work submitted by students—all process work, drafts, low-stakes writing, final versions, and all other submissions—will be generated by the students themselves, working individually or in groups. The following would be considered violations of academic integrity: if a student has another person/entity do the writing of any substantive portion of an assignment for them, which includes hiring a person or a company to write essays and drafts and/or other assignments, research-based or otherwise, and using AI affordances like ChatGPT. I make an exception for statistical programming—you may write and debug your code using AI-supported tools (such as ChatGPT or GitHub Copilot).

Recordings

I will not record our in-person classes. Also, you are not allowed to record the class, unless you have an accommodation plan that requires a recording.

Drop policy

Following UCI’s policy, no drops are allowed after the second week of instruction.

Doing well

A personal note

If you tell me you’re having trouble, I will not judge you or think less of you. I hope you’ll extend me the same grace.

You never owe me personal information about your health (mental or physical). You are always welcome to talk to me about things that you’re going through, though. If I can’t help you, I usually know somebody who can.

If you need extra help, or if you feel like you’re behind or not understanding everything, do not suffer in silence. Talk to me! I will work with you.

Student hours

Please watch this video:

Student hours are set times dedicated to all of you (most professors call these “office hours”; I don’t).4 This means that I will be waiting to come by to talk to me—either remotely or in-person—with whatever questions you have. This is the best and easiest way to find me and the best chance for discussing class material and concerns.

It is easiest to meet with me online. Make an appointment with me here—the confirmation e-mail will contain a link for a Zoom meeting (the default). If you prefer an in-person meeting, (1) schedule a Zoom meeting first, then (2) contact me via e-mail or Slack, at least one day before our meeting.

On-campus resources

Life at UCI can be complicated and challenging. Fortunately, UCI offers a wealth of resources to support you.

Please take the time to seriously consider the following (confidential) resources available to you.

One of the (very few) things I regret from college is that I did not ask for enough help. I’m sure at least one of the above can help you do better—do reach out.

Safety

I will listen to and believe you if someone is threatening you.

If you are in immediate danger, call 911. UCI police provides a 24-hour safety escort (949-824-5223) and a 24-hour non-emergency line (also at 949-824-5223). Safe this number.

Please double-check and make sure you are signed up to zotAlert.

If you are experiencing sexual assault or misconduct, domestic or relationship violence, stalking, or retaliation, please report it to me and I will connect you to resources. The University of California, Irvine is committed to creating and maintaining an environment in which all persons who live, work, and learn in our campus community can be free of sexual assault, sexual misconduct, relationship violence, stalking, and retaliation. Every member of the University community should be aware that sexual violence is prohibited by law as well as our University policy and will not be tolerated.

At any time—even if you are in doubt, or would like to know more about what you can do to prevent cases—do not hesitate to reach out to UCI Care and the Sexual Harassment/Title IX Officer (SHO).

Accommodations and accessibility

If you wish to request academic accommodations you may do so through the Disability Services Center (DSC). To obtain services, please register with DSC and complete the DSC Intake Form. You should then make an appointment with me during the first week of class to discuss any accommodations that need to be made.

I will do everything possible to make sure our course materials are accessible. Please let me know if you know of ways to improve on this further. Even a short message on, say, the colors I use on a slide can help a lot.

Petr

Once you have read this entire syllabus and the assignments page, please click here and e-mail me your favorite picture of Peter the Anteater.5 Extra credit if it’s animated.

Peter the Anteater giving a thumbs-up

Footnotes

  1. “Yey” won’t do.↩︎

  2. Executing the evaluation might require review from UC Irvine’s Institional Review Board (IRB). We will learn about the ethics of evaluations as part of our course.↩︎

  3. So seriously, just don’t cheat or plagiarize! If you have not already, review this website.↩︎

  4. There’s fairly widespread misunderstanding about what office hours actually are! Many students often think that they are the times I shouldn’t be disturbed, which is the exact opposite of what they’re for!↩︎

  5. Petr stickers are great. Fun facts count, too.↩︎