Black student achievement plan evaluation and assessment

The study analyzes focus groups and individual interviews with 179 adult participants and 116 student participants during the 2022-23 academic year. Adult participants included principals, administrative coordinators of instruction, teachers, staff members and parents of students within a large school district in California. All of these adults and students were active benefactors and staff members of a Black Student Achievement Plan (BSAP), an initiative created to foster educational equity and college preparedness for schools with over 200 Black students or a document history of academic underperformance for Black students. We examine the implementation of the Black student achievement plan i.e., the adoption of culturally sustaining curricula and instruction, the allocations of funds to hire necessary staff, funding of culturally enriching activity and experiences, etc. The findings suggest that BSAP hires are serving as invaluable support systems for many students and often serve as respite for students experiencing adverse conditions. Findings suggest that BSAP is a crucial part in increasing students’ college readiness and provides students with supports and educational experience e.g., field trips that increase their likelihood for attending college. This research informs districts and college readiness programs about the necessary supports to provide to Black and historically underrepresented students.


Introduction
In February 2021, a California School Board committed to significant investments in Black students' education and emotional wellbeing by establishing the Black Student Achievement Plan (BSAP). BSAP addresses Black students' unique needs related to historic and ongoing inequitable educational opportunities, particularly as it is paired with engaging "the current landscape of local and national advocacy for racial equity have served as the inspiration to act now" (Black Student Achievement Plan, 2021, p.1;Ladson-Billings, 2006). A-G completion, California's required courses for high school college entry, sits among the top of the 16-success metrics identified by the district. BSAP intends to remove barriers to postsecondary completion by championing "high academic performance, social-emotional awareness and management, and a positive cultural identity" for Black students. (Black Student Achievement Plan -Board Amended, 2021).
In January 2022, the school board approved an evaluation proposal submitted by USC's Center for Education, Identity, and Social Justice and the RAND Corporation. The core objective of the BSAP evaluation is to examine implementation of BSAP across school sites to determine the extent to which the BSAP levers improve higher education for career success among Black students. We designed this research to provide actionable formative feedback to inform operations of and implementation changes to the ongoing plan. Our evaluation is driven by the following research questions: 1. How does BSAP improve or hinder Black students' academic success, particularly considering access to diverse representation, high-quality curricula, and culturallyrelevant teaching? 2. How does BSAP improve or hinder students' access to a supportive school climate and mental and social-emotional supports? 3. How does BSAP improve levels of engagement between parents/guardians and schools, Black students' engagement with extracurricular activities, and the presence of community organizations on campus?

Logic Model
The logic model highlights three key levers of inputs: (1) curriculum and instruction, (2) school climate and wellness, and (3) community partnerships. Within each of these levers, we outline more detailed supports designed to palliate the systemic barriers to the pursuit of postsecondary certificates and degrees. Within our research team, we used the model to inform development of our interview and focus group protocols and to indicate which forms of secondary data we would need to collect in order to track trends in outcomes over time.
Our evaluation strategies and research tools are sensitive to how identities such as race, ethnicity, gender, sexual orientation, religion, and disability can highlight systemic issues, inform solutions that create equity, and foster shared values and democratic ideals.

Case Study Method
While we do not outline the full design of the BSAP evaluation plan in this short paper, we explain methods as conducted thus far. We collected data from different groups of high schools for the two primary sides of the evaluation-implementation and impact. We initially selected 15 schools from the 53 Group 1 schools for more in-depth qualitative data collection; we are calling these 15 our case study schools. Case studies utilize a wide range of data types, and allow us to narrow in on how features of an intervention might operate differently in different school environments (Yin, 2018;p. 3). Case study methodology is particularly useful for evaluating educational program implementation, as it allows us to obtain rich data that capture teachers' and students' perspectives on the processes of teaching and learning. Such data can allow for unexpected insights to emerge, and inform future refinement of the program.

1217
Black student achievement plan evaluation and assessment

Case Study Site Selection
The research/evaluation team selected case study schools based on various factors, and in consultation with the BSAP team. For the proposes of this short paper, we analyzed data from 5 high schools from our 15-school sample to resemble the larger group of 53 Group 1 BSAP schools. We used criteria including: racial/ethnic composition of student body, percent of students qualifying for free and reduced-price lunch, graduation rates and percent of students fulfilling A-G requirements, and school model (e.g., traditional, magnet, charter). We matched these criteria across our case study schools and the full sample of 53 BSAP schools, aiming for a sample that would closely mirror the larger group of schools.

Data collection
The 15 case study schools are receiving a greater degree of qualitative attention than the remainder of the BSAP Group 1 sites. In September and October of 2022, our research team visited each of the case study school campuses for a full day, during which time we conducted multiple interviews, focus groups, observations, and gathered documents. Through these various data collection methods, our team spoke to principals, ACIs, BSAP staff, teachers, students, parents, and community organizations. Approximately 54% of the adult participants identified as African American/Black, 10% identified as Latinx/Hispanic, 2% identified as Mexican American/Chicano, 3% identified as Other Hispanic, 2% identified as Asian, and 13% identified as White. Nearly 71% of the adult participants identified as women, 28% as men, and 1% marked Other. All participants have also been assigned a pseudonym to protect their identity. All interviews and focus groups were audio recorded and transcribed. The table below presents all the data collection tasks completed at the sample school sites during Fall 2022.

Diverse representation in learning materials
One dimension of BSAP Lever 1, as illustrated in Figure 1, is to build greater diversity of representation in curricula that Black students are exposed to. Across 11/15 sites, adult or student participants shared that there had been a noticeable increase in culturally relevant books, pedagogy, and educational experiences. Students and parents especially observed the presence of historical Black figures in hallways and classrooms. The reading areas were often filled with books that featured Black protagonists and included a diversity of plot lines that students might relate to. A father of an elementary school child noted that his children were responding well to seeing themselves in the lessons, "They can see themselves in the lesson, they can see themselves on the board, they can see themselves in the books to actually help them and encourage them to actually read and be involved in the classroom." Like other parents at the sites, this father found that the reading material in the classrooms and posters in the hallways had more of a positive impact on the cultural identity of his children than the negative stereotypes they were used to seeing in the media.
One high school principal echoed the importance of providing Black students with positive, diverse representation in their reading material. Describing his approach to aligning his school's curriculum with the goals of BSAP, he said: One of my goals right now is to ensure that every English class…has a culturally relevant novel that every grade level will read…So, what they do is they read the book with a facilitator from [community-based organization], who is our partner, and they go over to their [center], which is right next to our campus, and they perform either a little skit or a vignette or a monologue. So they'll actually act out a scene that really resonated with them.
By bringing in a facilitator from a community-based organization and giving students an opportunity to perform a monologue, this principal is providing a wide range of ways students can engage and make sense of diverse representation in their reading material.
A notable strength of BSAP is its initiative to increase the diverse representation in books, posters, and curriculum that students are exposed to. When the curriculum is implemented, it can significantly correlate with students' sense of belonging on campus and their academic engagement in the classroom. Moreover, other cultures stand to learn and benefit from curricula that discusses Black experiences and history. Another elementary teacher described the enriching nature of culturally relevant curriculum for all students when he said, "It not only impacted my African-American students, but also some of my other students, my Hispanic students…I had one of the Hispanic students who was like, my favorite thing was learning about all the different African-American people this year." Critical opportunities for improvement

Adopt culturally relevant pedagogy in classrooms
In the previous section, we unpacked diverse representation and culturally relevant curriculum as a strength. In the 11/15 instances where sites adapted and taught with the new, Black-centered reading material, it certainly is. However, we found that across 4/15 schools, there still is a lot of progress to be made toward establishing compliance with BSAP's mission to incorporate culturally resonating and sustaining pedagogy in all schools. For instance, multiple teachers at one school felt that the initial rollout of BSAP was rushed and did not allow teachers to incorporate culturally relevant units in an impactful way. A science teacher described her frustration with adding cultural relevance to her lesson, saying: Even though the idea of the unit plan was to integrate it deeply, because of the timing, the roll out and the people, it became something tacked on top. And so as a science teacher, the example I always use is, it's like the Black astronaut, the Black astronaut lesson where you're like, "Okay, be culturally relevant. Here's a Black astronaut. They did this. Isn't that cool?" Move on to the next thing. And that is not an authentic way of empowering our students to feel like they belong in an underrepresented field to understand the history… This teacher expressed discontent with taking a previously constructed lesson and incorporating race as an afterthought. She would rather make the lesson relevant to students' environment in a meaningful way than spotlight the race of the subject or author as a garnish and hope that the students will relate. Teachers shared that the rollout simply did not permit them enough time to produce a curriculum that would be culturally relevant and impactful.
One parent of an elementary student shared that she felt that a lack of Black-centered instruction was derived from non-Black teachers' preferences to center their own respective histories. When asked if she believed the curriculum was relevant to her child, she responded: No, absolutely not. No sir…But sometimes I think that the teachers, they still kind of intertwine to tweak whatever that they want to teach their own kind. So it's basically like the black kids are going to get left out because now you're over here tweaking the curriculum for the other children, but not tweaking it for the black children because they can teach about they [sic] heritage.
This parent believed that teachers might have been asked to teach a certain curriculum, but they may have refused to do so because they were reluctant to prioritize the stories or needs of Black students. As a result, Black children's history remained tangential to many lessons.

Culturally resonant math education
Math education is another critical improvement opportunity. A high school counselor presented site-specific data that spoke to the need for improvement in math education for Black students when she stated, "Last year 23% of our students were either proficient or above in math. That means that 77% were not…only 7% of our Black students met that goal, and zero of them exceeded the standard." She conveyed disbelief that only 7% of Black students were sufficient in math, and suggested providing professional development that was more focused on students' varying learning styles.
In the achievement plan, a local district states that they are committed to providing students with learning opportunities that "validate and affirm who they are" honoring the "cultural heritages, languages, and histories" of its students and their families. The district also lists "dismantling racism in mathematics" as a part of its instructional foci. We found, though, most case study schools (8/15) have not started or are resistant to BSAP's mission to make math curricula more culturally resonant for Black students. One elementary principal shared, "typically the math problems…may not reflect some of the cultures that we have represented." Concerning the implementation of culturally resonant math, a high school principal stated "I think it was a lot to ask… [teachers] were having a hard time fitting in where that made sense." Further, another principal echoed the sentiment of time being a limitation when he shared, "I think it's still a work in progress. It's been a challenge to find time during the school day to do it." Increasing proficiency by way of ensuring equity in mathematics instruction is listed as a key initiative in the district's achievement plan. We found that several schools encountered limitations when attempting to adopt the curriculum. Therefore, we assert that supporting teachers and schools to make math instruction more culturally resonant is a critical improvcment opportunity.

Lever 2: School Climate and Wellness
Implementation Strengths

BSAP staff as support system and advocates for Black students
Across all 15 case study sites, there was evidence that BSAP staff played a vital role in Black students' lives, learning, and experience in school. BSAP staff also demonstrated care and familiarity with students and their lives outside school, allowing them to advocate for students in ways other educators could not. BSAP staff were often like parental figures to students and were very aware of life circumstances that could impact students' experience in school.
Students at one middle school described their relationship with one BSAP staff memberwe will call him Mr. Nathan Blue 1 -whom they spoke with daily. One of the students said he was his "brother, uncle, nephew." Another student responded, "Mr. Nathan Blue is my dad." A third clarified, saying, "He's just like a father figure." One student reassured, "That's my dad. You know, when I call him my dad, he calls me his son. Why? Because me and him have a great, strong relationship and we always share things with each other." Mr. Nathan Blue was not related to any of the students at this school but had built such a strong relationship with students that he became like family to them.
Parents at an elementary school felt similarly about Ms. Williams, one of the BSAP staff members at their campus. Parents described Ms. Williams as "the school" and "part of the village" because she knew all the students and everything about the school to support students and their families. Parents reassured us that she had a high level of care for students and that they knew if their children were with Ms. Williams, they would be comfortable, safe, and happy. Ms. Williams cared so deeply about students that she fostered children that attended the school, and they referred to her as their other mom.
Critical Improvement Opportunities

Language that minimizes advocacy for racial equity
Some ACIs, principals, teachers, and BSAP staff utilized language that communicated that BSAP was a program for only Black students, or a student club, alienating students from the broader campus community. The use of this language further reinforced the idea that BSAP could go away after a period of implementation, and may have contributed to a lack of investment by some teachers. One high school ACI expressed frustration at how schools were adopting BSAP, saying: One of the biggest disdains I have is schools and folks identifying students as BSAP students. I hate it. And I feel like some of us ACI's are guilty of it because we almost made it like a club, like it's a frat, like it's a sorority…We have a shirt that says BSAP. But they like wearing it like all the BSAP kids got the BSAP T-shirt. So it's like a club. So now I'm finding and this is my big disdain. Do not call these students BSAP students. Don't do it. We don't call them GATE-ers and we don't call them ELL-ers and we don't call them SPED-ers because they are being supported by a plan. So why are we call them BSAP students? We should not be identifying students, calling them out or categorizing them by race, language acquisition or level of intellect or ability.
Addressing students by the plan that supports them rather than as students, further minimizes BSAP's advocacy for racial equity and fosters and environment that otherizes students. This is equivalent to defining students with disabilities by their disability, calling them disabled students. Such that when teachers, school staff, and leaders reference students only within the context of their disability, not only is that a reductionist view of who they are and a slight or insult, but also it minimizes how and why an institution should respond to the full humanity of these students. In this case, an educational institution is responsible for addressing more than a student's disability. A parallel construction exists for ESL (English as a Second Language) students, who are now referred to as ELL (English language learners). We suggest referring to students with BSAP support might be a more useful reference. Additionally, using such language indicated that these students were BSAP's responsibility, and may have decreasing some staff members' sense of responsibility in supporting them.

Districtwide and schoolwide hiring and staffing challenges
Non-BSAP leaders and staff at 9/15 schools struggled to fully understand the roles of BSAP staff members and how to best leverage their support. The lack of clarity at times created tension between BSAP staff members and other staff members at the school sites. To complicate matters further, several schools had unfilled BSAP positions and incomplete teams to carry out the plan with limited support.
Although the plan includes several school-level roles with specific goals and duties at Group 1 schools-i.e., Administrative Coordinator of Instruction, Pupil Services & Attendance, Psychiatric Social Worker, and Restorative Justice Teacher-once on site, BSAP staff found themselves performing additional job duties that were not part of the original plan. Several BSAP staff members shared that principals and other school leaders often asked them to perform unrelated tasks and duties. BSAP staff members obliged because they cared about Black students' well-being, even when these tasks were beyond the scope of their jobs. Two BSAP staff members at a high school explained how difficult it was to get time for planning programs and initiatives when administrators consistently called them to perform other duties. One explained their challenges, saying: We get pulled in two different directions. I think a lot of that is definitely tied to the sense that overall, it's not like they would look at us as working on "just BSAP", like this is what they do. It's more so like you're [a restorative justice teacher], you're basically the "mini dean". You're school climate, you're campus aid. We need you to go get these kids from these classes and bring them over to the mini dean. And that's not really what we're supposed to be doing.
BSAP staff members at other campuses also felt like the disciplinarians at their schools; some wore yellow "campus aide" vests and walked through the hallways making sure students got to class on time.
The lack of understanding of the plan and BSAP staff members' roles and responsibilities prevented many of them from providing better support for Black students. Additionally, it created additional work for ACIs to support their BSAP staff members and communicate with principals and school leadership regarding how they should utilize their BSAP staff members at their schools. Tina, the ACI at Promise Elementary, explained: I spent more time this year pouring into my team, making sure my teams knew what their roles were, making sure they understood our 16 indicators, our success indicators, and how those relate to their jobs. And how you know, every day, this is what this is your focus, you know, as it relates to the black students and with the principals as well, making sure that they understand that their PSA isn't just a PSA.
Similarly, BSAP staff at several sites mentioned that school leadership often did not recognize that some staff members were specifically designated for BSAP and not for the rest of the school community. Several BSAP staff members performed duties far beyond their scope, so ACIs needed to continually refocus them to ensure that the plan's goals were understood and met.

Conclusion and Recommendations
While the above findings are preliminary, and will continue to develop over the course of our research, they do point to some specific and actionable recommendations for LAUSD leadership, school leaders, and school-level BSAP staff members.
We provide these recommendations not only to evaluate the program and make suggestions that will better serve Black students, but also to bring awareness to the legal liability associated with a racially hostile environment. Above all other recommendations below, we want to highlight a critical first step for school and district leaders: to better understand what constitutes a racially hostile educational environment and how it may open the individual, the school, as well as the district to legal liability. We included this early in the report to demonstrate its urgency. School leaders would benefit from assessing ways that their own schools may have hostile environments for Black students, and working with other staff members to address these concerns.