Read / talk / write / Five stages

The ORACLE Ethics Audit

Examine five fictional incidents before a school decides whether ORACLE could be used responsibly.

Work with a partner. Read the three files, talk about what the school should do, then answer three short questions. Keep your answers for the school debate.

All rooms
Start here

What to do

Read the three files and talk with your partner. Then write three short answers. Brief notes are fine.

AI words in this glossary
Algorithm
A set of steps a computer follows to calculate something or make a recommendation.
IF/THEN rule
A written instruction: if a certain situation occurs, then take a stated action.
Learning from examples
Finding patterns in sample data instead of writing a separate rule for every case.
Generative AI
A system that makes new text or images from learned patterns; its claims still need checking.
AI winter
A period when enthusiasm and funding fell because AI had not met high expectations.
Bias
A pattern of unfair results, even when nobody meant to treat people unfairly.
Safeguard
A specific action that reduces a risk, such as a check by a named person.
Appeal
A chance to challenge a decision and have the evidence checked again.
Pair workYour answers save in this browser. Print a copy whenever you need one.
Investigation progress0 of 5 stages complete

The invented revision guide

ORACLE writes a useful-looking guide but makes a claim that the teacher cannot verify. Decide what to keep and what to check.

Three files

File A
ORACLE draft | not shared with pupils

ORACLE drafts a Year 9 revision page about rain and rivers. It says heat from the Sun can turn water into vapour and cooling vapour can form clouds. It then claims, 'Every drop of rain that lands on the ground immediately reaches a river.' It credits a 2021 article called 'Where Every Raindrop Goes' but gives no author, link or page number. The polished layout makes the page look ready to share.

File B
Teacher's checks

The school's geography textbook supports the explanation about vapour and clouds. It also says some rain flows over land into streams and rivers, some soaks into the ground, and some later evaporates. The teacher cannot find the cited article in the school library catalogue or by searching for its exact title online. That does not prove it exists nowhere, but it means the draft's source has not been verified.

File C
Department decision note

The department needs revision material this week. One teacher wants to keep the useful explanation after checking it; another prefers the existing textbook. Drafts can be regenerated or shortened after review, and the shared folder does not distinguish reviewed versions from new ones. The team has not agreed who approves changes or which version pupils will receive.

Should pupils use this AI-written revision page?

Now imagine: A teacher checks the page. Someone then asks ORACLE to shorten it, and the wrong rain claim comes back. It still says 'checked by a teacher'.

Optional multiple-choice questions
What does the textbook say happens to rain on land?
What was the result of the teacher's citation search?

Answer all three questions to continue. Brief notes are fine.

Need a nudge?

A source can sound convincing without being findable. Notice that the guide has not yet been shared.

The opportunity algorithm

A school selection score uses apparently neutral inputs. Test whether its results give everyone a fair chance.

Three files

File A
Enrichment programme score

The fictional school offers a limited number of places on a computing enrichment programme. ORACLE awards points for a home computer, optional online practice, after-school club attendance and past teacher recommendations. The form does not ask about income or family background. Staff say the score is quick and rewards interest. A pupil who shares a device at home can still be keen and capable, even if they cannot complete every optional task.

File B
Pilot review | 40 fictional applicants

Of 20 applicants with reliable home computer access, 14 score above the invitation line. Of 20 without it, four do. The groups' average class-work marks are similar. Some high-mark pupils with fewer out-of-school activities are missed. This small pilot raises a question; it cannot show that every score is unfair or identify one cause of the gap on its own.

File C
Selection panel discussion

The panel has fewer places than applicants and wants a consistent process. Practice and club attendance may show interest, but transport, caring duties and device access affect participation. A proposal removes the direct home-computer question while retaining those other inputs. Reviewing class work or hearing applicants' explanations would take longer. The revised score has not yet been tested.

Optional: read more about this topic

How can bias enter training data?

Raspberry Pi Foundation, Experience AI | 2-minute focused read: find 'Bias in, bias out'

An example learning activity shows how a limited dataset can produce flawed or biased predictions.

Investigate: How is the selection case similar, and what is different about the opportunity score's inputs?

Read at Raspberry Pi Foundation, Experience AI (opens a new tab) ↗

If this site is blocked, use the short description and the files.

How should the school choose pupils for its computing programme?

Now imagine: The school stops asking whether pupils have a home computer. It still gives points for online practice and after-school clubs. One pupil shares a device and has no transport home after a club.

Optional multiple-choice questions
Which invitation counts were recorded?
What did the pilot report about average class-work marks?

Answer all three questions to continue. Brief notes are fine.

Need a nudge?

Compare the two invitation counts and the similar class-work marks. What else could affect the score?

The schoolwork monitor

ORACLE asks for far more information than a schoolwork check-in seems to need. Decide what is necessary and what is not.

Three files

File A
Proposal | why collect data?

The school wants to spot when a pupil may need help with class work so a teacher can offer a check-in. The draft plan starts with recent assignments and school-platform sign-ins. ORACLE also requests school messages, search history, location through the day, private social-media posts, friendship information and past safeguarding records. The plan gives no reason why those extra sources would help with the stated schoolwork check-in.

File B
Privacy review

A reviewer asks who could see the records, when they would be deleted and how a pupil could correct an error. The plan has no answers yet. Even school-platform sign-ins can mislead: a pupil might work offline. The reviewer suggests trying the least intrusive schoolwork information first, checking it with a teacher, and refusing unrelated personal information.

File C
Meeting note | two viewpoints

A member of staff worries that fewer records might miss someone needing help. A pupil representative worries about tracking location and personal posts. Another pupil completes assignments on paper rather than online. Staff have limited time for checking alerts. The group must decide whether automated signals add useful evidence or mainly create more checks and unnecessary monitoring.

Optional: read more about this topic

Collecting only the data you need

UK Information Commissioner's Office | 2-minute focused read: first two sections

The ICO explains data minimisation as collecting only the personal data needed for a clear purpose. This guidance is written for organisations, so use the short explanation above if the full page is difficult.

Investigate: Which requested ORACLE data seems unnecessary for the stated schoolwork check-in purpose?

Read at UK Information Commissioner's Office (opens a new tab) ↗

If this site is blocked, use the short description and the files.

Should the school use ORACLE to spot pupils who need help with schoolwork?

Now imagine: A pupil does their homework on paper and rarely signs in online. ORACLE says they are not doing their work.

Optional multiple-choice questions
What is the stated purpose of the proposal?
Which questions are unanswered in the draft plan?

Answer all three questions to continue. Brief notes are fine.

Need a nudge?

Keep the purpose in mind: a schoolwork check-in. Which data is directly relevant, and which controls are still missing?

The unexplained punishment

A high score leads to a trip recommendation. Can the pupil and staff find out why before anyone makes a final decision?

Three files

File A
Trip screen | recommendation, not decision

ORACLE shows a fictional pupil a behaviour-risk score of 82 out of 100 and recommends not allowing them on a school trip. A teacher wants the trip to be safe for everyone, but the screen does not show which events or records produced the score. The trip decision has not been made. Staff are due to discuss it tomorrow.

File B
Request to check the record

The pupil asks why they might miss the trip. A support teacher finds that two late arrivals are listed as behaviour incidents, but cannot tell whether those entries caused the high score. The pupil says a bus delay explains the late arrivals and wants the school to check. The current process has no clear way to show the reasons, correct the entries or appeal the recommendation.

File C
Governor's question

The governor wants staff to consider safety without treating a number as the final decision. The trip leaves on Friday. The next scheduled appeal meeting is the following week. A family could provide information about the bus delay before Friday, but staff have not requested it. No named person currently has responsibility for resolving the disputed records before the trip decision.

How should staff decide whether this pupil can go on the trip?

Now imagine: The pupil can ask staff to look at the decision again. But the meeting is booked for next week, after the trip has already left.

Optional multiple-choice questions
What does the screen reveal about the score of 82?
What has the support teacher found?

Answer all three questions to continue. Brief notes are fine.

Need a nudge?

The trip decision is still open. What must be checked and explained before it becomes final?

Who is responsible?

Another incident shows how one wrong record can pass through several unchecked steps. Trace who could have stopped it.

Three files

File A
Incident timeline | fictional case

On Monday, ORACLE links one fictional pupil to another pupil's record and recommends excluding the first from an activity. On Tuesday, a staff member sends the recommendation to a family without checking which record it used. On Wednesday, the family questions it and staff discover the mismatch. The exclusion was not carried out, but the family has spent two days worrying about a claim that was not based on their child's record.

File B
Warnings and missed checks

The developer's trial log had already noted that the matching tool could link two different records. The school had not tested it with its own made-up test records before use. A staff member assumed a high score meant the record had been checked. The contract named no school contact responsible for reviewing important recommendations or pausing the tool after an error.

File C
Repair meeting

The developer offers a software fix. Staff say the activity exclusion was never carried out, but the family received an incorrect recommendation and worried for two days. The contract names no person able to pause the tool. Nobody has checked whether other records were affected. The school must decide what evidence and responsibilities would be needed before restarting, rather than treating a code change as a complete repair.

What should happen before the school uses ORACLE again?

Now imagine: The developer fixes the software. The family still has not been told the claim was wrong, and nobody at school can stop the next wrong message being sent.

Optional multiple-choice questions
What happened before staff found the mismatch?
What had the developer's trial log recorded?

Answer all three questions to continue. Brief notes are fine.

Need a nudge?

Follow Monday, Tuesday and Wednesday. Which warning, test and human check could have changed the outcome?

Pair case file

Your answers for the approval hearing.

All stages are shown, including unfinished answers. Return to a stage to edit or print this case file.

Try the optional companion escape mission → Continue to the other room →

Project ORACLE / The ORACLE Ethics Audit

Year 9 paired investigation / CodeBash / Stages complete: 0 of 5

Stage 01 / The invented revision guide

Should pupils use this AI-written revision page?

What we found

What the school should do

New problem

A teacher checks the page. Someone then asks ORACLE to shorten it, and the wrong rain claim comes back. It still says 'checked by a teacher'.

Would our plan work?

Stage 02 / The opportunity algorithm

How should the school choose pupils for its computing programme?

What we found

What the school should do

New problem

The school stops asking whether pupils have a home computer. It still gives points for online practice and after-school clubs. One pupil shares a device and has no transport home after a club.

Would our plan work?

Research note

Research lead: Raspberry Pi Foundation, Experience AI | https://www.raspberrypi.org/blog/experience-ai-how-research-continues-to-shape-the-resources

Stage 03 / The schoolwork monitor

Should the school use ORACLE to spot pupils who need help with schoolwork?

What we found

What the school should do

New problem

A pupil does their homework on paper and rarely signs in online. ORACLE says they are not doing their work.

Would our plan work?

Research note

Research lead: UK Information Commissioner's Office | https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/childrens-information/childrens-code-guidance-and-resources/age-appropriate-design-a-code-of-practice-for-online-services/8-data-minimisation

Stage 04 / The unexplained punishment

How should staff decide whether this pupil can go on the trip?

What we found

What the school should do

New problem

The pupil can ask staff to look at the decision again. But the meeting is booked for next week, after the trip has already left.

Would our plan work?

Stage 05 / Who is responsible?

What should happen before the school uses ORACLE again?

What we found

What the school should do

New problem

The developer fixes the software. The family still has not been told the claim was wrong, and nobody at school can stop the next wrong message being sent.

Would our plan work?

Project ORACLE / Case file for the school approval hearing.