Filming the Shop Floor and Analyzing It with AI | Consent, Camera Setup, and the Analysis Pattern

Hello, this is Ryuta Hamamoto from TIMEWELL.

This is the fifth article in our FDE practice series. So far I have covered asking, watching, being there, and turning fragments into structure. This time, as an extension of watching, I want to describe a method our engineers have recently started using: filming work on site, with permission, and analyzing the footage afterward with AI.

Observation means seeing in the moment. So things get missed, things get miscounted, and you learn nothing about the time you stepped away. Video can be reviewed as many times as needed. And now that machines can read video at a practical level, most of the reviewing can be handed to them. But what you must decide before filming outweighs what you learn after, and that has been the lesson of the past year. This article covers how to obtain consent, how to film, the analysis pattern, and how it differs from work analysis in contract engagements. If you first want to check how far your own team can handle shop-floor data, take the AI literacy check.

Decide before filming: consent is the premise of the analysis, not a formality

The most important thing first. Before filming a workplace, obtain the consent of the people who will appear. That is not a formality; it is the premise of the analysis. Footage filmed without consent is not only legally problematic; it is useless as analysis, because a person who does not know they are being filmed and a person who knows and has agreed do different work.

Here is the legal frame to the extent practice requires. Footage in which a face is identifiable is personal data under Japan's Act on the Protection of Personal Information. When you collect it you must specify the purpose of use as concretely as possible and notify the person or make it public, and you cannot use it beyond that purpose.¹ The same applies to filming employees. The camera-image utilization guidebook compiled by Japan's Ministry of Internal Affairs and Communications and Ministry of Economy, Trade and Industry organizes the points to consider when using camera images in business, stage by stage: planning, acquisition, storage, use.² It was written with store and street cameras in mind, but its ideas, stating the purpose, notifying the people who will appear in advance, limiting retention, handling third-party provision, apply directly to filming work on site.

Before filming, we explain the following six items in writing to the people who will appear and to their manager, and obtain their agreement. Purpose: work analysis in order to build a working artifact. Scope: whose work, which tasks, which hours. Retention: deletion once analysis is complete. Viewers: which of our engineers will see it. Exclusions: not for evaluating individuals, attendance, or personnel decisions. And masking of faces and name badges before any AI reads the footage. The sixth is less a legal requirement than our own commitment: strip anything that identifies a person before a machine reads it.

One more thing that always happens. When you raise filming, the person feels they are about to be watched. Naturally. So we explain the purpose as "to build something that reduces your work," and we show the first analysis to the person first. For example: "There were 47 minutes of approval waiting in a day. Next week we'll build a screen that shortens that." That one sentence turns surveillance into analysis. Filming whose results are not returned to the person is surveillance.

How to film: fixed camera, screen recording, and time you do not film

We combine three things.

First, a fixed camera. A small camera on a tripod, placed diagonally behind the person, the same position as in observation. The frame takes in the screen, the papers on the desk, and the motion of getting up from the seat. Faces are kept out of frame. Designing the frame this way cuts masking work later. The advantage of a fixed camera is that it records without an observer present: the time the observer stepped away, the rush before lunch, the close before the end of the day.

Second, screen recording. The person records their own PC screen, operated by them. A fixed camera cannot read the text on the screen; screen recording captures exactly which system, which screen, in which order. Customer data appears here, so scope and storage are governed more strictly than for the fixed camera. Our default is that screen recordings never leave the customer's machine and the analysis runs inside the customer's environment.

Third, time you do not film. Phone calls, visitors, breaks, and any task the person asks not to have filmed. Deciding the unfilmed time in advance lets the person relax and do the filmed work as usual. Try to film everything and everything becomes work for display.

Filming length is two sessions of the same task on different days. As I wrote last time, being watched changes how people move. The second session is closer to normal than the first. We almost never film a third; once the difference between two sessions is visible, that is enough, and beyond it the burden on the person outweighs the gain.

What AI analyzes and what people keep

What do you do with the footage? This is what changed this past year. The major multimodal models now accept video as input, answer questions about specified timestamps, and produce transcriptions of events.³ We use that capability to have a machine fill in the five observation columns.

We hand AI four analyses. First, a timestamped transcription of actions: "10:03:12 opens approval screen," "10:03:40 picks up paper form," "10:04:05 speaks to the next desk." Of the five columns a person wrote in observation, the machine fills time, action, and screen. Second, counting: total approval wait, number of screen switches, returns to the same screen, hand movements between paper and keyboard. Things the eye could not count well come out exactly. Third, transcription of speech from the fixed camera's audio. Fourth, candidate mismatches: the machine reads the interview transcript and the video transcription side by side and lists contradictions such as "explanation said section-manager approval; footage shows the assistant manager stamping."

People keep three analyses. First, meaning. A machine can count "switched screens three times," but whether those were hesitation, habit, or a constraint of the system is known only to someone who was there. The unease column is written by a person. Second, a ban on summaries that read as evaluations. Ask a machine to summarize footage and you get sentences beginning "worker A's efficiency is." We do not use them. The unit of analysis is the task, not the person, and outputs begin "the approval wait in this task is." Third, no unmasked footage goes into a machine. Faces and badges are removed, and customer names are removed from screen recordings, before anything is read.

This division makes the difference from observation clear. In observation, a person writes all five columns while watching, and things get missed. With video, a machine writes three columns and the person concentrates on the unease column. Last time I wrote that the eye for mismatch grows by writing that column every day; video is a way of spending human time on that column alone.

Turning the footage into next week's working artifact

Let me be clear about why an FDE films. Not to produce a work-analysis report. To decide what to put down next week.

The numbers that come out of the footage are material for the Friday session described in the KJ-method article. For example: "47 minutes of approval wait in total," "212 screen switches a day," "18 hand movements between paper and keyboard an hour." On their own, these are not a working artifact. On Friday, laid out beside the interview transcripts and observation records, grouped and named, they reveal a structure such as "the approval wait is long because the approver never looks at the screen, so it travels on paper," and next week's build is decided. Video is a way of adding numbers to the material for the groups.

And the effect of what you built is measured the same way. The week after the working artifact goes in, film the same task again and count the same things. You get differences such as "total approval wait went from 47 minutes to 12" or "screen switches from 212 to 96." That difference is the outcome for the person, and for us the basis for deciding whether the pattern is worth returning to the product. Not a technical metric such as 95 percent accuracy, but a metric of the person's day.

How it differs from work analysis in contract engagements

Filming work and analyzing it has a long history in industrial engineering. It was systematized as motion study and used in manufacturing as an improvement tool. Contract-development and consulting engagements sometimes film as part of business analysis too. Judged by motion alone they look the same, so here is the difference.

Aspect	Work analysis in a contract engagement	FDE video analysis
Purpose	Visualize the current state and report improvement proposals	Decide next week's working artifact and measure its effect afterward
Unit of analysis	Process step, standard time	The task and the waits, switches, and back-and-forth inside it
Exit	Report and improvement proposals	A working artifact and a pattern returned to the product
How effect is measured	Estimate at proposal time	Film the same task the following week and take the difference
Handling of footage	Often retained by the analyst	Never leaves the customer environment; deleted after analysis

The biggest difference is the exit. Contract work analysis ends with a report and proposals. Whether the proposals get implemented is a matter for a separate contract. FDE video analysis is one process that runs through putting down a working artifact next week and filming again the week after to take the difference. Analysis, implementation, and measurement turn in the hands of the same engineer.

The other difference is how effect is measured. A contract proposal carries an estimate: "expected savings of N hours a year." An FDE produces not an estimate but the measurement from the following week. Whether 47 minutes became 12 is known by filming. That measurement is what grounds the decision to return a pattern to the product. The test a16z gives for the real FDE model, whether engineering effort declines on mature accounts,⁴ can only be answered on the strength of accumulated measurements.

What we do and where it stops

Let me be honest. We have only recently started this, and we plan for it not to go smoothly. Three failures are easy to fall into. Getting the fixed camera's frame wrong so that the paper form that matters is out of view. The person forgetting to stop the screen recording so that time we promised not to film gets recorded (in that case we delete that portion in front of them). The machine's transcription counting more screen switches than actually happened. All three are preventable by deciding the frame, the stop procedure, and who reviews the transcription before filming.

There are limits. Video can analyze only what appears in it. The judgment in the person's head, what the caller said, what the neighbor whispered: none of it is on the footage. So video does not replace observation and interviews. Observation writes the unease column, video adds the numbers, interviews get what is in people's heads. All three together become the material for the groups.

And the weight of consent. Filming goes deeper into someone's work than observation does. We do not treat consent as obtained once; we confirm it verbally at every session, show the analysis to the person first, and tell them when the footage has been deleted. Skip that loop and you will not get consent for the next session. Consent decides not only the quality of the analysis but whether you can enter the next site at all.

Summary

Where observation is seeing once, in the moment, video is seeing again as often as needed. Before filming, commit in writing to six things, purpose, scope, retention, viewers, exclusions, and masking, and obtain the agreement of the people who will appear and their manager. Film with a fixed camera for the screen and the desk, screen recording for system operations, and decide the unfilmed time first. Hand the machine the timestamped action transcription, the counting, the speech transcription, and the candidate mismatches; keep meaning, the ban on evaluative summaries, and masking with people.

The purpose of filming is not a report but deciding next week's working artifact and filming again the week after to take the difference. Whether 47 minutes became 12 is known by filming. That measurement grounds the decision to return a pattern to the product.

If you want to build an engineering team that can handle shop-floor data, or are looking for a partner to film and analyze your own workplace and turn it into something that runs, take a look at the WARP program or talk to us. The next article covers turning the nouns and verbs collected so far into a structure of meaning that AI can traverse: an ontology.

Footnotes

1. Guidelines on the Act on the Protection of Personal Information (Personal Information Protection Commission, in Japanese). The definition of personal data, specification and notification of the purpose of use, and the restriction on use beyond purpose are from the general guidelines ↩

2. Camera Image Utilization Guidebook ver. 3.0 (IoT Acceleration Consortium, MIC, METI, March 2022, in Japanese). The stage-by-stage points to consider are from this guidebook ↩

3. Video understanding (Gemini API documentation, Google). Cited as one example of video input, timestamp-based queries, and transcription generation; it does not indicate which model we use ↩

4. The Palantirization of everything (Marc Andrusko, a16z, January 16, 2026) ↩