Recovering a schedule nobody wrote down
A Socratic walk-through of recovering a schedule nobody wrote down — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #How do you reconstruct the order a nightly process runs in when no document describes it?
Something runs every night between eleven and five. Roughly two hundred jobs. It has worked for years, and the document describing it either never existed or describes a version retired in 2011.
The obvious move is to read the scheduler. But the scheduler tells you what is triggered, not what is required — it will happily show two jobs in sequence when one merely follows the other out of habit, and it will show nothing at all for the job a person kicks off by hand when a file arrives. So what would count as evidence of the real order, and how would you know when you had enough?
Reasoning it through
REASONING #Begin by separating the question in two, because they are usually asked as one and answered differently. What happened last night is a matter of observation. What must happen, in what order is a matter of constraint. A recovered schedule that only records the first is a photograph; it will be wrong the first time volumes shift.
For observation, the sources are richer than they first appear. Job accounting records give start and end times per step. File and dataset catalogues give creation timestamps, which is often the single most useful signal because it timestamps effects rather than activity. Log files, database audit columns, tape catalogues, and the operator's own console history all carry the same shape of evidence. None of them was designed to document a schedule, which is exactly why they survive when documentation does not — they are exhaust from running rather than a description of running.
Now the constraint half, and here the key inference is about data rather than time. If job A writes a file and job B reads that file, B depends on A, and that is a fact about the programs, not about last Tuesday. You can read it out of the job definitions themselves without running anything. This is the strongest evidence available, and it should be gathered first, because everything else is weaker.
Which raises the harder question: what do you do about pairs that always run in a given order but have no data flowing between them? Ask what could produce that pattern besides a real dependency. A shared exclusive resource — one tape drive, one lock, one connection pool — forces a sequence with no logical need for it. A clock trigger makes two independent jobs look ordered simply because one fires at midnight and the other at one. And a habit can persist for a decade because nobody ever tried the other order.
Can observation distinguish these? Partly. Look at the variance across many nights, not the order on one. If B reliably begins within seconds of A finishing, something is coupling them. If B begins at 01:00 whatever time A ended, B is on the clock and the apparent dependency is an artefact of A usually finishing first. And the case worth watching for is the night A ran long: did B wait, or did B run anyway? A single night where the usual order broke is worth more than a hundred nights where it held, because only the exception separates coupling from coincidence.
The decisive test is perturbation — change the order and see what breaks — and it is usually unavailable in production, which is worth admitting rather than glossing. What you can often get is a rerun environment, a disaster-recovery exercise, or the historical record of nights when something failed, since incident logs are a catalogue of naturally occurring perturbations that someone already paid for.
Finally, what is the output? Not a list. A directed graph, with each edge carrying its evidence and its confidence: proven by data flow, inferred from timing, suspected, or clock-only. Once you have that, the critical path — the longest chain of genuine dependencies — falls out, and it is what actually determines whether the window can shrink. Most of the two hundred jobs turn out not to constrain it at all.
The analogy
THE ANALOGY #Reconstructing a dance from a room full of footprints. The prints tell you where feet landed and, from their overlaps, roughly in what order. But two dancers who never crossed may have moved simultaneously or hours apart, and you cannot tell which from the floor alone. The pairs you can be sure about are the ones where one print is pressed into the other's mark — physical evidence that one came after, not just a plausible sequence.
footprints are a record of one performance, while a batch schedule must hold for every night including the heavy ones, so the reconstruction has to explain not only what happened but what would have happened had a step taken three times as long.
Clarifying the model
THE MODEL #Three refinements.
First, an inferred dependency and a real one deserve different labels permanently, not just during the investigation. If the recovered graph is used later to parallelise the window, the difference between "proven by data flow" and "always ran in this order" is the difference between a safe change and an outage. Confidence is part of the finding.
Second, the absence of a dependency is a claim too, and a stronger one than it looks. Saying two jobs are independent licenses running them together. That claim should rest on positive evidence — no shared file, no shared table, no shared exclusive resource — rather than on nobody having found a link.
Third, the schedule is not only the machine's. Real nightly processes contain human steps: a check before the posting run, a file that arrives from outside by a certain hour, a decision on an exception. These leave the thinnest evidence of all and are the most common cause of a reconstruction that looks complete and fails in practice.
A picture of it
THE PICTURE #How to readthe three inputs on the left are the evidence that survives when documentation does not. The first gate asks the strong question — does data actually flow — and anything answering yes is settled. Everything else falls to the weaker timing test, where the discriminator is whether the second job's start tracks the first job's completion or merely the clock. The red path is the residue: an ordering with no data flow and no shared resource, which stays unexplained until a night when the usual sequence broke gives you a natural experiment. That back-edge is deliberate — this loop does not terminate on schedule, it terminates on evidence.
What became clearer
WHAT CLEARED #The schedule is not hidden in one place waiting to be found; it is distributed across exhaust that was never meant to describe it, and the reconstruction is an inference with grades of confidence rather than a transcription. The crucial distinction is between order that is required and order that merely occurred, and the only cheap way to separate them is to stop looking at typical nights and start looking at the ones that went wrong.
Where to go next
ONWARD #- How to safely test a suspected dependency when production perturbation is not permitted.
- What a recovered dependency graph needs to carry to remain trustworthy as volumes change.
- How to represent the human steps in a nightly process so they are not silently dropped from automation.
Key terms
TERMS #| Term | What it means |
|---|---|
| Batch window | the period available for a scheduled bulk process to complete before online service resumes. |
| Critical path | the longest chain of genuine dependencies, which sets the minimum achievable window. |
| Data dependency | an ordering forced because one job consumes what another produces. |
| Resource contention | an apparent ordering caused by two jobs competing for something exclusive. |
| Proxy evidence | a record produced for another purpose that happens to testify about what ran, and when. |
Every term the collection defines is gathered in the glossary.