Field note / AI & Automation
The Best Python Script Is the One You Can Run Again Tomorrow
Make repeatable Python catalogue jobs with explicit inputs, rejected rows, dry runs and deterministic output, then define what a successful rerun means.
The first catalogue script usually has a clear purpose: read a supplier file, transform a few columns and produce an import file. The second run introduces the questions that determine whether the script is useful operationally. Which input did it use? Did it overwrite anything? What happened to rejected rows? Can the same job be repeated after an interruption?
My work with catalogue and CSV automation makes those questions more interesting than the cleverness of the transformation itself. A script becomes dependable when another run has a predictable meaning. The techniques below describe a practical design approach; they are not claims about a particular supplier’s private data or measured business results.
Automation becomes infrastructure the second someone depends on running it twice.
Make the input contract explicit
Start by defining the expected columns, identifiers, encoding and value conventions. A file extension cannot establish any of those. Two CSV files can represent the same apparent table while disagreeing about delimiter, decimal separator or the meaning of an empty cell.
Python’s CSV documentation provides readers, writers and dialect handling for working with these variations. Those tools handle parsing mechanics. The application still needs to decide which dialect is accepted and which business rules each parsed field must satisfy.
For a supplier catalogue, treat identifiers as strings unless the contract explicitly says otherwise. Leading zeros can carry meaning. Parse quantities and prices under a known convention rather than letting a permissive conversion silently reinterpret them.
Reject a missing required header before processing the file. If a supplier renames a column, an early contract failure is easier to diagnose than a successful job that emits empty values for every product. Keep the original input unchanged so the result can be traced back to what was actually received.
Separate transformation from delivery
A script is easier to test when reading, transforming and publishing are distinct steps. The transformation can accept normalized records and return accepted records plus explicit rejections. Delivery can then decide whether and how those results reach the target system.
This separation makes a dry run meaningful. A dry run should execute the same parsing and validation rules as a real run while stopping before external changes. If it uses a simplified path, it may provide false confidence about the operation that will actually happen.
Write generated output to a new location before replacing an approved artifact. Where the filesystem and deployment method support it, a final rename can reduce the chance of readers observing a partially written local file. For remote systems, define the publication mechanism separately rather than assuming local filesystem behavior applies.
The CSV interface article discusses why this boundary matters. A convenient file format does not remove the need for an explicit agreement between producer and consumer.
Define what idempotent means here
For a pure transformation, running the same code with the same input and configuration should produce the same meaningful output. For a script that updates a remote catalogue, repetition should not create duplicate products or repeat an unintended side effect. These are related but different guarantees.
Choose the identity used to match records. A stable supplier reference scoped to that supplier may be appropriate; a product title is usually a poor substitute. Record whether the job creates, updates, skips or rejects each item.
If the remote system supports a revision or idempotency key, understand its actual semantics before relying on it. If it does not, the script may need to inspect existing state or maintain a durable mapping. A local “completed” flag alone may not resolve uncertainty after the remote update succeeded but the response was lost.
Avoid advertising exactly-once behavior unless the entire workflow supports it. A more honest contract may be that reruns converge on the intended state while preserving a record of uncertain operations for review.
Rejected rows are an output
A script that logs “some rows failed” leaves an operator with little useful work to do. A rejection report should identify the source row, relevant identifier, failure category and the field that needs attention. It should avoid dumping unrelated sensitive data into logs.
Distinguish correctable input problems from infrastructure failures. A missing product identifier may reject one row. An unavailable destination may require stopping delivery for the whole batch. Treating both as generic exceptions makes recovery unnecessarily difficult.
A rejected record should leave a repair instruction, not a mystery.
The policy for partial success must be explicit. Some workflows can publish valid records while holding invalid ones. Others require the entire batch to remain unchanged until every record passes. The right choice depends on the business agreement, not on which loop is easiest to write.
Include counts that reconcile: read, accepted, rejected, unchanged and delivered where relevant. Counts are useful diagnostics, but they do not prove semantic correctness. A mapping error can affect every row while leaving the totals perfectly balanced.
Keep configuration outside hidden assumptions
A script should not require editing source code to select a new input file or destination. Use explicit arguments or a documented configuration file, with validation at startup. Secrets belong in an appropriate secret store or environment boundary, not in committed configuration.
Defaults should be safe and visible. If a job can update production, requiring an explicit destination is often clearer than silently selecting one. A dry-run flag should be reflected in the final summary so the operator cannot confuse a preview with a completed delivery.
Version mapping rules alongside the code when they define business behavior. A category mapping changed in an untracked local spreadsheet can make two identical-looking runs produce different results without an explanation in history.
Record the configuration identity and code revision in the run summary. Do not record secret values. The goal is reproducibility: enough information to reconstruct the relevant conditions without creating a credential leak.
Determinism makes differences useful
Stable output ordering and consistent serialization make review easier. If records arrive in a different order but their content is unchanged, a generated diff should not suggest that the entire catalogue changed. Sort by a stable key where order has no business meaning.
Keep volatile run metadata separate from the business artifact when possible. A timestamp inserted into every generated row can make unchanged data appear different on each run. A sidecar report can retain operational timing without contaminating the comparison.
Determinism also helps tests. Small fixtures can cover missing headers, duplicate identifiers, malformed values and known mappings. Expected outputs should demonstrate the contract rather than mirror the implementation line by line.
When a rule intentionally changes, review the resulting difference. A deterministic artifact makes that review more informative because unrelated variation has been removed. It is a practical aid to human judgment, not merely an aesthetic preference for tidy files.
Exit status is another part of the interface. A scheduler needs to distinguish a completed run from a failed one, even when both produced a report file. Define whether rejected rows make the job fail, produce a warning status or require approval. Keep that policy consistent with the summary shown to a human operator so automated monitoring and manual interpretation do not disagree.
Test the interrupted run
The most revealing test may stop the job between transformation and delivery, or after some remote updates have succeeded. Restart it and examine what happens. Does it duplicate work, skip uncertain records or incorrectly report success?
A recovery procedure should explain how to identify the last confirmed state and what evidence is needed before resuming. For a small file-based workflow, that may be an approved output artifact and a run manifest. A more complex integration may need per-record delivery status and reconciliation against the destination.
Do not add a database solely because recovery sounds serious. Choose the smallest durable state that supports the actual failure model. Conversely, do not rely on an in-memory list if the job must survive process termination.
The script is ready for tomorrow when a person can supply a known input, inspect the planned changes, run it, understand the summary and repeat it safely. That operational clarity is often worth more than another clever transformation hidden inside the loop.