Storage experiment · Raspberry Pi target · isolated host execution
Replacing a File: Visible Now Is Not Proven Durable
Your logger writes a new snapshot, replaces its old file and then reports an error. Is the old snapshot still there? This experiment performs real file operations in disposable directories so you can observe the answer without risking an existing log or pretending that a process exit reproduces a power cut.
1. Define the three questions
Atomic visibility means another process opening a pathname sees a complete old or complete new file after a successful replacement. It does not make two files a transaction. Durability asks whether the change survives a later failure. A synchronization acknowledgement records that the operating system (OS) returned from a requested synchronization operation. These are related questions with different evidence.
The intended target is a Raspberry Pi 4 Model B running Raspberry Pi OS Bookworm64 and CPython 3.11. CPython is the reference Python interpreter. The supplied code uses the documented Portable Operating System Interface (POSIX) family of file operations, not a Pi peripheral library. It accepts no destination path on its command line and creates only new temporary directories. No General-purpose input/output (GPIO) wires or external power switching hardware are required.
The author executed the actual program and tests on macOS27.0.1 with CPython3.12.14, not on the Pi target. Directory synchronization returned successfully on that host. A read-only probe records the actual temporary-filesystem type and block-size units; these describe the host, and no result is labelled a Pi microSD result. A micro Secure Digital (microSD) card contains a controller and storage behaviour that a Python read-back cannot fully characterize.
Use a supported 5.1 volt (V), 3 ampere (A) Universal Serial Bus Type-C (USB-C) supply for a Pi4. Keep normal power connected throughout this lesson. It does not instruct you to pull power from a running computer, build a relay, alter the card partition table or test a valuable logging directory. The useful outcome is a precise software failure record, not a storage reliability certificate.
2. Trace names and file objects
An application programming interface (API) is a software function contract. Here fsync requests synchronization, while mkstemp creates a uniquely named private temporary file; its octal0600 permissions allow only its owner to read and write. A pathname such as state.json is a directory entry that refers to a file object. A file descriptor is an open process handle for that object. Renaming the new object over the old name changes which object a future open resolves to. An already-open reader may continue to read the old object through its existing descriptor.
The temporary file is created in the same directory as state.json, keeping it on the same filesystem. Python's successful os.replace is an atomic rename under the documented POSIX contract; a move between filesystems may fail and is not replaced here with a copy-and-delete fallback. A partially written temporary file is therefore never deliberately installed as the destination.
The directory is private and trusted, with one writer. This is not a defence against an attacker changing paths between checks, a concurrent configuration editor, a network filesystem with different semantics or a multi-file database. The replacement creates a new file with the temporary file's private permissions; it does not preserve the old file's owner, permissions, extended attributes or hard-link relationships.
3. Separate data and namespace synchronization
The program creates a temporary file with mkstemp, writes all bounded bytes, asks fsync to synchronize that file, closes its descriptor, replaces state.json, asks fsync to synchronize the containing directory and compares a fresh read-back with the intended bytes. fsync is the POSIX file-synchronization operation. There is no buffered Python file object here; the program uses os.write and handles a short write by continuing from the returned byte count.
File synchronization and directory synchronization serve different purposes. The Linux manual explicitly distinguishes synchronization of file contents from synchronization of the directory entry naming that file. If directory synchronization fails after a replacement, the new bytes may already be visible. Reporting the old file as safe, or automatically restoring it, would hide the uncertain outcome.
create temporary file in destination directory
write every byte → fsync(file) → close(file)
os.replace(temporary, state.json)
fsync(directory) → fresh read-back comparison
The success label is deliberately long: OS_SYNC_CALLS_RETURNED_AND_READBACK_MATCHED. It describes completed calls and observed bytes. It does not claim that a future power loss cannot lose them. Storage firmware, volatile caches, filesystem implementation, kernel behaviour and the actual supply remain part of any physical durability assessment.
4. Observe an error after the change
Logical event stages with actual isolated author-host exit observations, not a measured time axis or power-loss simulation. Editable SVG; full-size PNG.
The experiment exits a child process immediately after writing, after file synchronization, after replacement and after directory synchronization. It uses os._exit(71), which terminates that process without running its Python finally blocks. The parent and operating system continue running. An exit code of71 is only a deliberate stage marker, not a measured failure mode of a card.
In the author's isolated run, new opens read OLD in the first two cases and NEW in the last two. A pending temporary file remained in the early-exit cases until the enclosing test directory was removed. These observations establish what this running host saw after those process exits. They do not establish what a reboot, kernel panic or loss of electricity would recover.
Three further cases raise a returned error after file synchronization, replacement and directory synchronization. The error type reports visibility and durability as unknown and the experiment inspects the pathname separately. A new file visible after an error is a legitimate ambiguous-commit outcome: the operation may have taken effect before its caller learned that something failed.
5. Calculate only what was observed
The example snapshots are bounded JavaScript Object Notation (JSON) records. JSON is a text format for named fields; it does not make storage transactional. OLD and NEW each contain31 bytes with different sequence values and message strings. A Secure Hash Algorithm 256-bit (SHA-256) digest distinguishes the intended byte sequences during the test. A matching digest is not authentication against a malicious writer or a guarantee about later reads.
The test limit is65536 bytes, or64 kibibytes: one kibibyte is1024 bytes. The writer loops until the sum of returned positive write counts equals the intended length. A zero, negative, malformed or excessive returned count is rejected. For example, a600-byte payload written in three-byte chunks needs200 successful write calls; the supplied short-write test exercises that loop using actual files.
64 × 1024 = 65536 byte maximum
600 bytes / 3 bytes per returned short write = 200 writes
successful read-back bytes == intended bytes
No durability percentage, mean time between failures or maximum synchronization latency follows from eight finite cases. The functions can block inside the operating system; a finite number of cases is not a hard completion deadline. Subprocess cases have an application timeout, which can end a hung child but still does not emulate a physical power interruption.
6. Complete isolated experiment
Download the complete program or the source-and-test archive. Extract the archive together, then run the program with no arguments. It refuses a destination path, uses fresh TemporaryDirectory contexts and removes only their own contents. The report is printed after those directories have closed.
python3 file_replace_lab.py
"""Isolated file replacement experiment; OS acknowledgements are not power-loss tests."""
from pathlib import Path
import hashlib
import json
import os
import platform
import subprocess
import sys
import tempfile
class ReplaceFailure(RuntimeError):
def __init__(self, phase, replace_attempted, cleanup_errors, cause):
self.phase=phase;self.replace_attempted=replace_attempted
self.cleanup_errors=cleanup_errors;self.cause_type=type(cause).__name__
self.visibility='UNKNOWN_INSPECT_SEPARATELY'
self.durability='UNKNOWN'
super().__init__('replacement failed at '+phase+'; no rollback or durable-state claim')
def snapshot_bytes(sequence, message):
if type(sequence)is not int or not 0<=sequence<=1000000:raise ValueError('SEQUENCE')
if not isinstance(message,str)or len(message)>200:raise ValueError('MESSAGE')
return (json.dumps({'sequence':sequence,'message':message},sort_keys=True,separators=(',',':'),ensure_ascii=True)+'\n').encode('ascii')
def publish_snapshot(directory, data, hook=lambda stage:None, ops=os):
"""Single writer, private trusted directory, fixed state.json name, bytes <=64KiB.
All files are on this one filesystem. No symlink/concurrent-writer security
contract, permissions preservation, multi-file transaction or retry is offered.
"""
if type(data)is not bytes or not 1<=len(data)<=65536:raise ValueError('BOUNDED_BYTES')
directory=Path(directory)
if not directory.is_dir()or directory.is_symlink():raise ValueError('PRIVATE_REAL_DIRECTORY')
target=directory/'state.json';pending=None;fd=None;dfd=None
phase='OPEN_DIRECTORY';attempted=False;error=None;cleanup=[];receipt=None
try:
dfd=ops.open(str(directory),os.O_RDONLY|getattr(os,'O_DIRECTORY',0))
phase='CREATE_TEMP';fd,name=tempfile.mkstemp(prefix='.pending-',dir=directory);pending=Path(name)
hook('temp_created')
phase='WRITE_TEMP';offset=0
while offset<len(data):
n=ops.write(fd,data[offset:])
if type(n)is not int or not 0<n<=len(data)-offset:raise OSError('NON_PROGRESS_WRITE')
offset+=n
hook('data_written')
phase='SYNC_FILE';ops.fsync(fd);hook('file_synced')
phase='CLOSE_FILE';closing=fd;fd=None;ops.close(closing)
phase='REPLACE';attempted=True;ops.replace(str(pending),str(target));hook('replaced')
phase='SYNC_DIRECTORY';ops.fsync(dfd);hook('directory_synced')
phase='READBACK';observed=target.read_bytes()
if observed!=data:raise OSError('READBACK_MISMATCH')
hook('verified')
receipt={'state':'OS_SYNC_CALLS_RETURNED_AND_READBACK_MATCHED','bytes':len(data),
'sha256':hashlib.sha256(observed).hexdigest(),'replace_attempted':True,
'physical_power_loss_tested':False,'future_durability_certified':False}
except BaseException as exc:error=exc
finally:
if fd is not None:
try:ops.close(fd)
except BaseException as exc:cleanup.append('temp_close:'+type(exc).__name__)
if pending is not None:
try:pending.unlink(missing_ok=True)
except BaseException as exc:cleanup.append('temp_unlink:'+type(exc).__name__)
if dfd is not None:
try:ops.close(dfd)
except BaseException as exc:cleanup.append('directory_close:'+type(exc).__name__)
if error is not None or cleanup:
cause=error if error is not None else OSError('CLEANUP_FAILED')
if isinstance(cause,(KeyboardInterrupt,SystemExit)):
cause.cleanup_errors=cleanup
raise cause
raise ReplaceFailure(phase,attempted,cleanup,cause)from cause
return receipt
CHILD_CODE='''import importlib.util,os,sys
s=importlib.util.spec_from_file_location("provided110",sys.argv[1]);m=importlib.util.module_from_spec(s);s.loader.exec_module(m)
def hook(stage):
if stage==sys.argv[3]:os._exit(71)
m.publish_snapshot(sys.argv[2],m.snapshot_bytes(2,"NEW"),hook)
'''
def observe_case(directory, old, new):
p=Path(directory)/'state.json'
try:raw=p.read_bytes()
except FileNotFoundError:return {'visible':'MISSING','bytes':0}
return {'visible':'OLD'if raw==old else 'NEW'if raw==new else 'OTHER',
'bytes':len(raw),'sha256':hashlib.sha256(raw).hexdigest(),
'leftover_pending_count':len(list(Path(directory).glob('.pending-*')))}
def filesystem_observation(directory):
"""Read only a type label and statvfs units; no full mount/device metadata."""
info={'type':None,'type_confirmed':False}
try:
v=os.statvfs(directory);info.update(block_size=v.f_bsize,fragment_size=v.f_frsize)
if sys.platform=='darwin' and os.stat(directory).st_dev==os.stat('/System/Volumes/Data').st_dev:
import plistlib
p=subprocess.run(['/usr/sbin/diskutil','info','-plist','/System/Volumes/Data'],capture_output=True,timeout=3,check=True)
info['type']=plistlib.loads(p.stdout).get('FilesystemType')
elif sys.platform.startswith('linux'):
p=subprocess.run(['stat','-f','-c','%T',str(directory)],capture_output=True,text=True,timeout=3,check=True)
info['type']=p.stdout.strip()
info['type_confirmed']=isinstance(info['type'],str)and 0<len(info['type'])<64
if not info['type_confirmed']:info['type']=None
except (OSError,ValueError,subprocess.SubprocessError):pass
return info
def run_lab():
"""No user target path: every case uses a freshly created TemporaryDirectory."""
old=snapshot_bytes(1,'OLD');new=snapshot_bytes(2,'NEW');cases=[]
with tempfile.TemporaryDirectory(prefix='file-replace-lab-')as root:
root=Path(root);filesystem=filesystem_observation(root)
for stage in ('data_written','file_synced','replaced','directory_synced'):
d=root/stage;d.mkdir();publish_snapshot(d,old)
child=subprocess.run([sys.executable,'-c',CHILD_CODE,str(Path(__file__).resolve()),str(d),stage],
stdin=subprocess.DEVNULL,stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,timeout=10)
if child.returncode!=71:raise RuntimeError('CHILD_DID_NOT_REACH_STAGE '+stage)
cases.append({'kind':'process_exit_without_finally','stage':stage,'exit_code':child.returncode,
**observe_case(d,old,new)})
for stage in ('file_synced','replaced','directory_synced'):
d=root/('error-'+stage);d.mkdir();publish_snapshot(d,old)
def hook(actual,selected=stage):
if actual==selected:raise OSError('INJECTED_AFTER_STAGE')
try:publish_snapshot(d,new,hook)
except ReplaceFailure as exc:
cases.append({'kind':'injected_returned_error','stage':stage,'reported_visibility':exc.visibility,
'reported_durability':exc.durability,**observe_case(d,old,new)})
else:raise AssertionError('FAULT_NOT_RAISED')
d=root/'success';d.mkdir();publish_snapshot(d,old);r=publish_snapshot(d,new)
cases.append({'kind':'successful_calls','receipt':r,**observe_case(d,old,new)})
return {'state':'CLOSED_ISOLATED_HOST_EXPERIMENT','environment':{'python':platform.python_version(),
'platform':platform.platform(),'os_name':os.name},'cases':cases,
'physical_power_loss_tested':False,'kernel_crash_tested':False,
'filesystem_observation':filesystem,'temporary_directories_removed':True,
'scope':'Observed process/error outcomes with OS still running; not Pi SD-card or physical durability qualification.'}
def main():
if len(sys.argv)!=1 or os.name!='posix':raise SystemExit('Run python3 file_replace_lab.py on a POSIX host; no target path accepted.')
print(json.dumps(run_lab(),indent=2))
if __name__=='__main__':main()
publish_snapshot is the reusable single-file operation, while run_lab owns the disposable test locations. The function records that replacement was attempted before entering the rename call, because an exception after a side effect cannot safely be interpreted as “nothing changed.” It does not attempt a second replacement as automatic rollback. Cleanup independently attempts to close retained descriptors and remove a remaining temporary name.
KeyboardInterrupt and SystemExit propagate after cleanup attempts. Other failures become ReplaceFailure with the phase, attempted-replacement flag and cleanup diagnostics. Even a close error prevents a successful final receipt. Process termination can bypass cleanup entirely; this is why leftovers are inspected inside the enclosing private experiment rather than quietly treated as a completed transaction.
7. Reproduce without touching an existing log
First read the program and verify that the command contains no path argument. Record the interpreter and OS versions. On a Pi target, record the actual card, mount configuration, filesystem and supply separately; the program does not discover or qualify all of them. Use a spare development setup with normal shutdown available, not a production logger.
Run the program once. Retain the printed report outside the experiment only if you choose to save it through your ordinary tools. Inspect each case's stage and visible byte class. Compare a before-replacement exit with an after-replacement exit; then compare the latter with an injected error after the same stage. Do not change an error to success merely because a later read happened to show NEW.
Fill the blank comma-separated values (CSV) worksheet with your own environment and observations. A CSV file is a row-and-column text record. Leave physical_power_removed false for this procedure. Host examples in the explanation are labelled as author-host outcomes, while the supplied bench worksheet contains only its header and no claimed Pi measurements.
For application design, decide what an uncertain commit means before adding retries. A unique sequence and an idempotent consumer may help identify a repeated request, but this example offers neither a database nor a coordination protocol. If several files must change together, or several writers can race, use a separately reviewed transaction design rather than extending a one-file recipe by assumption.
8. Exercise real files and bounded fault paths
Run the supplied test_replace.py with Python3. Its26 test groups exercise the delivered function, real temporary files, short writes, file and directory synchronization errors before and after effects, rename ambiguity, cleanup and interrupts. A reader thread opens the pathname during eight replacements and checks that every observed result is an entire OLD or NEW version. An already-open descriptor test separately verifies retention of the old file object.
The four child-exit stages execute the actual program in subprocesses. Returned-error injections wrap real operations or raise at explicit stage hooks. They do not simulate flash cells, cut the storage supply or reboot a kernel. A successful host suite therefore remains useful evidence about control flow and observed filesystem semantics while leaving the physical durability question open.
Unsupported directory synchronization, permissions errors or a read-back mismatch should remain failures. Do not suppress an exception to make a particular platform look compatible. The code uses CPython3.11-documented interfaces; the supplied author results came from3.12.14 on macOS. Running the same code on a Pi is an additional experiment with its own evidence and limitations.
9. Challenge: NEW exists after failure
The program raises after os.replace but a new open reads NEW. Should it immediately restore OLD and report a failed update with the old state intact?
Show the answer
No. The operation has already crossed a visibility boundary and the caller does not know a durable recovered state. A second replacement would be another state change with its own possible failure. Record the ambiguous outcome, preserve the sequence information and let a separately designed recovery policy reconcile it. Seeing NEW now is evidence about this read, not a future power-loss guarantee.
The official Pi hardware guide covers the intended computer and power connection. Continue through the electronics course or existing category menu. There are no hidden hardware results: no Pi execution, physical power-loss test, kernel crash, card qualification or promise that an existing application can safely replace its storage layer with this experiment.
