This is a point-in-time record of the week of 2026-06-29 to 2026-07-05. It was written from the repository's history for that week and is not revised afterwards, so later pages may supersede what it says.
Week of 2026-06-29 to 2026-07-05
Window: 2026-06-29 00:00:00 to 2026-07-05 23:59:59, local author time Recapped: 2026-08-24 Commits in window: 39 (30 non-merge, 9 merges)
This is a backfill written 50 days after the window closed. It is the week July began by changing the physical storage contract: silver small files gained a compactor, bronze compression was measured rather than guessed, and the Parquet lake moved to normalized month-level paths. A separate production fix stopped successfully scraped but dealer-unenriched listings from being fetched again every fifteen minutes.
What shipped
Plan 109 — compact the silver small-file stream
Plan 109 exists because a five-minute
flush cadence produced roughly 150 Parquet files per source per day, multiplying
query overhead and weakening compression. 2a0037c added daily, sorted,
watermark-protected compaction with a reader-safe temporary-file sequence and a
bounded backlog drain. Review follow-ups be01ca7, 2ce776a and b8bd1c1
made delete failure abort publication, isolated integration prefixes, and
bypassed PyArrow's conflicting Hive inference. Those follow-ups name no plan
and remain in the unattributed ledger.
Plan 115 — a successful scrape must close its retry circuit
Plan 115 fixed a
production loop where customer_id IS NULL meant both “never fetched” and
“successfully fetched without dealer metadata.” dd14e04 added
last_detail_scraped_at, made only detail writes advance it, and suppressed
another detail attempt for seven days. The same commit removed roughly 145
seconds of unit-test TCP timeouts by mocking external I/O. 9c9f6eb updated
direct integration-test callers for the new write contract.
Plan 116 — measure recompression before building a migration
Plan 116 was a read-only
gate for Plan 110's proposed bronze rewrite. 6212e85 added the estimator;
6f4cda5 replaced a prefix-wide buffered listing with a streaming paginator;
and cffc1bd recorded the production result: level 9 saved a consistent 8%
across May and June, too little to justify rewriting millions of old objects.
The decision was to improve new writes and pursue higher-value storage work.
Plan 110 — normalize storage before experiments depend on it
Plan 110 broadened from HTML
compression into storage-layout hygiene and Iceberg readiness. Its seven named
commits were 9723180, dd52f31, 313f58c, 0c2f6f8, 2767ac9, 39b118d
and b63a24a.
The implementation raised new bronze writes to zstd level 9, supplied an
operator-only recompression tool, audited Parquet using listings and footers,
selected a normalized month-level layout, and built a non-destructive rewrite
with baseline checks and write-before-rename publication. The unnumbered
18c2121 then cut writers and dbt readers over to silver_normalized/ and
ops_normalized/; 1e35c99 moved CI fixtures, while 3e2d308 removed physical
partition columns so historical rewrites and fresh flushes shared one schema.
Plan 117 connected
that storage work to the next feature, experiment and production-policy stages.
58a87af made Plans 110–113 a single evidence-driven arc: normalized storage,
interpretable features, reproducible backtesting, then conservative deployment.
What moved between states
Three index revisions introduced six build-order entries:
| Plan | Before | After |
|---|---|---|
| 110 | absent | build order |
| 111 | absent | build order |
| 112 | absent | build order |
| 113 | absent | build order |
| 114 | absent | build order |
| 115 | absent | build order |
Plan 109 shipped in the window but produced no net state change in the index. Plan 116 also opened, measured its question and closed without appearing in a net transition above.
What is still owed
- Plan 110's normalized cutover landed, but the later window still had to prove the feature layer and experimental substrate on top of it.
- Plan 109's implementation was present, but production backlog drainage and longer-running verification were not recorded in this week.
- Plan 115 fixed one requeue cause; its seven-day horizon meant the circuit breaker still depended on downstream ownership of “fetched” state, a boundary that later Plan 147 would revisit more generally.
- Plans 111–114 were newly filed work. None of their principal deliverables shipped in this window.
Unattributed commits
Eighteen non-merge commits name no numbered plan in their own subject or body. They are included in the narrative above where their work is clear.
| Commit | Subject |
|---|---|
be01ca7 |
fix: address three code review findings in compact_silver |
2ce776a |
fix: ruff lint errors in compact_silver files |
b8bd1c1 |
fix: use pq.ParquetFile().read() to bypass hive partition inference |
9c9f6eb |
fix: update integration processing tests for last_detail_scraped_at upsert param |
f687daa |
fix: ruff lint errors in analysis scripts (E501, E741, I001) |
6f4cda5 |
fix: use boto3 paginator in iter_prefix to avoid blocking s3fs.ls |
1be9ba0 |
docs: fix Phase 6 deploy intent and compose service names |
d41d6bd |
docs: fix Phase 6 Steps 4+5 to use host curl for archiver endpoints |
54c9760 |
docs: fix Phase 6 Step 10/11 ordering — validate before unpausing DAGs |
713b5e3 |
docs: fix rollback completeness after Step 10 controlled flush |
924cd3c |
docs: remove fake CLI flags from Phase 6 Option B recovery |
6171a8d |
docs: add ops_normalized validation to Phase 6 Step 10 and rollback |
4088829 |
docs: check all five ops/ prefixes in Step 10 and fix flushed=0 wording |
d19ccd1 |
finalizing audit script for furture use. |
18c2121 |
Implement normalized parquet cutover |
1e35c99 |
ci: update MinIO seed paths to normalized layout |
c724d08 |
Pin dbt core below 2 in runner |
3e2d308 |
fix: drop Hive partition cols from rewritten Parquet physical schema |
Merges
Nine merges are named and not recapped separately.
| Merge | PR | Branch |
|---|---|---|
0d33512 |
#130 | feature/plan-109-silver-compaction |
a8d2622 |
#131 | bugfix/detail-unenriched-circuit-breaker |
7ee58c7 |
#132 | feature/plan-116-estimate-recompression-savings |
6310858 |
#133 | feature/storage-refresh-implementation-plans |
1217f9f |
#134 | feature/plan-110-storage-layout-hygiene |
3cd7fde |
#135 | feature/plan-110-storage-layout-hygiene |
a6ece65 |
#136 | feature/plan-110-storage-layout-hygiene |
5f7237e |
#137 | feature/plan-110-storage-layout-hygiene |
11f736d |
#138 | feature/plan-110-storage-layout-hygiene |
Deferred to the next recap
419 commits sit after this window's end as of 2026-08-24.
git rev-list --count --all --since='2026-07-05 23:59:59' # 419
That raw tail includes the July 6–12 recap being written alongside this one and the already-covered weeks ending July 19, July 26, August 2, August 9, August 16 and August 23. Once both files in this backfill are present, none of the following complete weeks is genuinely uncovered.