Managing Delta Lake VACUUM & Time Travel for DPDP Erasure

Delta Lake’s time travel feature is one of the most operationally powerful capabilities in the modern lakehouse. It is also, if managed without regulatory intent, a serious compliance liability under India’s Digital Personal Data Protection Act.

Every SQL DELETE you execute on a Delta table is a promise, not a delivery. The Parquet files containing that personal data remain physically intact on your cloud object storage until VACUUM removes them. Under DPDP Section 12 and Rule 8, the Data Principal’s right to erasure is not satisfied by a logical deletion record. It is satisfied only when the underlying bytes no longer exist on the primary storage layer covered by the Delta table.

Delta’s default data file retention threshold (delta.deletedFileRetentionDuration) is 7 days, which means old Parquet files containing deleted rows remain on storage and are eligible for VACUUM removal only after that window elapses. This default is built for operational recovery, not regulatory compliance. For DPDP PII tables, it must be managed deliberately.

This article delivers the complete VACUUM configuration reference for DPDP compliance: how to manage time travel windows without breaking operational needs, how to execute forced zero-hour purges safely, how the deletion vector default change in April 2025 creates a hidden compliance gap in Lakeflow pipelines, and how to build the per-table retention architecture that maps each legal obligation to a precise Delta property setting.

For the broader erasure architecture, the five-artifact evidence package, and the erasure registry control table pattern, see the Hub article: DPDP Retention and Erasure on Databricks: How to Prove Deletion and Audit Evidence.

---

Section 1 : Understanding What VACUUM Actually Does

The Delta File Lifecycle and the Two Properties That Govern It

1.1 The Delta Lake File Lifecycle

Every DML operation on a Delta table writes new Parquet files to object storage and records “add” and “remove” entries in the transaction log. When you run DELETE FROM customer_profiles WHERE customer_id = 'dp-00142' on a table without deletion vectors enabled, Delta rewrites the entire affected data file group, producing new Parquet files that exclude the deleted rows. The old Parquet files are marked as “removed” in the transaction log but remain physically intact on object storage.

VACUUM finds those “remove” entries older than the configured retention threshold and physically deletes the corresponding files from cloud storage. It records VACUUM START and VACUUM END entries in the transaction log. The VACUUM END entry, containing the count of deleted files, is the primary technical proof of physical erasure within the Delta table’s storage layer.

1.2 The Two Properties That Govern VACUUM

Property	What It Controls	Default	DPDP Setting
delta.deletedFileRetentionDuration	Minimum age before VACUUM can physically delete removed data files	7 days	‘interval 0 hours’ on PII tables requiring on-demand forced erasure
delta.logRetentionDuration	How long Delta transaction log entries are retained	30 days	‘interval 1 year’ to satisfy DPDP Rule 8(3) one-year log retention

These properties have independent clocks. A critical constraint from Databricks Runtime 18.0: logRetentionDuration must be greater than or equal to deletedFileRetentionDuration. The 1-year and 0-hour combination is valid and is the correct DPDP configuration.

ALTER TABLE catalog.schema.customer_profiles
SET TBLPROPERTIES (
  'delta.logRetentionDuration'         = 'interval 1 year',
  'delta.deletedFileRetentionDuration' = 'interval 0 hours'
);

1.3 The VACUUM RETAIN Override Trap

The explicit RETAIN N HOURS clause in any VACUUM command overrides the table-level deletedFileRetentionDuration property. If a global maintenance job runs VACUUM table_name RETAIN 168 HOURS, the PII files that should have been purged at 0 hours are kept for another 7 days. The job completes without errors. The personal data remains on disk.

The fix: exclude PII and legal-hold tables from all global VACUUM jobs. Query Unity Catalog tags to build the exclusion list:

SELECT DISTINCT table_catalog || '.' || table_schema || '.' || table_name
FROM system.information_schema.table_tags
WHERE tag_name IN ('pii_class', 'legal_hold')
  AND tag_value IS NOT NULL;

Route these tables to a dedicated compliance VACUUM workflow instead.

---

Section 2 : Time Travel, Erasure, and the DPDP Compliance Tension

Why Time Travel Is Both the Problem and Part of the Solution

2.1 Time Travel as a Compliance Liability

Without a targeted compliance VACUUM, a time travel query returns deleted PII weeks after the erasure request was processed:

SELECT * FROM catalog.schema.customer_profiles
TIMESTAMP AS OF '2025-06-08 14:00:00'
WHERE customer_id = 'dp-00142';
-- Returns deleted PII if VACUUM has not run with 0-hour retention

Under DPDP Rule 14, Data Fiduciaries have 90 days to address erasure requests. PII remaining physically accessible via time travel during that window is a compliance failure.

2.2 Time Travel as a Compliance Asset

The Delta transaction log, preserved via logRetentionDuration, is the evidence chain that a DPB inquiry will require. DESCRIBE HISTORY returns every operation performed on a table with timestamps and operation parameters. The chronological sequence of DELETE, REORG TABLE, and VACUUM END entries constitutes the technical audit chain of custody.

DPDP Rule 8(3) requires one-year retention of processing logs from the date of processing. Setting logRetentionDuration = 'interval 1 year' directly satisfies this. The key architectural insight: set logRetentionDuration to 1 year (audit history preserved) while setting deletedFileRetentionDuration to 0 hours (immediate physical purge enabled). These clocks are independent.

2.3 Per-Table Compliance Retention Matrix

Table Classification	Legal Basis	logRetentionDuration	deletedFileRetentionDuration	VACUUM Strategy
Core Customer PII (e-commerce, gaming, social media)	DPDP Third Schedule (3-yr inactivity deadline) + Rule 8(3)	1 year	0 hours (on erasure)	Dedicated on-demand compliance VACUUM. Excluded from global schedule.
Financial and KYC Records	RBI KYC (5-yr minimum) + DPDP Rule 8(3)	1 year	180 days	Legal hold tag. Manual compliance VACUUM after hold is cleared.
Trade and Order Records	SEBI (8-yr minimum) + DPDP Rule 8(3)	1 year	365 days	Legal hold tag. VACUUM scheduled only after hold expiry.
Gold Aggregates (non-PII)	Operational only	30 days (default)	7 days (default)	Standard Predictive Optimization or global schedule.
Operational Logs (non-personal)	Operational	7 days	7 days	Standard global VACUUM schedule.

Apply legal hold tags in Unity Catalog to protect datasets from automated erasure:

ALTER TABLE catalog.schema.transaction_records
SET TAGS (
  'legal_hold'        = 'true',
  'legal_hold_basis'  = 'RBI_Master_Direction_KYC_2016',
  'legal_hold_expiry' = '2030-03-31'
);

---

Section 3 : The Safe VACUUM Execution Playbook for DPDP

Step-by-Step Production Procedure for Forced Erasure

Step 0: Audit table configuration

DESCRIBE DETAIL catalog.schema.customer_profiles;
SHOW TBLPROPERTIES catalog.schema.customer_profiles;

Confirm deletedFileRetentionDuration = 'interval 0 hours' and logRetentionDuration = 'interval 1 year' before proceeding.

Step 1: Set table properties for on-demand erasure

Use the ALTER TABLE statement shown in Section 1.2.

Step 2: Run VACUUM DRY RUN as a compliance pre-flight

DRY RUN is an audit-supporting artifact, not just an operational preview. The file manifest it returns proves which specific Parquet files were in the deletion queue before execution. Archive this output alongside the final VACUUM END entry.

VACUUM catalog.schema.customer_profiles DRY RUN;
-- Archive output to: s3://compliance-archive/dryrun/ER-2025-00142/preflight_manifest.json

Step 3: Forced zero-hour purge

VACUUM RETAIN 0 HOURS with the safety check disabled can delete uncommitted files from concurrent write transactions, causing data loss. Mandatory mitigation: pause all streaming and batch write jobs targeting the table before executing, then resume after.

-- Pause all concurrent writers before this block

SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM catalog.schema.customer_profiles FULL RETAIN 0 HOURS;
SET spark.databricks.delta.retentionDurationCheck.enabled = true;

-- Resume ingestion pipelines after this block

Step 4: Capture DESCRIBE HISTORY as an audit-supporting artifact

DESCRIBE HISTORY catalog.schema.customer_profiles LIMIT 5;
-- Locate VACUUM END: operation, numDeletedFiles, numVacuumedDirectories, timestamp
-- Archive to: s3://compliance-archive/history/ER-2025-00142/vacuum_end_snapshot.json

What VACUUM FULL covers and what it does not

After VACUUM FULL RETAIN 0 HOURS completes, no readable form of the deleted PII remains within the Delta table’s managed storage layer on the primary object store, provided the following conditions hold:

• Cloud object-store bucket versioning is disabled (enabled versioning retains deleted file versions separately from Delta’s view of storage).
• Shallow or deep Delta table clones created before erasure are mapped and addressed independently.
• The Databricks internal disk cache on running clusters may temporarily hold data from old file versions until a cluster restart occurs.
• Automated cloud backup snapshots may retain copies outside Delta’s control.
• Raw landing zone data (Kafka topic retention, S3 raw ingestion paths, upstream operational databases) is a completely separate persistence layer that must be addressed outside the Delta VACUUM workflow.

VACUUM FULL is not a guarantee of global erasure across all systems. It is the correct mechanism for physical erasure within the Delta table’s primary storage path.

VACUUM FULL vs VACUUM LITE

VACUUM LITE (DBR 16.1+, Public Preview) uses the transaction log rather than listing the full table directory. It is faster for routine non-PII maintenance. For compliance erasure, always use VACUUM FULL because VACUUM LITE fails with DELTA_CANNOT_VACUUM_LITE when the Delta log has been pruned. Since compliance tables carry long logRetentionDuration settings, that failure mode cannot be excluded. VACUUM FULL scans the actual directory regardless of log state.

-- Compliance erasure: always FULL
VACUUM catalog.schema.customer_profiles FULL RETAIN 0 HOURS;

-- Routine non-PII maintenance: LITE acceptable
VACUUM catalog.schema.analytics_gold LITE;

---

Section 4 : Deletion Vectors and VACUUM: The Complete Compliance Sequence

Why the April 2025 Default Change Creates an Invisible Gap

4.1 The Optimization Trap

Deletion vectors are a performance optimization that changes how DELETE operations work on Delta tables. When deletion vectors are enabled, a DELETE does not rewrite any data files. Instead, it writes a small sidecar bitmap file (the deletion vector) that marks specific row positions as logically absent. The original Parquet file, containing the personal data, is completely untouched on object storage.

After DELETE on a deletion-vector table, both the original Parquet files and the sidecar files remain actively referenced by the current state of the table, making them ineligible for standard VACUUM removal. Running DELETE followed by VACUUM FULL RETAIN 0 HOURS alone does not remove the original Parquet data from physical storage. REORG TABLE APPLY (PURGE) must execute first to physically rewrite the data blocks and sever these active file references, producing new compacted files that exclude the deleted rows.

As of April 28, 2025, deletion vectors are enabled by default for materialized views and streaming tables in Lakeflow Declarative Pipelines. Any team using Lakeflow for PII ingestion is now operating in deletion-vector territory by default.

4.2 The Mandatory Three-Step Compliance Sequence

Step 1: DELETE

DELETE FROM catalog.schema.customer_profiles
WHERE customer_id = 'dp-00142';

Step 2: REORG TABLE APPLY (PURGE) to physically rewrite data files and sever active references

-- Advanced tuning flag: validate against current runtime support before enabling
SET spark.databricks.delta.reorg.purgeMode = 'rows';

REORG TABLE catalog.schema.customer_profiles APPLY (PURGE);
-- REORG is idempotent: safe to run twice if the job fails partway through
-- The REORG completion timestamp is the binding anchor for VACUUM retention

Step 3: VACUUM FULL

SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM catalog.schema.customer_profiles FULL RETAIN 0 HOURS;
SET spark.databricks.delta.retentionDurationCheck.enabled = true;

4.3 The Streaming and Materialized View Compliance Gap

Lakeflow Declarative Pipelines run VACUUM automatically within 24 hours of a streaming table or materialized view update (when Predictive Optimization is not enabled). This automated VACUUM does not run REORG TABLE APPLY (PURGE) first. For compliance tables with deletion vectors enabled, the automated Lakeflow VACUUM is insufficient. A dedicated orchestration workflow must trigger REORG followed by VACUUM FULL RETAIN 0 HOURS on all affected streaming tables, independent of Lakeflow’s automatic cycle.

4.4 Auditing Deletion Vector Status Across Unity Catalog

SELECT t.table_catalog, t.table_schema, t.table_name,
       p.tag_value AS pii_classification
FROM system.information_schema.tables     t
JOIN system.information_schema.table_tags p
  ON  t.table_catalog = p.catalog_name
  AND t.table_schema  = p.schema_name
  AND t.table_name    = p.table_name
WHERE p.tag_name = 'pii_class'
ORDER BY t.table_catalog, t.table_schema, t.table_name;
-- For each result: run DESCRIBE DETAIL and check tableFeatures for 'deletionVectors'

---

Section 5: Automating VACUUM for DPDP: Orchestration Architecture

Two Workflows, One Compliance Standard

5.1 Why Global VACUUM Schedules Are a Compliance Anti-Pattern

A single global VACUUM job creates three simultaneous problems for DPDP compliance. The RETAIN N HOURS clause overrides per-table property settings, leaving PII files intact despite correct table configuration. Global jobs cannot be safely quiesced around specific tables without halting broad ingestion. And global jobs run on a fixed schedule rather than responding to specific erasure events, breaking the per-request evidence chain needed for an inquiry response.

5.2 The Two-Workflow Architecture

Workflow A: Global Maintenance VACUUM runs daily, excludes all pii_class and legal_hold tagged tables, uses VACUUM LITE for performance, and is supplemented by Predictive Optimization where available.

Workflow B: Compliance Purge VACUUM runs on-demand, triggered by the erasure_requests registry. Task dependency chain:

Read erasure_requests: erasure window elapsed
Check tags: skip if legal_hold = 'true'
Pause streaming/batch write jobs on affected tables
VACUUM DRY RUN: archive manifest
If deletion vectors present: REORG TABLE APPLY (PURGE)
Disable safety check
VACUUM FULL RETAIN 0 HOURS
Re-enable safety check
Resume pipelines
DESCRIBE HISTORY: archive VACUUM END entry
Update erasure_requests: status = 'completed', evidence_path set

Each task has explicit dependencies. A failure at any step halts the workflow before proceeding.

5.3 Predictive Optimization: What It Provides and What It Does Not

Databricks Predictive Optimization automates VACUUM and other maintenance tasks for Unity Catalog managed tables as an asynchronous, cost-driven background process. Because it operates on its own schedule and optimization criteria, it cannot be relied upon as a deterministic substitute for on-demand, zero-hour compliance VACUUM execution. It also does not run REORG TABLE APPLY (PURGE) before VACUUM for deletion-vector tables, is not available in all Databricks cloud regions, and does not apply to external tables. Treat it as the maintenance layer for non-PII tables only.

---

Section 6 : Streaming Tables, CDF, and Delete Propagation

Completing the Medallion Erasure Chain

Delta Lake streaming tables are append-only by design. A DELETE on a Bronze source does not automatically propagate to a Silver streaming table. Use Change Data Feed with apply_changes in Lakeflow:

import dlt

dlt.apply_changes(
    target           = "silver_customer_profiles",
    source           = "STREAM(bronze_customer_raw)",
    keys             = ["customer_id"],
    sequence_by      = col("_commit_timestamp"),
    apply_as_deletes = expr("_change_type = 'delete'")
)

For downstream consumers that should not process delete events directly, add skipChangeCommits to prevent pipeline failures when physical blocks are removed upstream:

df = (spark.readStream
        .option("skipChangeCommits", "true")
        .table("silver_customer_profiles"))

Executing physical VACUUM purges in Gold-first, Silver-then-Bronze order is the recommended default pattern for this evidence-preserving workflow, unless lineage and downstream recomputation are otherwise proven for your specific architecture. If Silver is vacuumed before Gold has processed the deletion, Gold tables may contain PII rows whose lineage Silver can no longer confirm. Complete all layer deletions and verify propagation before running any VACUUM. Use Unity Catalog column-level lineage to identify every downstream table containing derived PII fields before initiating the multi-layer purge sequence. For the full audit log query framework and lineage trace procedure, see How to Use Unity Catalog Audit Logs for DPDP Deletion and Audit Evidence.

---

Section 7 : Common VACUUM Compliance Mistakes

Mistake 1: Assuming DELETE alone satisfies DPDP erasure. Without deletion vectors, DELETE rewrites affected file groups and marks old files as removed, but those old Parquet files remain on storage until VACUUM runs. Always follow DELETE with REORG (on deletion-vector tables) and VACUUM FULL.

Mistake 2: Missing REORG TABLE APPLY (PURGE) on deletion-vector tables. After DELETE on a deletion-vector table, both the original Parquet files and the sidecar remain actively referenced. REORG must run first to sever those references before VACUUM can remove the old files. Check for ‘deletionVectors’ in DESCRIBE DETAIL on every PII table.

Mistake 3: Allowing global VACUUM RETAIN to override per-table properties. Exclude all pii_class and legal_hold tables from global VACUUM jobs. Route them to Workflow B.

Mistake 4: Running VACUUM RETAIN 0 HOURS with concurrent writers active. Quiesce all concurrent write pipelines before disabling the safety check.

Mistake 5: Treating Predictive Optimization as a compliance VACUUM mechanism. It operates asynchronously and cannot deliver on-demand zero-hour purges. It also omits REORG for deletion-vector tables. Use it for non-PII maintenance only.

Mistake 6: Using VACUUM LITE for compliance erasure. VACUUM LITE fails with DELTA_CANNOT_VACUUM_LITE on pruned logs. Always use VACUUM FULL for forced erasure.

Mistake 7: Skipping VACUUM DRY RUN. The pre-flight file manifest is an audit-supporting artifact. Archive it before every live purge.

Mistake 8: Not archiving DESCRIBE HISTORY after VACUUM. The VACUUM END entry is a primary technical record of physical erasure. Capture and archive it immediately before the log ages out.

---

Section 8 : Implementation Checklist: VACUUM for DPDP Compliance

1. Audit deletion vector status on every PII table using DESCRIBE DETAIL. Identify which require the three-step sequence.
2. Set logRetentionDuration = 'interval 1 year' on all compliance tables to satisfy Rule 8(3).
3. Set deletedFileRetentionDuration = 'interval 0 hours' on PII tables requiring on-demand forced erasure.
4. Verify the DBR 18.0 constraint: logRetentionDuration must be greater than or equal to deletedFileRetentionDuration. The 1-year and 0-hour combination is valid.
5. Exclude PII-tagged and legal-hold-tagged tables from the global VACUUM maintenance schedule.
6. Apply legal hold tags in Unity Catalog for all RBI, SEBI, or sectoral retention-governed tables.
7. Build Workflow B as a Databricks Workflow triggered by the erasure_requests registry, with all 11 task dependencies in Section 5.2.
8. Set spark.databricks.delta.reorg.purgeMode = 'rows' on large deletion-vector tables before REORG as an advanced tuning option; validate against current runtime support before enabling.
9. Run VACUUM DRY RUN first and archive the manifest for every erasure request.
10. Quiesce all concurrent write pipelines before running VACUUM RETAIN 0 HOURS with the safety check disabled.
11. Always use VACUUM FULL for compliance erasure operations, never LITE.
12. Archive DESCRIBE HISTORY immediately after every compliance VACUUM run.
13. Re-enable spark.databricks.delta.retentionDurationCheck.enabled after every compliance VACUUM session.
14. For Lakeflow streaming tables and materialized views: build a standalone REORG + VACUUM compliance workflow outside Lakeflow’s automatic cycle.
15. Use Unity Catalog column-level lineage to map all downstream Gold tables containing derived PII fields before starting the cross-layer VACUUM sequence.

---

Section 9 : Conclusion

Every Retention Setting Is Now a Legal Metric

Delta Lake retention properties, time travel windows, and VACUUM execution modes are now legal compliance instruments under India’s DPDP Rules 2025, notified November 13, 2025 with full applicability from May 13, 2027. The configuration decisions in this article translate directly into whether your organization can produce defensible technical structures in response to a DPB inquiry.

Sinki.ai’s Data Erasure solution is designed to automate this entire lifecycle on Databricks. It handles the two-phase 48-hour notification workflow, deletion vector detection and REORG orchestration, cross-layer medallion cascade purging, session safety override management, and automated DPDP evidence package generation as a managed service on your existing environment. Explore the full architecture at sinki.ai/solutions/data-erasure.

Disclaimer: This article provides technical architecture and implementation guidance only and does not constitute formal legal advice. Organizations should consult qualified legal counsel to assess their specific compliance obligations under the DPDP Act 2023 and applicable sectoral regulations.

---

Frequently Asked Questions