mirror of
https://github.com/anthropics/skills.git
synced 2026-08-02 13:05:28 +08:00
Update docx, pptx, and xlsx skills (#1447)
Add support for template formats (.dotx, .potx, .xltx) across validation and the helper scripts. Consolidate the shared office helpers into a single module, replacing the pack/unpack pipeline with explicit zip/unzip steps. Extraction now rejects symlink and path-traversal archive entries. Move run merging to a standalone docx script, since it only ever applied to Word documents. Fix the redlining validator so it compares against the original even when a document has no tracked changes, which is when an untracked edit would otherwise go unreported. Provision a LibreOffice user profile per invocation so conversions work in sandboxed environments. Trim the skill docs to the guidance that earns its place.
This commit is contained in:
Peter Lai
+82
-275
@@ -1,292 +1,99 @@
|
||||
---
|
||||
name: xlsx
|
||||
description: "Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like \"the xlsx in my downloads\") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved."
|
||||
description: "Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like \"the xlsx in my downloads\") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved."
|
||||
license: Proprietary. LICENSE.txt has complete terms
|
||||
---
|
||||
|
||||
# Requirements for Outputs
|
||||
# XLSX creation, editing, and analysis
|
||||
|
||||
## All Excel files
|
||||
| Task | Approach |
|
||||
|---|---|
|
||||
| **Create** or **edit** with formulas/formatting | `openpyxl` — see gotchas below |
|
||||
| **Bulk data** in or out | `pandas` (`read_excel`, `to_excel`) |
|
||||
| **Quick look** at a sheet | `markitdown file.xlsx` — `## SheetName` per sheet; reads `.xlsm` too. No cell coordinates, so don't plan edits from it |
|
||||
| **Read** a model (formulas *and* values) | two `load_workbook` passes — see gotchas |
|
||||
|
||||
### Professional Font
|
||||
- Use a consistent, professional font (e.g., Arial, Times New Roman) for all deliverables unless otherwise instructed by the user
|
||||
> `openpyxl`, `pandas`, and `markitdown` are preinstalled — do not run `pip install` first; write the script and import directly. Only if an import fails (or the `markitdown` command is missing): `pip install` the missing package.
|
||||
|
||||
### Zero Formula Errors
|
||||
- Every Excel model MUST be delivered with ZERO formula errors (#REF!, #DIV/0!, #VALUE!, #N/A, #NAME?)
|
||||
> Script paths below are relative to this skill's directory.
|
||||
|
||||
### Preserve Existing Templates (when updating templates)
|
||||
- Study and EXACTLY match existing format, style, and conventions when modifying files
|
||||
- Never impose standardized formatting on files with established patterns
|
||||
- Existing template conventions ALWAYS override these guidelines
|
||||
## Requirements for every output
|
||||
|
||||
- **Professional font** (Arial, Times New Roman) throughout, unless the user says otherwise.
|
||||
- **Zero formula errors.** Never ship while `recalc.py` reports `errors_found`. If you think an error predates you, prove it: load the *original* with `data_only=True` and look at that cell. An error you introduced looks exactly like one you inherited.
|
||||
- **Use formulas, never hardcoded results.** Write `sheet['B10'] = '=SUM(B2:B9)'`, not the Python-computed total. The sheet must recalculate when its inputs change.
|
||||
- **Follow the user's spec literally.** Exact tab names, exact column headers, and the formula they spelled out. A redesign that computes something else fails, however elegant.
|
||||
- **Document every assumption and hardcoded number** where the reader will see it — a cell comment, or an adjacent cell at a table's end. Cite a real source when one exists (`Source: Company 10-K, FY2024, Page 45, Revenue Note, [SEC EDGAR URL]`); when the number came from the user, say so plainly.
|
||||
- **A workbook *you create* for someone to fill in** needs a short legend naming which cells to edit, and one example row of realistic values showing the expected format. Never add such a row to a file you were asked to edit.
|
||||
- **Editing an existing file: match its conventions exactly.** They override every guideline here. Find its designated input cells first — a distinct font color, fill, or shading marks them — write only there, and leave every existing formula untouched.
|
||||
|
||||
## Recalculate (mandatory whenever the file contains formulas)
|
||||
|
||||
openpyxl writes formulas as strings with **no cached values**. Until you recalculate, every
|
||||
formula cell reads back as `None` to anything reading cached values — `pandas`,
|
||||
`load_workbook(data_only=True)`, and most previewers.
|
||||
|
||||
```bash
|
||||
python scripts/recalc.py output.xlsx [timeout_seconds] # default 30
|
||||
```
|
||||
|
||||
LibreOffice computes every formula, the file is **rewritten in place**, and you get JSON:
|
||||
`status` (`success` | `errors_found`), `total_formulas`, `total_errors`, and an
|
||||
`error_summary` naming up to 100 cells per error type (`locations_truncated` says how many it
|
||||
withheld — trust `total_errors`, not the length of the list). Fix what it names and run it
|
||||
again. **JSON with an `error` key instead of a `status` means nothing was recalculated**, and
|
||||
only that case exits non-zero — `errors_found` exits 0, so never treat a clean exit as a clean
|
||||
workbook.
|
||||
|
||||
**A green recalc proves your formulas *evaluate*, not that they are *right*.** An off-by-one
|
||||
range or a reference to the wrong row yields a clean, error-free file with wrong numbers.
|
||||
Write 2–3 formulas first and check they pull the values you expect, before building out a grid.
|
||||
|
||||
**A workbook that links to another file loses those links** if you re-save it with openpyxl and
|
||||
then recalculate. Such a formula reads `='[1]Returns Analysis'!$B$2` — the `[1]` is an index
|
||||
into the workbook's external-reference list, naming a *separate file on disk*, not a sheet.
|
||||
That file is rarely present here, so the cell's cached value is the only thing holding its
|
||||
data. openpyxl strips that value on save; LibreOffice then has to resolve the reference for
|
||||
real, fails, writes `#NAME?`, and deletes every link. `recalc.py` refuses to run in that state
|
||||
— copy those cells' values out of the original before you save over them (`--force` overrides,
|
||||
and accepts the loss).
|
||||
|
||||
## Choosing formulas that survive verification
|
||||
|
||||
LibreOffice implements fewer functions than Excel, and one it cannot evaluate becomes a
|
||||
literal `#NAME?` baked into the file you deliver.
|
||||
|
||||
- **Prefer Excel-2007-era functions** — `SUMIFS`, `INDEX`, `MATCH`, `IFERROR`, `SUMPRODUCT` — which need no prefix.
|
||||
- **Six post-2007 functions work, but only with an `_xlfn.` prefix**, because openpyxl writes your formula into the XML verbatim and Excel stores post-2007 names prefixed (its UI hides the prefix): `_xlfn.TEXTJOIN`, `_xlfn.CONCAT`, `_xlfn.IFS`, `_xlfn.SWITCH`, `_xlfn.MAXIFS`, `_xlfn.MINIFS`. Written bare, each yields `#NAME?`.
|
||||
- **Never use `XLOOKUP`, `XMATCH`, `SORT`, `FILTER`, `UNIQUE`, or `SEQUENCE`.** The runtime's LibreOffice cannot evaluate them under *any* prefix. Newer builds do evaluate them, but they are spilling array functions and an openpyxl-written file has no spill metadata, so only the top-left cell of the range gets a value — and `recalc.py` reports `total_errors: 0` on the truncated result. Use `INDEX`/`MATCH` for lookups, and sort, filter, and de-duplicate in Python before writing the cells.
|
||||
- A formula LibreOffice could not parse is written back **lowercased** — a quick tell beside a `#NAME?`.
|
||||
|
||||
## openpyxl gotchas
|
||||
|
||||
- **Reading a model takes two loads.** `data_only=True` yields cached values with the formulas gone; the default yields formula strings with no values. One pass cannot give you both.
|
||||
- **`data_only=True` is destructive if you save.** That workbook has no formulas left, so saving replaces every one with a literal — permanently.
|
||||
- **`data_only=True` on a file openpyxl just wrote returns `None` everywhere** — run `recalc.py` first. (A formula whose result is `""` also reads back as `None`.)
|
||||
- **Merged cells: write the top-left anchor only.** Every other cell in the range is a `MergedCell` whose `.value` is read-only.
|
||||
- **`.xlsm` loses its macros unless you pass `keep_vba=True`** to `load_workbook`.
|
||||
- **A sheet name containing a space must be quoted** in a cross-sheet reference: `='Assumptions Inputs'!$B$5`. Unquoted, it evaluates to `#VALUE!`.
|
||||
|
||||
## Financial models
|
||||
|
||||
### Color Coding Standards
|
||||
Unless otherwise stated by the user or existing template
|
||||
Unless the user says otherwise, or the existing file already does something else.
|
||||
|
||||
#### Industry-Standard Color Conventions
|
||||
- **Blue text (RGB: 0,0,255)**: Hardcoded inputs, and numbers users will change for scenarios
|
||||
- **Black text (RGB: 0,0,0)**: ALL formulas and calculations
|
||||
- **Green text (RGB: 0,128,0)**: Links pulling from other worksheets within same workbook
|
||||
- **Red text (RGB: 255,0,0)**: External links to other files
|
||||
- **Yellow background (RGB: 255,255,0)**: Key assumptions needing attention or cells that need to be updated
|
||||
**Color:** blue text (`0,0,255`) for hardcoded inputs and scenario levers · black for formulas ·
|
||||
green (`0,128,0`) for links to another sheet · red (`255,0,0`) for links to another file ·
|
||||
yellow fill (`255,255,0`) for key assumptions and cells the user should fill in.
|
||||
|
||||
### Number Formatting Standards
|
||||
**Numbers:** currency `$#,##0`, with the unit named in the header (`Revenue ($mm)`) · zeros
|
||||
render as `-`, including in percentages (`$#,##0;($#,##0);-`) · negatives in parentheses ·
|
||||
percentages `0.0%`, **stored as fractions** (`0.15` renders `15.0%`; storing `15` renders
|
||||
`1500.0%`) · valuation multiples `0.0x` · years as text (`"2024"`, never `2,024`).
|
||||
|
||||
#### Required Format Rules
|
||||
- **Years**: Format as text strings (e.g., "2024" not "2,024")
|
||||
- **Currency**: Use $#,##0 format; ALWAYS specify units in headers ("Revenue ($mm)")
|
||||
- **Zeros**: Use number formatting to make all zeros "-", including percentages (e.g., "$#,##0;($#,##0);-")
|
||||
- **Percentages**: Default to 0.0% format (one decimal)
|
||||
- **Multiples**: Format as 0.0x for valuation multiples (EV/EBITDA, P/E)
|
||||
- **Negative numbers**: Use parentheses (123) not minus -123
|
||||
**Structure:** every assumption in its own labeled cell, referenced by the formulas that use it
|
||||
(`=B5*(1+$B$6)`, never `=B5*1.05`) · formulas consistent across every projection period, since a
|
||||
lone edited cell mid-row is the commonest silent error · guard denominators that can be zero.
|
||||
|
||||
### Formula Construction Rules
|
||||
## Dependencies
|
||||
|
||||
#### Assumptions Placement
|
||||
- Place ALL assumptions (growth rates, margins, multiples, etc.) in separate assumption cells
|
||||
- Use cell references instead of hardcoded values in formulas
|
||||
- Example: Use =B5*(1+$B$6) instead of =B5*1.05
|
||||
|
||||
#### Formula Error Prevention
|
||||
- Verify all cell references are correct
|
||||
- Check for off-by-one errors in ranges
|
||||
- Ensure consistent formulas across all projection periods
|
||||
- Test with edge cases (zero values, negative numbers)
|
||||
- Verify no unintended circular references
|
||||
|
||||
#### Documentation Requirements for Hardcodes
|
||||
- Comment or in cells beside (if end of table). Format: "Source: [System/Document], [Date], [Specific Reference], [URL if applicable]"
|
||||
- Examples:
|
||||
- "Source: Company 10-K, FY2024, Page 45, Revenue Note, [SEC EDGAR URL]"
|
||||
- "Source: Company 10-Q, Q2 2025, Exhibit 99.1, [SEC EDGAR URL]"
|
||||
- "Source: Bloomberg Terminal, 8/15/2025, AAPL US Equity"
|
||||
- "Source: FactSet, 8/20/2025, Consensus Estimates Screen"
|
||||
|
||||
# XLSX creation, editing, and analysis
|
||||
|
||||
## Overview
|
||||
|
||||
A user may ask you to create, edit, or analyze the contents of an .xlsx file. You have different tools and workflows available for different tasks.
|
||||
|
||||
## Important Requirements
|
||||
|
||||
**LibreOffice Required for Formula Recalculation**: You can assume LibreOffice is installed for recalculating formula values using the `scripts/recalc.py` script. The script automatically configures LibreOffice on first run, including in sandboxed environments where Unix sockets are restricted (handled by `scripts/office/soffice.py`)
|
||||
|
||||
## Reading and analyzing data
|
||||
|
||||
### Data analysis with pandas
|
||||
For data analysis, visualization, and basic operations, use **pandas** which provides powerful data manipulation capabilities:
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
# Read Excel
|
||||
df = pd.read_excel('file.xlsx') # Default: first sheet
|
||||
all_sheets = pd.read_excel('file.xlsx', sheet_name=None) # All sheets as dict
|
||||
|
||||
# Analyze
|
||||
df.head() # Preview data
|
||||
df.info() # Column info
|
||||
df.describe() # Statistics
|
||||
|
||||
# Write Excel
|
||||
df.to_excel('output.xlsx', index=False)
|
||||
```
|
||||
|
||||
## Excel File Workflows
|
||||
|
||||
## CRITICAL: Use Formulas, Not Hardcoded Values
|
||||
|
||||
**Always use Excel formulas instead of calculating values in Python and hardcoding them.** This ensures the spreadsheet remains dynamic and updateable.
|
||||
|
||||
### ❌ WRONG - Hardcoding Calculated Values
|
||||
```python
|
||||
# Bad: Calculating in Python and hardcoding result
|
||||
total = df['Sales'].sum()
|
||||
sheet['B10'] = total # Hardcodes 5000
|
||||
|
||||
# Bad: Computing growth rate in Python
|
||||
growth = (df.iloc[-1]['Revenue'] - df.iloc[0]['Revenue']) / df.iloc[0]['Revenue']
|
||||
sheet['C5'] = growth # Hardcodes 0.15
|
||||
|
||||
# Bad: Python calculation for average
|
||||
avg = sum(values) / len(values)
|
||||
sheet['D20'] = avg # Hardcodes 42.5
|
||||
```
|
||||
|
||||
### ✅ CORRECT - Using Excel Formulas
|
||||
```python
|
||||
# Good: Let Excel calculate the sum
|
||||
sheet['B10'] = '=SUM(B2:B9)'
|
||||
|
||||
# Good: Growth rate as Excel formula
|
||||
sheet['C5'] = '=(C4-C2)/C2'
|
||||
|
||||
# Good: Average using Excel function
|
||||
sheet['D20'] = '=AVERAGE(D2:D19)'
|
||||
```
|
||||
|
||||
This applies to ALL calculations - totals, percentages, ratios, differences, etc. The spreadsheet should be able to recalculate when source data changes.
|
||||
|
||||
## Common Workflow
|
||||
1. **Choose tool**: pandas for data, openpyxl for formulas/formatting
|
||||
2. **Create/Load**: Create new workbook or load existing file
|
||||
3. **Modify**: Add/edit data, formulas, and formatting
|
||||
4. **Save**: Write to file
|
||||
5. **Recalculate formulas (MANDATORY IF USING FORMULAS)**: Use the scripts/recalc.py script
|
||||
```bash
|
||||
python scripts/recalc.py output.xlsx
|
||||
```
|
||||
6. **Verify and fix any errors**:
|
||||
- The script returns JSON with error details
|
||||
- If `status` is `errors_found`, check `error_summary` for specific error types and locations
|
||||
- Fix the identified errors and recalculate again
|
||||
- Common errors to fix:
|
||||
- `#REF!`: Invalid cell references
|
||||
- `#DIV/0!`: Division by zero
|
||||
- `#VALUE!`: Wrong data type in formula
|
||||
- `#NAME?`: Unrecognized formula name
|
||||
|
||||
### Creating new Excel files
|
||||
|
||||
```python
|
||||
# Using openpyxl for formulas and formatting
|
||||
from openpyxl import Workbook
|
||||
from openpyxl.styles import Font, PatternFill, Alignment
|
||||
|
||||
wb = Workbook()
|
||||
sheet = wb.active
|
||||
|
||||
# Add data
|
||||
sheet['A1'] = 'Hello'
|
||||
sheet['B1'] = 'World'
|
||||
sheet.append(['Row', 'of', 'data'])
|
||||
|
||||
# Add formula
|
||||
sheet['B2'] = '=SUM(A1:A10)'
|
||||
|
||||
# Formatting
|
||||
sheet['A1'].font = Font(bold=True, color='FF0000')
|
||||
sheet['A1'].fill = PatternFill('solid', start_color='FFFF00')
|
||||
sheet['A1'].alignment = Alignment(horizontal='center')
|
||||
|
||||
# Column width
|
||||
sheet.column_dimensions['A'].width = 20
|
||||
|
||||
wb.save('output.xlsx')
|
||||
```
|
||||
|
||||
### Editing existing Excel files
|
||||
|
||||
```python
|
||||
# Using openpyxl to preserve formulas and formatting
|
||||
from openpyxl import load_workbook
|
||||
|
||||
# Load existing file
|
||||
wb = load_workbook('existing.xlsx')
|
||||
sheet = wb.active # or wb['SheetName'] for specific sheet
|
||||
|
||||
# Working with multiple sheets
|
||||
for sheet_name in wb.sheetnames:
|
||||
sheet = wb[sheet_name]
|
||||
print(f"Sheet: {sheet_name}")
|
||||
|
||||
# Modify cells
|
||||
sheet['A1'] = 'New Value'
|
||||
sheet.insert_rows(2) # Insert row at position 2
|
||||
sheet.delete_cols(3) # Delete column 3
|
||||
|
||||
# Add new sheet
|
||||
new_sheet = wb.create_sheet('NewSheet')
|
||||
new_sheet['A1'] = 'Data'
|
||||
|
||||
wb.save('modified.xlsx')
|
||||
```
|
||||
|
||||
## Recalculating formulas
|
||||
|
||||
Excel files created or modified by openpyxl contain formulas as strings but not calculated values. Use the provided `scripts/recalc.py` script to recalculate formulas:
|
||||
|
||||
```bash
|
||||
python scripts/recalc.py <excel_file> [timeout_seconds]
|
||||
```
|
||||
|
||||
Example:
|
||||
```bash
|
||||
python scripts/recalc.py output.xlsx 30
|
||||
```
|
||||
|
||||
The script:
|
||||
- Automatically sets up LibreOffice macro on first run
|
||||
- Recalculates all formulas in all sheets
|
||||
- Scans ALL cells for Excel errors (#REF!, #DIV/0!, etc.)
|
||||
- Returns JSON with detailed error locations and counts
|
||||
- Works on both Linux and macOS
|
||||
|
||||
## Formula Verification Checklist
|
||||
|
||||
Quick checks to ensure formulas work correctly:
|
||||
|
||||
### Essential Verification
|
||||
- [ ] **Test 2-3 sample references**: Verify they pull correct values before building full model
|
||||
- [ ] **Column mapping**: Confirm Excel columns match (e.g., column 64 = BL, not BK)
|
||||
- [ ] **Row offset**: Remember Excel rows are 1-indexed (DataFrame row 5 = Excel row 6)
|
||||
|
||||
### Common Pitfalls
|
||||
- [ ] **NaN handling**: Check for null values with `pd.notna()`
|
||||
- [ ] **Far-right columns**: FY data often in columns 50+
|
||||
- [ ] **Multiple matches**: Search all occurrences, not just first
|
||||
- [ ] **Division by zero**: Check denominators before using `/` in formulas (#DIV/0!)
|
||||
- [ ] **Wrong references**: Verify all cell references point to intended cells (#REF!)
|
||||
- [ ] **Cross-sheet references**: Use correct format (Sheet1!A1) for linking sheets
|
||||
|
||||
### Formula Testing Strategy
|
||||
- [ ] **Start small**: Test formulas on 2-3 cells before applying broadly
|
||||
- [ ] **Verify dependencies**: Check all cells referenced in formulas exist
|
||||
- [ ] **Test edge cases**: Include zero, negative, and very large values
|
||||
|
||||
### Interpreting scripts/recalc.py Output
|
||||
The script returns JSON with error details:
|
||||
```json
|
||||
{
|
||||
"status": "success", // or "errors_found"
|
||||
"total_errors": 0, // Total error count
|
||||
"total_formulas": 42, // Number of formulas in file
|
||||
"error_summary": { // Only present if errors found
|
||||
"#REF!": {
|
||||
"count": 2,
|
||||
"locations": ["Sheet1!B5", "Sheet1!C10"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Library Selection
|
||||
- **pandas**: Best for data analysis, bulk operations, and simple data export
|
||||
- **openpyxl**: Best for complex formatting, formulas, and Excel-specific features
|
||||
|
||||
### Working with openpyxl
|
||||
- Cell indices are 1-based (row=1, column=1 refers to cell A1)
|
||||
- Use `data_only=True` to read calculated values: `load_workbook('file.xlsx', data_only=True)`
|
||||
- **Warning**: If opened with `data_only=True` and saved, formulas are replaced with values and permanently lost
|
||||
- For large files: Use `read_only=True` for reading or `write_only=True` for writing
|
||||
- Formulas are preserved but not evaluated - use scripts/recalc.py to update values
|
||||
|
||||
### Working with pandas
|
||||
- Specify data types to avoid inference issues: `pd.read_excel('file.xlsx', dtype={'id': str})`
|
||||
- For large files, read specific columns: `pd.read_excel('file.xlsx', usecols=['A', 'C', 'E'])`
|
||||
- Handle dates properly: `pd.read_excel('file.xlsx', parse_dates=['date_column'])`
|
||||
|
||||
## Code Style Guidelines
|
||||
**IMPORTANT**: When generating Python code for Excel operations:
|
||||
- Write minimal, concise Python code without unnecessary comments
|
||||
- Avoid verbose variable names and redundant operations
|
||||
- Avoid unnecessary print statements
|
||||
|
||||
**For Excel files themselves**:
|
||||
- Add comments to cells with complex formulas or important assumptions
|
||||
- Document data sources for hardcoded values
|
||||
- Include notes for key calculations and model sections
|
||||
`openpyxl`, `pandas`, `markitdown` (pip, preinstalled — install only if an import fails or the command is missing) · LibreOffice (`soffice`, auto-configured for sandboxed environments via `scripts/office/soffice.py`)
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
import os
|
||||
import posixpath
|
||||
import re
|
||||
import stat
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
OOXML_FAMILY = {
|
||||
".docx": "docx",
|
||||
".dotx": "docx",
|
||||
".pptx": "pptx",
|
||||
".potx": "pptx",
|
||||
".xlsx": "xlsx",
|
||||
".xltx": "xlsx",
|
||||
}
|
||||
|
||||
_SCHEME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9+.\-]*:")
|
||||
|
||||
SLIDE_REL_TYPE = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide"
|
||||
|
||||
|
||||
def opc_target(target: str, source_part: str, target_mode: str = "") -> str | None:
|
||||
if not target:
|
||||
return None
|
||||
if target_mode.lower() == "external":
|
||||
return None
|
||||
if _SCHEME_RE.match(target):
|
||||
return None
|
||||
|
||||
target = urllib.parse.unquote(target)
|
||||
|
||||
if "\\" in target:
|
||||
raise ValueError(f"relationship target is not a POSIX part name: {target!r}")
|
||||
|
||||
if target.startswith("/"):
|
||||
joined = target.lstrip("/")
|
||||
else:
|
||||
joined = posixpath.join(posixpath.dirname(source_part), target)
|
||||
|
||||
parts: list[str] = []
|
||||
for segment in posixpath.normpath(joined).split("/"):
|
||||
if segment in ("", "."):
|
||||
continue
|
||||
if segment == "..":
|
||||
if not parts:
|
||||
raise ValueError(f"relationship target escapes the package: {target!r}")
|
||||
parts.pop()
|
||||
else:
|
||||
parts.append(segment)
|
||||
|
||||
if not parts:
|
||||
raise ValueError(f"relationship target resolves to nothing: {target!r}")
|
||||
return "/".join(parts)
|
||||
|
||||
|
||||
def rels_source_part(rels_file: Path, unpacked_dir: Path) -> str:
|
||||
owner_dir = rels_file.parent.parent.relative_to(unpacked_dir)
|
||||
return posixpath.join(owner_dir.as_posix(), rels_file.name[: -len(".rels")]).lstrip("./")
|
||||
|
||||
|
||||
def part_text(data: bytes) -> str:
|
||||
return data.decode("utf-8", "surrogateescape")
|
||||
|
||||
|
||||
XML_SPACE = " \t\r\n"
|
||||
|
||||
|
||||
def rendered_text(text: str, preserve: bool) -> str:
|
||||
return text if preserve else text.strip(XML_SPACE)
|
||||
|
||||
|
||||
def safe_extract(zf: zipfile.ZipFile, dest: Path) -> None:
|
||||
dest = dest.resolve()
|
||||
for m in zf.infolist():
|
||||
if stat.S_ISLNK(m.external_attr >> 16):
|
||||
raise ValueError(f"symlink archive entry not allowed: {m.filename!r}")
|
||||
target = (dest / m.filename).resolve()
|
||||
if not target.is_relative_to(dest):
|
||||
raise ValueError(f"unsafe archive entry: {m.filename!r}")
|
||||
zf.extract(m, dest)
|
||||
|
||||
|
||||
def rezip(src_dir: Path, out_path: Path) -> None:
|
||||
files = sorted(p for p in src_dir.rglob("*") if p.is_file())
|
||||
ct = src_dir / "[Content_Types].xml"
|
||||
fd, tmp_name = tempfile.mkstemp(
|
||||
prefix=out_path.name + ".", suffix=".tmp", dir=out_path.parent
|
||||
)
|
||||
tmp_out = Path(tmp_name)
|
||||
try:
|
||||
with os.fdopen(fd, "wb") as fh:
|
||||
with zipfile.ZipFile(fh, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
if ct.exists():
|
||||
zf.write(ct, ct.relative_to(src_dir), compress_type=zipfile.ZIP_STORED)
|
||||
for f in files:
|
||||
if f == ct:
|
||||
continue
|
||||
zf.write(f, f.relative_to(src_dir))
|
||||
if out_path.exists():
|
||||
mode = out_path.stat().st_mode & 0o777
|
||||
else:
|
||||
umask = os.umask(0)
|
||||
os.umask(umask)
|
||||
mode = 0o666 & ~umask
|
||||
os.chmod(tmp_out, mode)
|
||||
os.replace(tmp_out, out_path)
|
||||
finally:
|
||||
if tmp_out.exists():
|
||||
tmp_out.unlink()
|
||||
|
||||
@@ -1,199 +0,0 @@
|
||||
"""Merge adjacent runs with identical formatting in DOCX.
|
||||
|
||||
Merges adjacent <w:r> elements that have identical <w:rPr> properties.
|
||||
Works on runs in paragraphs and inside tracked changes (<w:ins>, <w:del>).
|
||||
|
||||
Also:
|
||||
- Removes rsid attributes from runs (revision metadata that doesn't affect rendering)
|
||||
- Removes proofErr elements (spell/grammar markers that block merging)
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.minidom
|
||||
|
||||
|
||||
def merge_runs(input_dir: str) -> tuple[int, str]:
|
||||
doc_xml = Path(input_dir) / "word" / "document.xml"
|
||||
|
||||
if not doc_xml.exists():
|
||||
return 0, f"Error: {doc_xml} not found"
|
||||
|
||||
try:
|
||||
dom = defusedxml.minidom.parseString(doc_xml.read_text(encoding="utf-8"))
|
||||
root = dom.documentElement
|
||||
|
||||
_remove_elements(root, "proofErr")
|
||||
_strip_run_rsid_attrs(root)
|
||||
|
||||
containers = {run.parentNode for run in _find_elements(root, "r")}
|
||||
|
||||
merge_count = 0
|
||||
for container in containers:
|
||||
merge_count += _merge_runs_in(container)
|
||||
|
||||
doc_xml.write_bytes(dom.toxml(encoding="UTF-8"))
|
||||
return merge_count, f"Merged {merge_count} runs"
|
||||
|
||||
except Exception as e:
|
||||
return 0, f"Error: {e}"
|
||||
|
||||
|
||||
|
||||
|
||||
def _find_elements(root, tag: str) -> list:
|
||||
results = []
|
||||
|
||||
def traverse(node):
|
||||
if node.nodeType == node.ELEMENT_NODE:
|
||||
name = node.localName or node.tagName
|
||||
if name == tag or name.endswith(f":{tag}"):
|
||||
results.append(node)
|
||||
for child in node.childNodes:
|
||||
traverse(child)
|
||||
|
||||
traverse(root)
|
||||
return results
|
||||
|
||||
|
||||
def _get_child(parent, tag: str):
|
||||
for child in parent.childNodes:
|
||||
if child.nodeType == child.ELEMENT_NODE:
|
||||
name = child.localName or child.tagName
|
||||
if name == tag or name.endswith(f":{tag}"):
|
||||
return child
|
||||
return None
|
||||
|
||||
|
||||
def _get_children(parent, tag: str) -> list:
|
||||
results = []
|
||||
for child in parent.childNodes:
|
||||
if child.nodeType == child.ELEMENT_NODE:
|
||||
name = child.localName or child.tagName
|
||||
if name == tag or name.endswith(f":{tag}"):
|
||||
results.append(child)
|
||||
return results
|
||||
|
||||
|
||||
def _is_adjacent(elem1, elem2) -> bool:
|
||||
node = elem1.nextSibling
|
||||
while node:
|
||||
if node == elem2:
|
||||
return True
|
||||
if node.nodeType == node.ELEMENT_NODE:
|
||||
return False
|
||||
if node.nodeType == node.TEXT_NODE and node.data.strip():
|
||||
return False
|
||||
node = node.nextSibling
|
||||
return False
|
||||
|
||||
|
||||
|
||||
|
||||
def _remove_elements(root, tag: str):
|
||||
for elem in _find_elements(root, tag):
|
||||
if elem.parentNode:
|
||||
elem.parentNode.removeChild(elem)
|
||||
|
||||
|
||||
def _strip_run_rsid_attrs(root):
|
||||
for run in _find_elements(root, "r"):
|
||||
for attr in list(run.attributes.values()):
|
||||
if "rsid" in attr.name.lower():
|
||||
run.removeAttribute(attr.name)
|
||||
|
||||
|
||||
|
||||
|
||||
def _merge_runs_in(container) -> int:
|
||||
merge_count = 0
|
||||
run = _first_child_run(container)
|
||||
|
||||
while run:
|
||||
while True:
|
||||
next_elem = _next_element_sibling(run)
|
||||
if next_elem and _is_run(next_elem) and _can_merge(run, next_elem):
|
||||
_merge_run_content(run, next_elem)
|
||||
container.removeChild(next_elem)
|
||||
merge_count += 1
|
||||
else:
|
||||
break
|
||||
|
||||
_consolidate_text(run)
|
||||
run = _next_sibling_run(run)
|
||||
|
||||
return merge_count
|
||||
|
||||
|
||||
def _first_child_run(container):
|
||||
for child in container.childNodes:
|
||||
if child.nodeType == child.ELEMENT_NODE and _is_run(child):
|
||||
return child
|
||||
return None
|
||||
|
||||
|
||||
def _next_element_sibling(node):
|
||||
sibling = node.nextSibling
|
||||
while sibling:
|
||||
if sibling.nodeType == sibling.ELEMENT_NODE:
|
||||
return sibling
|
||||
sibling = sibling.nextSibling
|
||||
return None
|
||||
|
||||
|
||||
def _next_sibling_run(node):
|
||||
sibling = node.nextSibling
|
||||
while sibling:
|
||||
if sibling.nodeType == sibling.ELEMENT_NODE:
|
||||
if _is_run(sibling):
|
||||
return sibling
|
||||
sibling = sibling.nextSibling
|
||||
return None
|
||||
|
||||
|
||||
def _is_run(node) -> bool:
|
||||
name = node.localName or node.tagName
|
||||
return name == "r" or name.endswith(":r")
|
||||
|
||||
|
||||
def _can_merge(run1, run2) -> bool:
|
||||
rpr1 = _get_child(run1, "rPr")
|
||||
rpr2 = _get_child(run2, "rPr")
|
||||
|
||||
if (rpr1 is None) != (rpr2 is None):
|
||||
return False
|
||||
if rpr1 is None:
|
||||
return True
|
||||
return rpr1.toxml() == rpr2.toxml()
|
||||
|
||||
|
||||
def _merge_run_content(target, source):
|
||||
for child in list(source.childNodes):
|
||||
if child.nodeType == child.ELEMENT_NODE:
|
||||
name = child.localName or child.tagName
|
||||
if name != "rPr" and not name.endswith(":rPr"):
|
||||
target.appendChild(child)
|
||||
|
||||
|
||||
def _consolidate_text(run):
|
||||
t_elements = _get_children(run, "t")
|
||||
|
||||
for i in range(len(t_elements) - 1, 0, -1):
|
||||
curr, prev = t_elements[i], t_elements[i - 1]
|
||||
|
||||
if _is_adjacent(prev, curr):
|
||||
prev_text = prev.firstChild.data if prev.firstChild else ""
|
||||
curr_text = curr.firstChild.data if curr.firstChild else ""
|
||||
merged = prev_text + curr_text
|
||||
|
||||
if prev.firstChild:
|
||||
prev.firstChild.data = merged
|
||||
else:
|
||||
prev.appendChild(run.ownerDocument.createTextNode(merged))
|
||||
|
||||
if merged.startswith(" ") or merged.endswith(" "):
|
||||
prev.setAttribute("xml:space", "preserve")
|
||||
elif prev.hasAttribute("xml:space"):
|
||||
prev.removeAttribute("xml:space")
|
||||
|
||||
run.removeChild(curr)
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Find chart XML that PowerPoint refuses but the schema accepts.
|
||||
|
||||
Detection only: for either fault more than one repair is valid, and only the
|
||||
author knows which was meant.
|
||||
"""
|
||||
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Mapping
|
||||
|
||||
from . import part_text
|
||||
|
||||
|
||||
_CHART_PART_RE = re.compile(r"ppt/charts/chart\d+\.xml")
|
||||
|
||||
_GROUPING_RE = re.compile(r"""<c:grouping\b[^>]*?\bval=["'](\w+)["']""")
|
||||
_DLBL_POS_RE = re.compile(r"""<c:dLblPos\b[^>]*?\bval=["'](\w+)["']""")
|
||||
|
||||
def _strip_ext_lst(text: str) -> str:
|
||||
out, cursor = [], 0
|
||||
for lo, hi in _ext_lst_spans(text):
|
||||
out.append(text[cursor:lo])
|
||||
cursor = hi
|
||||
out.append(text[cursor:])
|
||||
return "".join(out)
|
||||
|
||||
_BAR_GROUP_RE = re.compile(r"<c:(bar3DChart|barChart)\b[^>]*(?<!/)>.*?</c:\1\s*>", re.DOTALL)
|
||||
|
||||
STACKED_GROUPINGS = frozenset({"stacked", "percentStacked"})
|
||||
ILLEGAL_ON_STACKED = frozenset({"outEnd"})
|
||||
LEGAL_ON_STACKED = ("ctr", "inEnd", "inBase")
|
||||
|
||||
|
||||
def _check_stacked_label_positions(part: str, xml: str) -> list[str]:
|
||||
problems: list[str] = []
|
||||
for match in _BAR_GROUP_RE.finditer(xml):
|
||||
block = _strip_ext_lst(match.group(0))
|
||||
group = match.group(1)
|
||||
|
||||
grouping = _GROUPING_RE.search(block)
|
||||
if grouping is None or grouping.group(1) not in STACKED_GROUPINGS:
|
||||
continue
|
||||
|
||||
bad = [p for p in _DLBL_POS_RE.findall(block) if p in ILLEGAL_ON_STACKED]
|
||||
for pos in sorted(set(bad)):
|
||||
problems.append(
|
||||
f'{part}: {bad.count(pos)} data label(s) use dLblPos="{pos}" on a '
|
||||
f"{grouping.group(1)} {group}; PowerPoint allows only "
|
||||
f"{', '.join(LEGAL_ON_STACKED)} there"
|
||||
)
|
||||
return problems
|
||||
|
||||
|
||||
|
||||
_ANY_CHART_GROUP_RE = re.compile(r"<c:(\w+Chart)\b[^>]*(?<!/)>.*?</c:\1\s*>", re.DOTALL)
|
||||
|
||||
_AXID_RE = re.compile(
|
||||
r"""\s*<c:axId\b[^>]*?\bval=["'](-?\d+)["']\s*(?:/>|>\s*</c:axId\s*>)"""
|
||||
)
|
||||
|
||||
_AXIS_DECL_RE = re.compile(
|
||||
r"""<c:(catAx|valAx|serAx|dateAx)\b[^>]*(?<!/)>\s*<c:axId\b[^>]*?\bval=["'](-?\d+)["']"""
|
||||
)
|
||||
|
||||
AXID_LIMIT = {
|
||||
"barChart": 2, "lineChart": 2, "areaChart": 2, "scatterChart": 2,
|
||||
"bubbleChart": 2, "radarChart": 2, "stockChart": 2,
|
||||
"bar3DChart": 3, "line3DChart": 3, "area3DChart": 3,
|
||||
"surfaceChart": 3, "surface3DChart": 3,
|
||||
}
|
||||
|
||||
AXID_MINIMUM = {
|
||||
"barChart": 2, "lineChart": 2, "areaChart": 2, "scatterChart": 2,
|
||||
"bubbleChart": 2, "radarChart": 2, "stockChart": 2,
|
||||
"bar3DChart": 2, "area3DChart": 2, "surfaceChart": 2,
|
||||
"line3DChart": 3, "surface3DChart": 3,
|
||||
}
|
||||
|
||||
|
||||
def _declared_axes(xml: str) -> dict[str, list[str]]:
|
||||
axes: dict[str, list[str]] = {}
|
||||
for kind, axid in _AXIS_DECL_RE.findall(xml):
|
||||
axes.setdefault(kind, []).append(axid)
|
||||
return axes
|
||||
|
||||
|
||||
def _canonical_ids(axes: dict[str, list[str]], limit: int) -> list[str] | None:
|
||||
category = axes.get("catAx", []) + axes.get("dateAx", [])
|
||||
value = axes.get("valAx", [])
|
||||
series = axes.get("serAx", [])
|
||||
if len(category) != 1 or len(value) != 1 or len(series) > 1:
|
||||
return None
|
||||
ids = [category[0], value[0]]
|
||||
if limit >= 3 and series:
|
||||
ids.append(series[0])
|
||||
return ids
|
||||
|
||||
|
||||
def _undeclared_axes(kind: str, block: str, axes: dict[str, list[str]]) -> list[str] | None:
|
||||
if kind not in AXID_LIMIT:
|
||||
return None
|
||||
ids = _AXID_RE.findall(block)
|
||||
declared = {i for group in axes.values() for i in group}
|
||||
if len([i for i in ids if i in declared]) >= 2:
|
||||
return None
|
||||
return ids
|
||||
|
||||
|
||||
def _check_chart_axis_references(part: str, xml: str) -> list[str]:
|
||||
axes = _declared_axes(xml)
|
||||
problems: list[str] = []
|
||||
declared = {i for group in axes.values() for i in group}
|
||||
for match in _ANY_CHART_GROUP_RE.finditer(xml):
|
||||
kind, block = match.group(1), match.group(0)
|
||||
ids = _undeclared_axes(kind, block, axes)
|
||||
if ids is None:
|
||||
continue
|
||||
if not ids:
|
||||
problems.append(
|
||||
f"{part}: <c:{kind}> declares no <c:axId> this part can resolve; a chart "
|
||||
f"group needs {AXID_MINIMUM[kind]}, and PowerPoint discards one with fewer"
|
||||
)
|
||||
continue
|
||||
dead = [i for i in ids if i not in declared]
|
||||
canonical = _canonical_ids(axes, AXID_LIMIT[kind])
|
||||
if canonical is not None and len(canonical) >= AXID_MINIMUM[kind]:
|
||||
hint = f"Fix: point them at the axes this part declares ({', '.join(canonical)})"
|
||||
else:
|
||||
hint = ("Fix: the part declares several axes of a kind -- declare the "
|
||||
"secondary axes the series expects, or drop them")
|
||||
detail = (f"of which {', '.join(dead)} name no declared axis"
|
||||
if dead else f"only {len(ids)} of which this part declares")
|
||||
problems.append(
|
||||
f"{part}: <c:{kind}> references axId {', '.join(ids)}, {detail}, "
|
||||
f"leaving fewer than two live axes; PowerPoint discards the chart. {hint}"
|
||||
)
|
||||
return problems
|
||||
|
||||
|
||||
def _ext_lst_spans(text: str) -> list[tuple[int, int]]:
|
||||
spans: list[tuple[int, int]] = []
|
||||
depth = 0
|
||||
start = 0
|
||||
for match in re.finditer(r"<(/?)c:extLst\b[^>]*?(/?)>", text):
|
||||
closing, self_closing = match.group(1), match.group(2)
|
||||
if self_closing:
|
||||
continue
|
||||
if closing:
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
spans.append((start, match.end()))
|
||||
else:
|
||||
if depth == 0:
|
||||
start = match.start()
|
||||
depth += 1
|
||||
return spans
|
||||
|
||||
|
||||
CHART_CHECKS = (_check_stacked_label_positions, _check_chart_axis_references)
|
||||
|
||||
|
||||
def find_chart_problems(files: Mapping[str, bytes]) -> list[str]:
|
||||
problems: list[str] = []
|
||||
for part in sorted(n for n in files if _CHART_PART_RE.fullmatch(n)):
|
||||
xml = part_text(files[part])
|
||||
for check in CHART_CHECKS:
|
||||
problems.extend(check(part, xml))
|
||||
return problems
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Pick the slide-XML schema errors PowerPoint refuses the file over.
|
||||
|
||||
A denylist over lxml's messages, so an unrecognised error class is a miss rather
|
||||
than a false alarm.
|
||||
"""
|
||||
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
SLIDE_PART_RE = re.compile(
|
||||
r"ppt/(slides|slideLayouts|slideMasters|notesSlides|notesMasters|handoutMasters)"
|
||||
r"/[^/]+\.xml"
|
||||
)
|
||||
|
||||
FATAL_SLIDE_ERRORS: tuple[tuple[re.Pattern[str], str], ...] = (
|
||||
(
|
||||
re.compile(r"\}tableStyleId': This element is not expected"),
|
||||
"two <a:tableStyleId> in one <a:tblPr> (the schema allows one)",
|
||||
),
|
||||
(
|
||||
re.compile(r"\}srgbClr', attribute 'val'"),
|
||||
"a colour that is not six hex digits",
|
||||
),
|
||||
(
|
||||
re.compile(r"\}txBody': Missing child element"),
|
||||
"a <p:txBody> with no children",
|
||||
),
|
||||
(
|
||||
re.compile(r"\}miter', attribute 'lim'"),
|
||||
'a line join with lim="NaN"',
|
||||
),
|
||||
(
|
||||
re.compile(r"\}uLnTx': This element is not expected"),
|
||||
"<a:uLnTx> in a position the schema forbids",
|
||||
),
|
||||
(
|
||||
re.compile(r"\}overrideClrMapping': This element is not expected"),
|
||||
"<p:overrideClrMapping> in a position the schema forbids",
|
||||
),
|
||||
(
|
||||
re.compile(r"\}nvGrpSpPr': Missing child element"),
|
||||
"a <p:nvGrpSpPr> with no children",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def is_schema_verdict(error: str) -> bool:
|
||||
return error.startswith("Element ")
|
||||
|
||||
|
||||
def fatal_slide_errors(errors: set[str]) -> list[str]:
|
||||
out = []
|
||||
for error in sorted(errors):
|
||||
for pattern, meaning in FATAL_SLIDE_ERRORS:
|
||||
if pattern.search(error):
|
||||
out.append(f"{meaning}: {error}")
|
||||
break
|
||||
return out
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Find masters sharing a theme part in the way PowerPoint refuses to open.
|
||||
|
||||
Reports only; the fix is to move <p:notesMasterIdLst> back to directly after
|
||||
<p:sldIdLst> in ppt/presentation.xml.
|
||||
"""
|
||||
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import posixpath
|
||||
import re
|
||||
from typing import Mapping
|
||||
|
||||
from . import part_text
|
||||
|
||||
THEME_REL_TYPE = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/theme"
|
||||
|
||||
_MASTER_RE = re.compile(
|
||||
r"^ppt/(?P<group>slideMasters|notesMasters|handoutMasters)/"
|
||||
r"(?:slide|notes|handout)Master(?P<num>\d+)\.xml$"
|
||||
)
|
||||
_GROUP_ORDER = {"slideMasters": 0, "notesMasters": 1, "handoutMasters": 2}
|
||||
|
||||
_RELATIONSHIP_RE = re.compile(
|
||||
r"<Relationship\b[^>]*?(?:/>|>.*?</Relationship\s*>)", re.DOTALL
|
||||
)
|
||||
|
||||
|
||||
def _sort_key(name: str) -> tuple[int, int]:
|
||||
m = _MASTER_RE.match(name)
|
||||
assert m is not None
|
||||
return (_GROUP_ORDER[m.group("group")], int(m.group("num")))
|
||||
|
||||
|
||||
def _rels_path(part: str) -> str:
|
||||
directory, base = posixpath.split(part)
|
||||
return f"{directory}/_rels/{base}.rels"
|
||||
|
||||
|
||||
def _resolve(rels_path: str, target: str) -> str:
|
||||
if target.startswith("/"):
|
||||
return target.lstrip("/")
|
||||
part_dir = posixpath.dirname(posixpath.dirname(rels_path))
|
||||
return posixpath.normpath(posixpath.join(part_dir, target))
|
||||
|
||||
|
||||
def _theme_rel(files: Mapping[str, bytes], master: str):
|
||||
rels_path = _rels_path(master)
|
||||
rels = files.get(rels_path)
|
||||
if rels is None:
|
||||
return None
|
||||
for element in _RELATIONSHIP_RE.findall(part_text(rels)):
|
||||
if f'Type="{THEME_REL_TYPE}"' not in element:
|
||||
continue
|
||||
target = re.search(r'\bTarget="([^"]+)"', element)
|
||||
if target is None:
|
||||
continue
|
||||
return rels_path, element, _resolve(rels_path, target.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def _masters(files: Mapping[str, bytes]) -> list[str]:
|
||||
return sorted((n for n in files if _MASTER_RE.match(n)), key=_sort_key)
|
||||
|
||||
|
||||
_PRESENTATION = "ppt/presentation.xml"
|
||||
_NOTES_MASTERS = "ppt/notesMasters/"
|
||||
_IGNORABLE_RE = re.compile(r"<!--.*?-->|<\?.*?\?>", re.DOTALL)
|
||||
_AFTER_SLDIDLST_RE = re.compile(
|
||||
r"<p:sldIdLst\b(?:[^>]*/>|[^>]*>.*?</p:sldIdLst\s*>)\s*(<[^>\s/]+)", re.DOTALL
|
||||
)
|
||||
|
||||
|
||||
def _notes_master_share_is_inert(files: Mapping[str, bytes]) -> bool:
|
||||
data = files.get(_PRESENTATION)
|
||||
if data is None:
|
||||
return False
|
||||
match = _AFTER_SLDIDLST_RE.search(_IGNORABLE_RE.sub("", part_text(data)))
|
||||
return match is not None and match.group(1) == "<p:notesMasterIdLst"
|
||||
|
||||
|
||||
def _shares(files: Mapping[str, bytes]):
|
||||
owner: dict[str, str] = {}
|
||||
for master in _masters(files):
|
||||
found = _theme_rel(files, master)
|
||||
if found is None:
|
||||
continue
|
||||
rels_path, element, theme = found
|
||||
if theme not in files:
|
||||
continue
|
||||
if theme in owner:
|
||||
yield master, rels_path, element, theme, owner[theme]
|
||||
else:
|
||||
owner[theme] = master
|
||||
|
||||
|
||||
def _is_inert(master: str, inert_notes: bool) -> bool:
|
||||
return inert_notes and master.startswith(_NOTES_MASTERS)
|
||||
|
||||
|
||||
def find_shared_master_themes(files: Mapping[str, bytes]) -> list[str]:
|
||||
return [
|
||||
f"{master} shares {theme} with {first}"
|
||||
for master, _, _, theme, first in _shares(files)
|
||||
]
|
||||
|
||||
|
||||
def live_shared_master_themes(files: Mapping[str, bytes]) -> list[str]:
|
||||
inert_notes = _notes_master_share_is_inert(files)
|
||||
return [
|
||||
f"{master} shares {theme} with {first}"
|
||||
for master, _, _, theme, first in _shares(files)
|
||||
if not _is_inert(master, inert_notes)
|
||||
]
|
||||
@@ -1,197 +0,0 @@
|
||||
"""Simplify tracked changes by merging adjacent w:ins or w:del elements.
|
||||
|
||||
Merges adjacent <w:ins> elements from the same author into a single element.
|
||||
Same for <w:del> elements. This makes heavily-redlined documents easier to
|
||||
work with by reducing the number of tracked change wrappers.
|
||||
|
||||
Rules:
|
||||
- Only merges w:ins with w:ins, w:del with w:del (same element type)
|
||||
- Only merges if same author (ignores timestamp differences)
|
||||
- Only merges if truly adjacent (only whitespace between them)
|
||||
"""
|
||||
|
||||
import xml.etree.ElementTree as ET
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.minidom
|
||||
|
||||
WORD_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
||||
|
||||
|
||||
def simplify_redlines(input_dir: str) -> tuple[int, str]:
|
||||
doc_xml = Path(input_dir) / "word" / "document.xml"
|
||||
|
||||
if not doc_xml.exists():
|
||||
return 0, f"Error: {doc_xml} not found"
|
||||
|
||||
try:
|
||||
dom = defusedxml.minidom.parseString(doc_xml.read_text(encoding="utf-8"))
|
||||
root = dom.documentElement
|
||||
|
||||
merge_count = 0
|
||||
|
||||
containers = _find_elements(root, "p") + _find_elements(root, "tc")
|
||||
|
||||
for container in containers:
|
||||
merge_count += _merge_tracked_changes_in(container, "ins")
|
||||
merge_count += _merge_tracked_changes_in(container, "del")
|
||||
|
||||
doc_xml.write_bytes(dom.toxml(encoding="UTF-8"))
|
||||
return merge_count, f"Simplified {merge_count} tracked changes"
|
||||
|
||||
except Exception as e:
|
||||
return 0, f"Error: {e}"
|
||||
|
||||
|
||||
def _merge_tracked_changes_in(container, tag: str) -> int:
|
||||
merge_count = 0
|
||||
|
||||
tracked = [
|
||||
child
|
||||
for child in container.childNodes
|
||||
if child.nodeType == child.ELEMENT_NODE and _is_element(child, tag)
|
||||
]
|
||||
|
||||
if len(tracked) < 2:
|
||||
return 0
|
||||
|
||||
i = 0
|
||||
while i < len(tracked) - 1:
|
||||
curr = tracked[i]
|
||||
next_elem = tracked[i + 1]
|
||||
|
||||
if _can_merge_tracked(curr, next_elem):
|
||||
_merge_tracked_content(curr, next_elem)
|
||||
container.removeChild(next_elem)
|
||||
tracked.pop(i + 1)
|
||||
merge_count += 1
|
||||
else:
|
||||
i += 1
|
||||
|
||||
return merge_count
|
||||
|
||||
|
||||
def _is_element(node, tag: str) -> bool:
|
||||
name = node.localName or node.tagName
|
||||
return name == tag or name.endswith(f":{tag}")
|
||||
|
||||
|
||||
def _get_author(elem) -> str:
|
||||
author = elem.getAttribute("w:author")
|
||||
if not author:
|
||||
for attr in elem.attributes.values():
|
||||
if attr.localName == "author" or attr.name.endswith(":author"):
|
||||
return attr.value
|
||||
return author
|
||||
|
||||
|
||||
def _can_merge_tracked(elem1, elem2) -> bool:
|
||||
if _get_author(elem1) != _get_author(elem2):
|
||||
return False
|
||||
|
||||
node = elem1.nextSibling
|
||||
while node and node != elem2:
|
||||
if node.nodeType == node.ELEMENT_NODE:
|
||||
return False
|
||||
if node.nodeType == node.TEXT_NODE and node.data.strip():
|
||||
return False
|
||||
node = node.nextSibling
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _merge_tracked_content(target, source):
|
||||
while source.firstChild:
|
||||
child = source.firstChild
|
||||
source.removeChild(child)
|
||||
target.appendChild(child)
|
||||
|
||||
|
||||
def _find_elements(root, tag: str) -> list:
|
||||
results = []
|
||||
|
||||
def traverse(node):
|
||||
if node.nodeType == node.ELEMENT_NODE:
|
||||
name = node.localName or node.tagName
|
||||
if name == tag or name.endswith(f":{tag}"):
|
||||
results.append(node)
|
||||
for child in node.childNodes:
|
||||
traverse(child)
|
||||
|
||||
traverse(root)
|
||||
return results
|
||||
|
||||
|
||||
def get_tracked_change_authors(doc_xml_path: Path) -> dict[str, int]:
|
||||
if not doc_xml_path.exists():
|
||||
return {}
|
||||
|
||||
try:
|
||||
tree = ET.parse(doc_xml_path)
|
||||
root = tree.getroot()
|
||||
except ET.ParseError:
|
||||
return {}
|
||||
|
||||
namespaces = {"w": WORD_NS}
|
||||
author_attr = f"{{{WORD_NS}}}author"
|
||||
|
||||
authors: dict[str, int] = {}
|
||||
for tag in ["ins", "del"]:
|
||||
for elem in root.findall(f".//w:{tag}", namespaces):
|
||||
author = elem.get(author_attr)
|
||||
if author:
|
||||
authors[author] = authors.get(author, 0) + 1
|
||||
|
||||
return authors
|
||||
|
||||
|
||||
def _get_authors_from_docx(docx_path: Path) -> dict[str, int]:
|
||||
try:
|
||||
with zipfile.ZipFile(docx_path, "r") as zf:
|
||||
if "word/document.xml" not in zf.namelist():
|
||||
return {}
|
||||
with zf.open("word/document.xml") as f:
|
||||
tree = ET.parse(f)
|
||||
root = tree.getroot()
|
||||
|
||||
namespaces = {"w": WORD_NS}
|
||||
author_attr = f"{{{WORD_NS}}}author"
|
||||
|
||||
authors: dict[str, int] = {}
|
||||
for tag in ["ins", "del"]:
|
||||
for elem in root.findall(f".//w:{tag}", namespaces):
|
||||
author = elem.get(author_attr)
|
||||
if author:
|
||||
authors[author] = authors.get(author, 0) + 1
|
||||
return authors
|
||||
except (zipfile.BadZipFile, ET.ParseError):
|
||||
return {}
|
||||
|
||||
|
||||
def infer_author(modified_dir: Path, original_docx: Path, default: str = "Claude") -> str:
|
||||
modified_xml = modified_dir / "word" / "document.xml"
|
||||
modified_authors = get_tracked_change_authors(modified_xml)
|
||||
|
||||
if not modified_authors:
|
||||
return default
|
||||
|
||||
original_authors = _get_authors_from_docx(original_docx)
|
||||
|
||||
new_changes: dict[str, int] = {}
|
||||
for author, count in modified_authors.items():
|
||||
original_count = original_authors.get(author, 0)
|
||||
diff = count - original_count
|
||||
if diff > 0:
|
||||
new_changes[author] = diff
|
||||
|
||||
if not new_changes:
|
||||
return default
|
||||
|
||||
if len(new_changes) == 1:
|
||||
return next(iter(new_changes))
|
||||
|
||||
raise ValueError(
|
||||
f"Multiple authors added new changes: {new_changes}. "
|
||||
"Cannot infer which author to validate."
|
||||
)
|
||||
@@ -1,159 +0,0 @@
|
||||
"""Pack a directory into a DOCX, PPTX, or XLSX file.
|
||||
|
||||
Validates with auto-repair, condenses XML formatting, and creates the Office file.
|
||||
|
||||
Usage:
|
||||
python pack.py <input_directory> <output_file> [--original <file>] [--validate true|false]
|
||||
|
||||
Examples:
|
||||
python pack.py unpacked/ output.docx --original input.docx
|
||||
python pack.py unpacked/ output.pptx --validate false
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import shutil
|
||||
import tempfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.minidom
|
||||
|
||||
from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator
|
||||
|
||||
def pack(
|
||||
input_directory: str,
|
||||
output_file: str,
|
||||
original_file: str | None = None,
|
||||
validate: bool = True,
|
||||
infer_author_func=None,
|
||||
) -> tuple[None, str]:
|
||||
input_dir = Path(input_directory)
|
||||
output_path = Path(output_file)
|
||||
suffix = output_path.suffix.lower()
|
||||
|
||||
if not input_dir.is_dir():
|
||||
return None, f"Error: {input_dir} is not a directory"
|
||||
|
||||
if suffix not in {".docx", ".pptx", ".xlsx"}:
|
||||
return None, f"Error: {output_file} must be a .docx, .pptx, or .xlsx file"
|
||||
|
||||
if validate and original_file:
|
||||
original_path = Path(original_file)
|
||||
if original_path.exists():
|
||||
success, output = _run_validation(
|
||||
input_dir, original_path, suffix, infer_author_func
|
||||
)
|
||||
if output:
|
||||
print(output)
|
||||
if not success:
|
||||
return None, f"Error: Validation failed for {input_dir}"
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_content_dir = Path(temp_dir) / "content"
|
||||
shutil.copytree(input_dir, temp_content_dir)
|
||||
|
||||
for pattern in ["*.xml", "*.rels"]:
|
||||
for xml_file in temp_content_dir.rglob(pattern):
|
||||
_condense_xml(xml_file)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with zipfile.ZipFile(output_path, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
for f in temp_content_dir.rglob("*"):
|
||||
if f.is_file():
|
||||
zf.write(f, f.relative_to(temp_content_dir))
|
||||
|
||||
return None, f"Successfully packed {input_dir} to {output_file}"
|
||||
|
||||
|
||||
def _run_validation(
|
||||
unpacked_dir: Path,
|
||||
original_file: Path,
|
||||
suffix: str,
|
||||
infer_author_func=None,
|
||||
) -> tuple[bool, str | None]:
|
||||
output_lines = []
|
||||
validators = []
|
||||
|
||||
if suffix == ".docx":
|
||||
author = "Claude"
|
||||
if infer_author_func:
|
||||
try:
|
||||
author = infer_author_func(unpacked_dir, original_file)
|
||||
except ValueError as e:
|
||||
print(f"Warning: {e} Using default author 'Claude'.", file=sys.stderr)
|
||||
|
||||
validators = [
|
||||
DOCXSchemaValidator(unpacked_dir, original_file),
|
||||
RedliningValidator(unpacked_dir, original_file, author=author),
|
||||
]
|
||||
elif suffix == ".pptx":
|
||||
validators = [PPTXSchemaValidator(unpacked_dir, original_file)]
|
||||
|
||||
if not validators:
|
||||
return True, None
|
||||
|
||||
total_repairs = sum(v.repair() for v in validators)
|
||||
if total_repairs:
|
||||
output_lines.append(f"Auto-repaired {total_repairs} issue(s)")
|
||||
|
||||
success = all(v.validate() for v in validators)
|
||||
|
||||
if success:
|
||||
output_lines.append("All validations PASSED!")
|
||||
|
||||
return success, "\n".join(output_lines) if output_lines else None
|
||||
|
||||
|
||||
def _condense_xml(xml_file: Path) -> None:
|
||||
try:
|
||||
with open(xml_file, encoding="utf-8") as f:
|
||||
dom = defusedxml.minidom.parse(f)
|
||||
|
||||
for element in dom.getElementsByTagName("*"):
|
||||
if element.tagName.endswith(":t"):
|
||||
continue
|
||||
|
||||
for child in list(element.childNodes):
|
||||
if (
|
||||
child.nodeType == child.TEXT_NODE
|
||||
and child.nodeValue
|
||||
and child.nodeValue.strip() == ""
|
||||
) or child.nodeType == child.COMMENT_NODE:
|
||||
element.removeChild(child)
|
||||
|
||||
xml_file.write_bytes(dom.toxml(encoding="UTF-8"))
|
||||
except Exception as e:
|
||||
print(f"ERROR: Failed to parse {xml_file.name}: {e}", file=sys.stderr)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Pack a directory into a DOCX, PPTX, or XLSX file"
|
||||
)
|
||||
parser.add_argument("input_directory", help="Unpacked Office document directory")
|
||||
parser.add_argument("output_file", help="Output Office file (.docx/.pptx/.xlsx)")
|
||||
parser.add_argument(
|
||||
"--original",
|
||||
help="Original file for validation comparison",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--validate",
|
||||
type=lambda x: x.lower() == "true",
|
||||
default=True,
|
||||
metavar="true|false",
|
||||
help="Run validation with auto-repair (default: true)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
_, message = pack(
|
||||
args.input_directory,
|
||||
args.output_file,
|
||||
original_file=args.original,
|
||||
validate=args.validate,
|
||||
)
|
||||
print(message)
|
||||
|
||||
if "Error" in message:
|
||||
sys.exit(1)
|
||||
@@ -4,20 +4,23 @@ sockets may be blocked (e.g., sandboxed VMs). Detects the restriction
|
||||
at runtime and applies an LD_PRELOAD shim if needed.
|
||||
|
||||
Usage:
|
||||
from office.soffice import run_soffice, get_soffice_env
|
||||
from office.soffice import run_soffice
|
||||
|
||||
# Option 1 – run soffice directly
|
||||
result = run_soffice(["--headless", "--convert-to", "pdf", "input.docx"])
|
||||
|
||||
# Option 2 – get env dict for your own subprocess calls
|
||||
env = get_soffice_env()
|
||||
subprocess.run(["soffice", ...], env=env)
|
||||
Call soffice through run_soffice, not through subprocess with get_soffice_env():
|
||||
the env dict carries the shim but names no user profile, and a non-root sandbox
|
||||
cannot bootstrap the default one -- soffice aborts with "User installation could
|
||||
not be completed" and converts nothing. get_soffice_env() stays public for the
|
||||
callers that build their own argv (they must pass -env:UserInstallation too).
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import tempfile
|
||||
from collections.abc import Iterable
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -32,9 +35,15 @@ def get_soffice_env() -> dict:
|
||||
return env
|
||||
|
||||
|
||||
def run_soffice(args: list[str], **kwargs) -> subprocess.CompletedProcess:
|
||||
env = get_soffice_env()
|
||||
return subprocess.run(["soffice"] + args, env=env, **kwargs)
|
||||
def run_soffice(args: Iterable[str], **kwargs) -> subprocess.CompletedProcess:
|
||||
args = list(args)
|
||||
with contextlib.ExitStack() as stack:
|
||||
if not any(str(a).startswith("-env:UserInstallation") for a in args):
|
||||
profile = stack.enter_context(
|
||||
tempfile.TemporaryDirectory(prefix="lo_profile_", ignore_cleanup_errors=True)
|
||||
)
|
||||
args = [f"-env:UserInstallation={Path(profile).as_uri()}"] + args
|
||||
return subprocess.run(["soffice"] + args, env=get_soffice_env(), **kwargs)
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,132 +0,0 @@
|
||||
"""Unpack Office files (DOCX, PPTX, XLSX) for editing.
|
||||
|
||||
Extracts the ZIP archive, pretty-prints XML files, and optionally:
|
||||
- Merges adjacent runs with identical formatting (DOCX only)
|
||||
- Simplifies adjacent tracked changes from same author (DOCX only)
|
||||
|
||||
Usage:
|
||||
python unpack.py <office_file> <output_dir> [options]
|
||||
|
||||
Examples:
|
||||
python unpack.py document.docx unpacked/
|
||||
python unpack.py presentation.pptx unpacked/
|
||||
python unpack.py document.docx unpacked/ --merge-runs false
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.minidom
|
||||
|
||||
from helpers.merge_runs import merge_runs as do_merge_runs
|
||||
from helpers.simplify_redlines import simplify_redlines as do_simplify_redlines
|
||||
|
||||
SMART_QUOTE_REPLACEMENTS = {
|
||||
"\u201c": "“",
|
||||
"\u201d": "”",
|
||||
"\u2018": "‘",
|
||||
"\u2019": "’",
|
||||
}
|
||||
|
||||
|
||||
def unpack(
|
||||
input_file: str,
|
||||
output_directory: str,
|
||||
merge_runs: bool = True,
|
||||
simplify_redlines: bool = True,
|
||||
) -> tuple[None, str]:
|
||||
input_path = Path(input_file)
|
||||
output_path = Path(output_directory)
|
||||
suffix = input_path.suffix.lower()
|
||||
|
||||
if not input_path.exists():
|
||||
return None, f"Error: {input_file} does not exist"
|
||||
|
||||
if suffix not in {".docx", ".pptx", ".xlsx"}:
|
||||
return None, f"Error: {input_file} must be a .docx, .pptx, or .xlsx file"
|
||||
|
||||
try:
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with zipfile.ZipFile(input_path, "r") as zf:
|
||||
zf.extractall(output_path)
|
||||
|
||||
xml_files = list(output_path.rglob("*.xml")) + list(output_path.rglob("*.rels"))
|
||||
for xml_file in xml_files:
|
||||
_pretty_print_xml(xml_file)
|
||||
|
||||
message = f"Unpacked {input_file} ({len(xml_files)} XML files)"
|
||||
|
||||
if suffix == ".docx":
|
||||
if simplify_redlines:
|
||||
simplify_count, _ = do_simplify_redlines(str(output_path))
|
||||
message += f", simplified {simplify_count} tracked changes"
|
||||
|
||||
if merge_runs:
|
||||
merge_count, _ = do_merge_runs(str(output_path))
|
||||
message += f", merged {merge_count} runs"
|
||||
|
||||
for xml_file in xml_files:
|
||||
_escape_smart_quotes(xml_file)
|
||||
|
||||
return None, message
|
||||
|
||||
except zipfile.BadZipFile:
|
||||
return None, f"Error: {input_file} is not a valid Office file"
|
||||
except Exception as e:
|
||||
return None, f"Error unpacking: {e}"
|
||||
|
||||
|
||||
def _pretty_print_xml(xml_file: Path) -> None:
|
||||
try:
|
||||
content = xml_file.read_text(encoding="utf-8")
|
||||
dom = defusedxml.minidom.parseString(content)
|
||||
xml_file.write_bytes(dom.toprettyxml(indent=" ", encoding="utf-8"))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _escape_smart_quotes(xml_file: Path) -> None:
|
||||
try:
|
||||
content = xml_file.read_text(encoding="utf-8")
|
||||
for char, entity in SMART_QUOTE_REPLACEMENTS.items():
|
||||
content = content.replace(char, entity)
|
||||
xml_file.write_text(content, encoding="utf-8")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Unpack an Office file (DOCX, PPTX, XLSX) for editing"
|
||||
)
|
||||
parser.add_argument("input_file", help="Office file to unpack")
|
||||
parser.add_argument("output_directory", help="Output directory")
|
||||
parser.add_argument(
|
||||
"--merge-runs",
|
||||
type=lambda x: x.lower() == "true",
|
||||
default=True,
|
||||
metavar="true|false",
|
||||
help="Merge adjacent runs with identical formatting (DOCX only, default: true)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--simplify-redlines",
|
||||
type=lambda x: x.lower() == "true",
|
||||
default=True,
|
||||
metavar="true|false",
|
||||
help="Merge adjacent tracked changes from same author (DOCX only, default: true)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
_, message = unpack(
|
||||
args.input_file,
|
||||
args.output_directory,
|
||||
merge_runs=args.merge_runs,
|
||||
simplify_redlines=args.simplify_redlines,
|
||||
)
|
||||
print(message)
|
||||
|
||||
if "Error" in message:
|
||||
sys.exit(1)
|
||||
@@ -6,7 +6,7 @@ Usage:
|
||||
|
||||
The first argument can be either:
|
||||
- An unpacked directory containing the Office document XML files
|
||||
- A packed Office file (.docx/.pptx/.xlsx) which will be unpacked to a temp directory
|
||||
- A packed Office file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx template) which will be unpacked to a temp directory
|
||||
|
||||
Auto-repair fixes:
|
||||
- paraId/durableId values that exceed OOXML limits
|
||||
@@ -19,20 +19,43 @@ import tempfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.ElementTree as ET
|
||||
from defusedxml.common import DefusedXmlException
|
||||
|
||||
from helpers import OOXML_FAMILY, rezip, safe_extract
|
||||
from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator
|
||||
|
||||
WORD_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
||||
|
||||
|
||||
def _fail(message: str):
|
||||
print(f"Error: {message}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def _has_tracked_changes(unpacked_dir: Path) -> bool:
|
||||
document = unpacked_dir / "word" / "document.xml"
|
||||
if not document.is_file():
|
||||
return False
|
||||
try:
|
||||
root = ET.parse(document).getroot()
|
||||
except (ET.ParseError, DefusedXmlException):
|
||||
return False
|
||||
tracked = {f"{{{WORD_NS}}}ins", f"{{{WORD_NS}}}del"}
|
||||
return any(elem.tag in tracked for elem in root.iter())
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Validate Office document XML files")
|
||||
parser.add_argument(
|
||||
"path",
|
||||
help="Path to unpacked directory or packed Office file (.docx/.pptx/.xlsx)",
|
||||
help="Path to unpacked directory or packed Office file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--original",
|
||||
required=False,
|
||||
default=None,
|
||||
help="Path to original file (.docx/.pptx/.xlsx). If omitted, all XSD errors are reported and redlining validation is skipped.",
|
||||
help="Path to original file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx). If omitted, all XSD errors are reported and redlining validation is skipped.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v",
|
||||
@@ -43,63 +66,102 @@ def main():
|
||||
parser.add_argument(
|
||||
"--auto-repair",
|
||||
action="store_true",
|
||||
help="Automatically repair common issues (hex IDs, whitespace preservation)",
|
||||
help="Automatically repair common issues (hex IDs, whitespace preservation). "
|
||||
"Modifies the input in place: repairs to a packed file are written back to it.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--author",
|
||||
default="Claude",
|
||||
help="Author name for redlining validation (default: Claude)",
|
||||
default=None,
|
||||
help="The name you are redlining under. Passing it turns on the "
|
||||
"tracked-change check: any text differing from --original without a "
|
||||
"<w:ins>/<w:del> recording it is reported. Untracked edits carry no "
|
||||
"author, so the check covers them whoever made them — the name marks "
|
||||
"the run as redlining work and is not used to filter. Requires "
|
||||
"--original; docx only.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.author is not None and not args.original:
|
||||
_fail("--author requires --original")
|
||||
|
||||
path = Path(args.path)
|
||||
assert path.exists(), f"Error: {path} does not exist"
|
||||
if not path.exists():
|
||||
_fail(f"{path} does not exist")
|
||||
|
||||
original_file = None
|
||||
if args.original:
|
||||
original_file = Path(args.original)
|
||||
assert original_file.is_file(), f"Error: {original_file} is not a file"
|
||||
assert original_file.suffix.lower() in [".docx", ".pptx", ".xlsx"], (
|
||||
f"Error: {original_file} must be a .docx, .pptx, or .xlsx file"
|
||||
if not original_file.is_file():
|
||||
_fail(f"{original_file} is not a file")
|
||||
if original_file.suffix.lower() not in OOXML_FAMILY:
|
||||
_fail(f"{original_file} must be one of: {', '.join(sorted(OOXML_FAMILY))}")
|
||||
|
||||
family = OOXML_FAMILY.get((original_file or path).suffix.lower())
|
||||
if family is None:
|
||||
_fail(
|
||||
f"Cannot determine file type from {path}. Use --original or provide one of: {', '.join(sorted(OOXML_FAMILY))}."
|
||||
)
|
||||
|
||||
file_extension = (original_file or path).suffix.lower()
|
||||
assert file_extension in [".docx", ".pptx", ".xlsx"], (
|
||||
f"Error: Cannot determine file type from {path}. Use --original or provide a .docx/.pptx/.xlsx file."
|
||||
)
|
||||
if args.author is not None and family != "docx":
|
||||
_fail(f"--author only applies to docx files, not {family}")
|
||||
|
||||
if path.is_file() and path.suffix.lower() in [".docx", ".pptx", ".xlsx"]:
|
||||
temp_dir = tempfile.mkdtemp()
|
||||
with zipfile.ZipFile(path, "r") as zf:
|
||||
zf.extractall(temp_dir)
|
||||
unpacked_dir = Path(temp_dir)
|
||||
packed_file = None
|
||||
temp_dir_ctx = None
|
||||
if path.is_file() and path.suffix.lower() in OOXML_FAMILY:
|
||||
packed_file = path
|
||||
temp_dir_ctx = tempfile.TemporaryDirectory()
|
||||
unpacked_dir = Path(temp_dir_ctx.name)
|
||||
try:
|
||||
with zipfile.ZipFile(path, "r") as zf:
|
||||
safe_extract(zf, unpacked_dir)
|
||||
except (zipfile.BadZipFile, ValueError, OSError) as e:
|
||||
_fail(f"cannot unpack {path}: {e}")
|
||||
else:
|
||||
assert path.is_dir(), f"Error: {path} is not a directory or Office file"
|
||||
if not path.is_dir():
|
||||
_fail(f"{path} is not a directory or Office file")
|
||||
unpacked_dir = path
|
||||
|
||||
match file_extension:
|
||||
case ".docx":
|
||||
match family:
|
||||
case "docx":
|
||||
validators = [
|
||||
DOCXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose),
|
||||
]
|
||||
if original_file:
|
||||
if args.author is not None:
|
||||
validators.append(
|
||||
RedliningValidator(unpacked_dir, original_file, verbose=args.verbose, author=args.author)
|
||||
RedliningValidator(unpacked_dir, original_file, verbose=args.verbose)
|
||||
)
|
||||
case ".pptx":
|
||||
elif original_file and _has_tracked_changes(unpacked_dir):
|
||||
print(
|
||||
"Note: this document has tracked changes; they were not "
|
||||
"checked against the original (pass --author to check)."
|
||||
)
|
||||
case "pptx":
|
||||
validators = [
|
||||
PPTXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose),
|
||||
]
|
||||
case "xlsx":
|
||||
exts = ", ".join(k for k, v in sorted(OOXML_FAMILY.items()) if v == "xlsx")
|
||||
print(
|
||||
f"No XSD schema validation is performed for xlsx-family files ({exts}). "
|
||||
"For formula-error checking, use scripts/recalc.py instead."
|
||||
)
|
||||
sys.exit(0)
|
||||
case _:
|
||||
print(f"Error: Validation not supported for file type {file_extension}")
|
||||
print(f"Error: Validation not supported for file type {family}")
|
||||
sys.exit(1)
|
||||
|
||||
if args.auto_repair:
|
||||
total_repairs = sum(v.repair() for v in validators)
|
||||
if total_repairs:
|
||||
print(f"Auto-repaired {total_repairs} issue(s)")
|
||||
if packed_file is not None:
|
||||
rezip(unpacked_dir, packed_file)
|
||||
print(f"Wrote repaired file to {packed_file}")
|
||||
|
||||
success = all(v.validate() for v in validators)
|
||||
success = all([v.validate() for v in validators])
|
||||
|
||||
if temp_dir_ctx is not None:
|
||||
temp_dir_ctx.cleanup()
|
||||
|
||||
if success:
|
||||
print("All validations PASSED!")
|
||||
|
||||
@@ -6,8 +6,20 @@ import re
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.minidom
|
||||
from functools import lru_cache
|
||||
|
||||
import lxml.etree
|
||||
|
||||
from helpers import safe_extract
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def _load_schema(schema_path: str):
|
||||
with open(schema_path, "rb") as xsd_file:
|
||||
xsd_doc = lxml.etree.parse(
|
||||
xsd_file, parser=lxml.etree.XMLParser(), base_url=schema_path
|
||||
)
|
||||
return lxml.etree.XMLSchema(xsd_doc)
|
||||
|
||||
class BaseSchemaValidator:
|
||||
|
||||
@@ -119,21 +131,28 @@ class BaseSchemaValidator:
|
||||
try:
|
||||
content = xml_file.read_text(encoding="utf-8")
|
||||
dom = defusedxml.minidom.parseString(content)
|
||||
modified = False
|
||||
pending = []
|
||||
|
||||
for elem in dom.getElementsByTagName("*"):
|
||||
if elem.tagName.endswith(":t") and elem.firstChild:
|
||||
text = elem.firstChild.nodeValue
|
||||
if text and (text.startswith((' ', '\t')) or text.endswith((' ', '\t'))):
|
||||
local_name = elem.tagName.rsplit(":", 1)[-1]
|
||||
if local_name in ("t", "delText", "instrText", "delInstrText"):
|
||||
text = "".join(
|
||||
child.data
|
||||
for child in elem.childNodes
|
||||
if child.nodeType in (child.TEXT_NODE, child.CDATA_SECTION_NODE)
|
||||
)
|
||||
ws = (" ", "\t", "\n", "\r")
|
||||
if text and (text.startswith(ws) or text.endswith(ws)):
|
||||
if elem.getAttribute("xml:space") != "preserve":
|
||||
elem.setAttribute("xml:space", "preserve")
|
||||
text_preview = repr(text[:30]) + "..." if len(text) > 30 else repr(text)
|
||||
print(f" Repaired: {xml_file.name}: Added xml:space='preserve' to {elem.tagName}: {text_preview}")
|
||||
repairs += 1
|
||||
modified = True
|
||||
pending.append(f" Repaired: {xml_file.name}: Added xml:space='preserve' to {elem.tagName}: {text_preview}")
|
||||
|
||||
if modified:
|
||||
if pending:
|
||||
xml_file.write_bytes(dom.toxml(encoding="UTF-8"))
|
||||
for message in pending:
|
||||
print(message)
|
||||
repairs += len(pending)
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
@@ -212,6 +231,8 @@ class BaseSchemaValidator:
|
||||
elem.getparent().remove(elem)
|
||||
|
||||
for elem in root.iter():
|
||||
if not hasattr(elem, "tag") or callable(elem.tag):
|
||||
continue
|
||||
tag = (
|
||||
elem.tag.split("}")[-1].lower()
|
||||
if "}" in elem.tag
|
||||
@@ -326,6 +347,8 @@ class BaseSchemaValidator:
|
||||
namespaces={"ns": self.PACKAGE_RELATIONSHIPS_NAMESPACE},
|
||||
):
|
||||
target = rel.get("Target")
|
||||
if rel.get("TargetMode") == "External":
|
||||
continue
|
||||
if target and not target.startswith(
|
||||
("http", "mailto:")
|
||||
):
|
||||
@@ -423,6 +446,8 @@ class BaseSchemaValidator:
|
||||
r_ns = self.OFFICE_RELATIONSHIPS_NAMESPACE
|
||||
rid_attrs_to_check = ["id", "embed", "link"]
|
||||
for elem in xml_root.iter():
|
||||
if not hasattr(elem, "tag") or callable(elem.tag):
|
||||
continue
|
||||
for attr_name in rid_attrs_to_check:
|
||||
rid_attr = elem.get(f"{{{r_ns}}}{attr_name}")
|
||||
if not rid_attr:
|
||||
@@ -747,18 +772,16 @@ class BaseSchemaValidator:
|
||||
|
||||
return xml_doc
|
||||
|
||||
def _validate_single_file_xsd(self, xml_file, base_path):
|
||||
schema_path = self._get_schema_path(xml_file)
|
||||
def _preprocess_for_schema(self, xml_doc, relative_path):
|
||||
return xml_doc
|
||||
|
||||
def _validate_single_file_xsd(self, xml_file, base_path, schema_path=None):
|
||||
schema_path = schema_path or self._get_schema_path(xml_file)
|
||||
if not schema_path:
|
||||
return None, None
|
||||
|
||||
try:
|
||||
with open(schema_path, "rb") as xsd_file:
|
||||
parser = lxml.etree.XMLParser()
|
||||
xsd_doc = lxml.etree.parse(
|
||||
xsd_file, parser=parser, base_url=str(schema_path)
|
||||
)
|
||||
schema = lxml.etree.XMLSchema(xsd_doc)
|
||||
schema = _load_schema(str(schema_path))
|
||||
|
||||
with open(xml_file, "r") as f:
|
||||
xml_doc = lxml.etree.parse(f)
|
||||
@@ -773,6 +796,8 @@ class BaseSchemaValidator:
|
||||
):
|
||||
xml_doc = self._clean_ignorable_namespaces(xml_doc)
|
||||
|
||||
xml_doc = self._preprocess_for_schema(xml_doc, relative_path)
|
||||
|
||||
if schema.validate(xml_doc):
|
||||
return True, set()
|
||||
else:
|
||||
@@ -784,7 +809,7 @@ class BaseSchemaValidator:
|
||||
except Exception as e:
|
||||
return False, {str(e)}
|
||||
|
||||
def _get_original_file_errors(self, xml_file):
|
||||
def _get_original_file_errors(self, xml_file, schema_path=None):
|
||||
if self.original_file is None:
|
||||
return set()
|
||||
|
||||
@@ -798,8 +823,11 @@ class BaseSchemaValidator:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
with zipfile.ZipFile(self.original_file, "r") as zip_ref:
|
||||
zip_ref.extractall(temp_path)
|
||||
try:
|
||||
with zipfile.ZipFile(self.original_file, "r") as zip_ref:
|
||||
safe_extract(zip_ref, temp_path)
|
||||
except (zipfile.BadZipFile, ValueError, OSError):
|
||||
return set()
|
||||
|
||||
original_xml_file = temp_path / relative_path
|
||||
|
||||
@@ -807,7 +835,7 @@ class BaseSchemaValidator:
|
||||
return set()
|
||||
|
||||
is_valid, errors = self._validate_single_file_xsd(
|
||||
original_xml_file, temp_path
|
||||
original_xml_file, temp_path, schema_path=schema_path
|
||||
)
|
||||
return errors if errors else set()
|
||||
|
||||
|
||||
@@ -6,10 +6,13 @@ import random
|
||||
import re
|
||||
import tempfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.minidom
|
||||
import lxml.etree
|
||||
|
||||
from helpers import safe_extract
|
||||
|
||||
from .base import BaseSchemaValidator
|
||||
|
||||
|
||||
@@ -186,7 +189,7 @@ class DOCXSchemaValidator(BaseSchemaValidator):
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with zipfile.ZipFile(original, "r") as zip_ref:
|
||||
zip_ref.extractall(temp_dir)
|
||||
safe_extract(zip_ref, Path(temp_dir))
|
||||
|
||||
doc_xml_path = temp_dir + "/word/document.xml"
|
||||
root = lxml.etree.parse(doc_xml_path).getroot()
|
||||
@@ -241,9 +244,12 @@ class DOCXSchemaValidator(BaseSchemaValidator):
|
||||
return True
|
||||
|
||||
def compare_paragraph_counts(self):
|
||||
original_count = self.count_paragraphs_in_original()
|
||||
new_count = self.count_paragraphs_in_unpacked()
|
||||
if self.original_file is None:
|
||||
print(f"\nParagraphs: {new_count}")
|
||||
return
|
||||
|
||||
original_count = self.count_paragraphs_in_original()
|
||||
diff = new_count - original_count
|
||||
diff_str = f"+{diff}" if diff > 0 else str(diff)
|
||||
print(f"\nParagraphs: {original_count} → {new_count} ({diff_str})")
|
||||
@@ -260,9 +266,15 @@ class DOCXSchemaValidator(BaseSchemaValidator):
|
||||
try:
|
||||
for elem in lxml.etree.parse(str(xml_file)).iter():
|
||||
if val := elem.get(para_id_attr):
|
||||
if self._parse_id_value(val, base=16) >= 0x80000000:
|
||||
try:
|
||||
if self._parse_id_value(val, base=16) >= 0x80000000:
|
||||
errors.append(
|
||||
f" {xml_file.name}:{elem.sourceline}: paraId={val} >= 0x80000000"
|
||||
)
|
||||
except ValueError:
|
||||
errors.append(
|
||||
f" {xml_file.name}:{elem.sourceline}: paraId={val} >= 0x80000000"
|
||||
f" {xml_file.name}:{elem.sourceline}: "
|
||||
f"paraId={val} is not valid hex"
|
||||
)
|
||||
|
||||
if val := elem.get(durable_id_attr):
|
||||
@@ -279,13 +291,19 @@ class DOCXSchemaValidator(BaseSchemaValidator):
|
||||
f"durableId={val} must be decimal in numbering.xml"
|
||||
)
|
||||
else:
|
||||
if self._parse_id_value(val, base=16) >= 0x7FFFFFFF:
|
||||
try:
|
||||
if self._parse_id_value(val, base=16) >= 0x7FFFFFFF:
|
||||
errors.append(
|
||||
f" {xml_file.name}:{elem.sourceline}: "
|
||||
f"durableId={val} >= 0x7FFFFFFF"
|
||||
)
|
||||
except ValueError:
|
||||
errors.append(
|
||||
f" {xml_file.name}:{elem.sourceline}: "
|
||||
f"durableId={val} >= 0x7FFFFFFF"
|
||||
f"durableId={val} is not valid hex"
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
except lxml.etree.XMLSyntaxError:
|
||||
continue
|
||||
|
||||
if errors:
|
||||
print(f"FAILED - {len(errors)} ID constraint violations:")
|
||||
@@ -389,52 +407,54 @@ class DOCXSchemaValidator(BaseSchemaValidator):
|
||||
return repairs
|
||||
|
||||
def repair_durableId(self) -> int:
|
||||
DURABLE_ID_ATTRS = ("w16cid:durableId", "w16cex:durableId")
|
||||
repairs = 0
|
||||
renames: dict = {}
|
||||
|
||||
for xml_file in self.xml_files:
|
||||
try:
|
||||
content = xml_file.read_text(encoding="utf-8")
|
||||
dom = defusedxml.minidom.parseString(content)
|
||||
is_numbering = xml_file.name == "numbering.xml"
|
||||
base = 10 if is_numbering else 16
|
||||
pending = []
|
||||
seen_in_file = set()
|
||||
modified = False
|
||||
|
||||
for elem in dom.getElementsByTagName("*"):
|
||||
if not elem.hasAttribute("w16cid:durableId"):
|
||||
continue
|
||||
for attr_name in DURABLE_ID_ATTRS:
|
||||
if not elem.hasAttribute(attr_name):
|
||||
continue
|
||||
|
||||
durable_id = elem.getAttribute("w16cid:durableId")
|
||||
needs_repair = False
|
||||
|
||||
if xml_file.name == "numbering.xml":
|
||||
durable_id = elem.getAttribute(attr_name)
|
||||
try:
|
||||
needs_repair = (
|
||||
self._parse_id_value(durable_id, base=10) >= 0x7FFFFFFF
|
||||
)
|
||||
except ValueError:
|
||||
needs_repair = True
|
||||
else:
|
||||
try:
|
||||
needs_repair = (
|
||||
self._parse_id_value(durable_id, base=16) >= 0x7FFFFFFF
|
||||
)
|
||||
key = self._parse_id_value(durable_id, base=base)
|
||||
needs_repair = key >= 0x7FFFFFFF
|
||||
except ValueError:
|
||||
key = durable_id
|
||||
needs_repair = True
|
||||
|
||||
if needs_repair:
|
||||
value = random.randint(1, 0x7FFFFFFE)
|
||||
if xml_file.name == "numbering.xml":
|
||||
new_id = str(value)
|
||||
else:
|
||||
new_id = f"{value:08X}"
|
||||
if needs_repair:
|
||||
if key in seen_in_file:
|
||||
value = random.randint(1, 0x7FFFFFFE)
|
||||
else:
|
||||
seen_in_file.add(key)
|
||||
if key not in renames:
|
||||
renames[key] = random.randint(1, 0x7FFFFFFE)
|
||||
value = renames[key]
|
||||
new_id = str(value) if is_numbering else f"{value:08X}"
|
||||
|
||||
elem.setAttribute("w16cid:durableId", new_id)
|
||||
print(
|
||||
f" Repaired: {xml_file.name}: durableId {durable_id} → {new_id}"
|
||||
)
|
||||
repairs += 1
|
||||
modified = True
|
||||
elem.setAttribute(attr_name, new_id)
|
||||
pending.append(
|
||||
f" Repaired: {xml_file.name}: durableId {durable_id} → {new_id}"
|
||||
)
|
||||
modified = True
|
||||
|
||||
if modified:
|
||||
xml_file.write_bytes(dom.toxml(encoding="UTF-8"))
|
||||
for message in pending:
|
||||
print(message)
|
||||
repairs += len(pending)
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -3,6 +3,9 @@ Validator for PowerPoint presentation XML files against XSD schemas.
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from helpers import opc_target, rels_source_part, safe_extract
|
||||
|
||||
from .base import BaseSchemaValidator
|
||||
|
||||
@@ -57,8 +60,171 @@ class PPTXSchemaValidator(BaseSchemaValidator):
|
||||
if not self.validate_no_duplicate_slide_layouts():
|
||||
all_valid = False
|
||||
|
||||
if not self.validate_master_theme_uniqueness():
|
||||
all_valid = False
|
||||
|
||||
if not self.validate_charts():
|
||||
all_valid = False
|
||||
|
||||
if not self.validate_slides():
|
||||
all_valid = False
|
||||
|
||||
return all_valid
|
||||
|
||||
def _package_map(self) -> dict:
|
||||
wanted = []
|
||||
wanted += list(self.unpacked_dir.glob("[[]Content_Types[]].xml"))
|
||||
wanted += list(self.unpacked_dir.glob("ppt/presentation.xml"))
|
||||
wanted += list(self.unpacked_dir.glob("ppt/theme/*.xml"))
|
||||
wanted += list(self.unpacked_dir.glob("ppt/theme/_rels/*.rels"))
|
||||
wanted += list(self.unpacked_dir.glob("ppt/charts/chart*.xml"))
|
||||
for group in ("slideMasters", "notesMasters", "handoutMasters"):
|
||||
wanted += list(self.unpacked_dir.glob(f"ppt/{group}/*.xml"))
|
||||
wanted += list(self.unpacked_dir.glob(f"ppt/{group}/_rels/*.rels"))
|
||||
return {
|
||||
p.relative_to(self.unpacked_dir).as_posix(): p.read_bytes()
|
||||
for p in wanted
|
||||
if p.is_file()
|
||||
}
|
||||
|
||||
def validate_master_theme_uniqueness(self):
|
||||
from helpers.pptx_theme import _NOTES_MASTERS, live_shared_master_themes
|
||||
|
||||
shared = live_shared_master_themes(self._package_map())
|
||||
if shared:
|
||||
print(f"FAILED - Found {len(shared)} master(s) sharing a theme part:")
|
||||
for message in shared:
|
||||
print(f" {message}")
|
||||
if any(m.startswith(_NOTES_MASTERS) for m in shared):
|
||||
print(" Fix: in ppt/presentation.xml, move <p:notesMasterIdLst> back to "
|
||||
"directly after <p:sldIdLst>. PowerPoint reads that happily.")
|
||||
else:
|
||||
print(" Fix: give each master its own theme part.")
|
||||
return False
|
||||
|
||||
if self.verbose:
|
||||
print("PASSED - No master shares a theme part in a way PowerPoint refuses")
|
||||
return True
|
||||
|
||||
def validate_charts(self):
|
||||
from helpers.pptx_chart import find_chart_problems
|
||||
|
||||
problems = find_chart_problems(self._package_map())
|
||||
if problems:
|
||||
print(f"FAILED - Found {len(problems)} chart problem(s) PowerPoint rejects:")
|
||||
for message in problems:
|
||||
print(f" {message}")
|
||||
return False
|
||||
|
||||
if self.verbose:
|
||||
print("PASSED - Charts satisfy the constraints PowerPoint enforces")
|
||||
return True
|
||||
|
||||
def _original_slide_defects(self, schema) -> set[str]:
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
from helpers.pptx_slide import SLIDE_PART_RE, fatal_slide_errors
|
||||
|
||||
if self.original_file is None:
|
||||
return set()
|
||||
|
||||
found: set[str] = set()
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
try:
|
||||
with zipfile.ZipFile(self.original_file, "r") as zf:
|
||||
safe_extract(zf, temp_path)
|
||||
except (zipfile.BadZipFile, ValueError, OSError):
|
||||
return set()
|
||||
|
||||
for part in sorted(temp_path.rglob("*.xml")):
|
||||
relative = part.relative_to(temp_path).as_posix()
|
||||
if not SLIDE_PART_RE.fullmatch(relative):
|
||||
continue
|
||||
ok, errors = self._validate_single_file_xsd(
|
||||
part.resolve(), temp_path.resolve(), schema_path=schema
|
||||
)
|
||||
if ok is None or ok or not errors:
|
||||
continue
|
||||
found |= set(fatal_slide_errors(set(errors)))
|
||||
return found
|
||||
|
||||
def validate_slides(self):
|
||||
from helpers.pptx_slide import (
|
||||
SLIDE_PART_RE,
|
||||
fatal_slide_errors,
|
||||
is_schema_verdict,
|
||||
)
|
||||
|
||||
schema = self.schemas_dir / self.SCHEMA_MAPPINGS["ppt"]
|
||||
inherited = self._original_slide_defects(schema)
|
||||
problems: list[str] = []
|
||||
broken: list[str] = []
|
||||
|
||||
for xml_file in self.xml_files:
|
||||
relative = xml_file.relative_to(self.unpacked_dir).as_posix()
|
||||
if not SLIDE_PART_RE.fullmatch(relative):
|
||||
continue
|
||||
ok, errors = self._validate_single_file_xsd(
|
||||
xml_file.resolve(), self.unpacked_dir.resolve(), schema_path=schema
|
||||
)
|
||||
if ok is None or not errors:
|
||||
continue
|
||||
|
||||
unreadable = [f"{relative}: {e}" for e in errors if not is_schema_verdict(e)]
|
||||
if unreadable:
|
||||
broken.extend(unreadable)
|
||||
continue
|
||||
if ok:
|
||||
continue
|
||||
|
||||
for message in fatal_slide_errors(set(errors)):
|
||||
if message in inherited:
|
||||
continue
|
||||
problems.append(f"{relative}: {message}")
|
||||
|
||||
if broken:
|
||||
print(f"FAILED - Could not check {len(broken)} slide part(s):")
|
||||
for message in sorted(broken):
|
||||
print(f" {message[:240]}")
|
||||
|
||||
if problems:
|
||||
print(f"FAILED - Found {len(problems)} slide problem(s) PowerPoint rejects:")
|
||||
for message in sorted(problems):
|
||||
print(f" {message[:240]}")
|
||||
|
||||
if broken or problems:
|
||||
return False
|
||||
|
||||
if self.verbose:
|
||||
print("PASSED - Slide XML has none of the defects PowerPoint refuses")
|
||||
return True
|
||||
|
||||
def _get_schema_path(self, xml_file):
|
||||
if xml_file.parent.name == "charts" and xml_file.name.startswith("chart"):
|
||||
return None
|
||||
return super()._get_schema_path(xml_file)
|
||||
|
||||
def _preprocess_for_schema(self, xml_doc, relative_path):
|
||||
if relative_path.as_posix() != "ppt/presentation.xml":
|
||||
return xml_doc
|
||||
|
||||
root = xml_doc.getroot()
|
||||
ns = f"{{{self.PRESENTATIONML_NAMESPACE}}}"
|
||||
notes = root.find(f"{ns}notesMasterIdLst")
|
||||
slides = root.find(f"{ns}sldIdLst")
|
||||
if notes is None or slides is None:
|
||||
return xml_doc
|
||||
|
||||
children = list(root)
|
||||
if children.index(notes) < children.index(slides):
|
||||
return xml_doc
|
||||
|
||||
root.remove(notes)
|
||||
root.insert(list(root).index(slides), notes)
|
||||
return xml_doc
|
||||
|
||||
def validate_uuid_ids(self):
|
||||
import lxml.etree
|
||||
|
||||
@@ -229,17 +395,17 @@ class PPTXSchemaValidator(BaseSchemaValidator):
|
||||
):
|
||||
rel_type = rel.get("Type", "")
|
||||
if "notesSlide" in rel_type:
|
||||
target = rel.get("Target", "")
|
||||
if target:
|
||||
normalized_target = target.replace("../", "")
|
||||
|
||||
part = opc_target(
|
||||
rel.get("Target", ""),
|
||||
rels_source_part(rels_file, self.unpacked_dir),
|
||||
rel.get("TargetMode", ""),
|
||||
)
|
||||
if part:
|
||||
slide_name = rels_file.stem.replace(
|
||||
".xml", ""
|
||||
)
|
||||
|
||||
if normalized_target not in notes_slide_references:
|
||||
notes_slide_references[normalized_target] = []
|
||||
notes_slide_references[normalized_target].append(
|
||||
notes_slide_references.setdefault(part, []).append(
|
||||
(slide_name, rels_file)
|
||||
)
|
||||
|
||||
|
||||
@@ -1,5 +1,14 @@
|
||||
"""
|
||||
Validator for tracked changes in Word documents.
|
||||
|
||||
Detects untracked edits in word/document.xml: text that differs from the
|
||||
original without a <w:ins>/<w:del> wrapper recording it. The tracked changes
|
||||
that are new relative to the original are undone, and the result is compared
|
||||
against the original; whatever text still differs was edited without being
|
||||
tracked.
|
||||
|
||||
Only the document body is compared. Headers, footers, footnotes and endnotes
|
||||
are separate parts and are not checked.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
@@ -7,14 +16,18 @@ import tempfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import defusedxml.ElementTree as ET
|
||||
from defusedxml.common import DefusedXmlException
|
||||
|
||||
from helpers import rendered_text, safe_extract
|
||||
|
||||
|
||||
class RedliningValidator:
|
||||
|
||||
def __init__(self, unpacked_dir, original_docx, verbose=False, author="Claude"):
|
||||
def __init__(self, unpacked_dir, original_docx, verbose=False):
|
||||
self.unpacked_dir = Path(unpacked_dir)
|
||||
self.original_docx = Path(original_docx)
|
||||
self.verbose = verbose
|
||||
self.author = author
|
||||
self.namespaces = {
|
||||
"w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
||||
}
|
||||
@@ -28,40 +41,12 @@ class RedliningValidator:
|
||||
print(f"FAILED - Modified document.xml not found at {modified_file}")
|
||||
return False
|
||||
|
||||
try:
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
tree = ET.parse(modified_file)
|
||||
root = tree.getroot()
|
||||
|
||||
del_elements = root.findall(".//w:del", self.namespaces)
|
||||
ins_elements = root.findall(".//w:ins", self.namespaces)
|
||||
|
||||
author_del_elements = [
|
||||
elem
|
||||
for elem in del_elements
|
||||
if elem.get(f"{{{self.namespaces['w']}}}author") == self.author
|
||||
]
|
||||
author_ins_elements = [
|
||||
elem
|
||||
for elem in ins_elements
|
||||
if elem.get(f"{{{self.namespaces['w']}}}author") == self.author
|
||||
]
|
||||
|
||||
if not author_del_elements and not author_ins_elements:
|
||||
if self.verbose:
|
||||
print(f"PASSED - No tracked changes by {self.author} found.")
|
||||
return True
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
try:
|
||||
with zipfile.ZipFile(self.original_docx, "r") as zip_ref:
|
||||
zip_ref.extractall(temp_path)
|
||||
safe_extract(zip_ref, temp_path)
|
||||
except Exception as e:
|
||||
print(f"FAILED - Error unpacking original docx: {e}")
|
||||
return False
|
||||
@@ -74,18 +59,16 @@ class RedliningValidator:
|
||||
return False
|
||||
|
||||
try:
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
modified_tree = ET.parse(modified_file)
|
||||
modified_root = modified_tree.getroot()
|
||||
original_tree = ET.parse(original_file)
|
||||
original_root = original_tree.getroot()
|
||||
except ET.ParseError as e:
|
||||
except (ET.ParseError, DefusedXmlException) as e:
|
||||
print(f"FAILED - Error parsing XML files: {e}")
|
||||
return False
|
||||
|
||||
self._remove_author_tracked_changes(original_root)
|
||||
self._remove_author_tracked_changes(modified_root)
|
||||
new_changes = self._new_tracked_changes(original_root, modified_root)
|
||||
self._remove_tracked_changes(modified_root, new_changes)
|
||||
|
||||
modified_text = self._extract_text_content(modified_root)
|
||||
original_text = self._extract_text_content(original_root)
|
||||
@@ -98,20 +81,91 @@ class RedliningValidator:
|
||||
return False
|
||||
|
||||
if self.verbose:
|
||||
print(f"PASSED - All changes by {self.author} are properly tracked")
|
||||
print(
|
||||
f"PASSED - All {len(new_changes)} change(s) against the original "
|
||||
"are properly tracked"
|
||||
)
|
||||
return True
|
||||
|
||||
def _tracked_change_elements(self, root):
|
||||
ins_tag = f"{{{self.namespaces['w']}}}ins"
|
||||
del_tag = f"{{{self.namespaces['w']}}}del"
|
||||
return [elem for elem in root.iter() if elem.tag in (ins_tag, del_tag)]
|
||||
|
||||
def _rendered_text(self, elem):
|
||||
preserve = elem.get("{http://www.w3.org/XML/1998/namespace}space") == "preserve"
|
||||
return rendered_text(elem.text or "", preserve)
|
||||
|
||||
def _text_elements(self, elem):
|
||||
w = self.namespaces["w"]
|
||||
return [
|
||||
node
|
||||
for node in elem.iter()
|
||||
if node.tag in (f"{{{w}}}t", f"{{{w}}}delText")
|
||||
]
|
||||
|
||||
def _tracked_change_key(self, elem):
|
||||
w = self.namespaces["w"]
|
||||
text = "".join(self._rendered_text(node) for node in self._text_elements(elem))
|
||||
return (elem.tag, elem.get(f"{{{w}}}author"), elem.get(f"{{{w}}}date"), text)
|
||||
|
||||
def _new_tracked_changes(self, original_root, modified_root):
|
||||
original = self._tracked_change_elements(original_root)
|
||||
modified = self._tracked_change_elements(modified_root)
|
||||
|
||||
pool = {}
|
||||
for elem in original:
|
||||
pool.setdefault(self._tracked_change_key(elem), []).append(elem)
|
||||
|
||||
matched, leftover = set(), []
|
||||
for elem in modified:
|
||||
bucket = pool.get(self._tracked_change_key(elem))
|
||||
if bucket:
|
||||
matched.add(bucket.pop())
|
||||
else:
|
||||
leftover.append(elem)
|
||||
|
||||
def group(elem):
|
||||
return self._tracked_change_key(elem)[:3]
|
||||
|
||||
def text_of(elems):
|
||||
return "".join(self._tracked_change_key(e)[3] for e in elems)
|
||||
|
||||
unmatched_original = {}
|
||||
for elem in original:
|
||||
if elem not in matched:
|
||||
unmatched_original.setdefault(group(elem), []).append(elem)
|
||||
|
||||
by_group = {}
|
||||
for elem in leftover:
|
||||
by_group.setdefault(group(elem), []).append(elem)
|
||||
|
||||
new = set()
|
||||
for key, elems in by_group.items():
|
||||
rebuilt = text_of(elems)
|
||||
if rebuilt and rebuilt == text_of(unmatched_original.get(key, [])):
|
||||
continue
|
||||
new.update(elems)
|
||||
return new
|
||||
|
||||
def _generate_detailed_diff(self, original_text, modified_text):
|
||||
error_parts = [
|
||||
f"FAILED - Document text doesn't match after removing {self.author}'s tracked changes",
|
||||
"FAILED - Document text doesn't match after removing the tracked changes",
|
||||
"",
|
||||
"Likely causes:",
|
||||
" 1. Modified text inside another author's <w:ins> or <w:del> tags",
|
||||
" 2. Made edits without proper tracked changes",
|
||||
" 3. Didn't nest <w:del> inside <w:ins> when deleting another's insertion",
|
||||
" 4. Rewrote another author's <w:ins>/<w:del> and changed its text on",
|
||||
" the way. A tracked change from the original is recognised by its",
|
||||
" author, date and text; anything that doesn't reproduce one exactly",
|
||||
" reads as new, and the text it carried is reported missing.",
|
||||
"",
|
||||
"For pre-redlined documents, use correct patterns:",
|
||||
" - To reject another's INSERTION: Nest <w:del> inside their <w:ins>",
|
||||
" - To reject PART of one: nest <w:del> around only the runs you reject.",
|
||||
" Their <w:ins> may be split around it, so long as the pieces keep",
|
||||
" their author and date and still spell out the same text.",
|
||||
" - To restore another's DELETION: Add new <w:ins> AFTER their <w:del>",
|
||||
"",
|
||||
]
|
||||
@@ -195,15 +249,14 @@ class RedliningValidator:
|
||||
|
||||
return None
|
||||
|
||||
def _remove_author_tracked_changes(self, root):
|
||||
def _remove_tracked_changes(self, root, targets):
|
||||
ins_tag = f"{{{self.namespaces['w']}}}ins"
|
||||
del_tag = f"{{{self.namespaces['w']}}}del"
|
||||
author_attr = f"{{{self.namespaces['w']}}}author"
|
||||
|
||||
for parent in root.iter():
|
||||
to_remove = []
|
||||
for child in parent:
|
||||
if child.tag == ins_tag and child.get(author_attr) == self.author:
|
||||
if child.tag == ins_tag and child in targets:
|
||||
to_remove.append(child)
|
||||
for elem in to_remove:
|
||||
parent.remove(elem)
|
||||
@@ -214,7 +267,7 @@ class RedliningValidator:
|
||||
for parent in root.iter():
|
||||
to_process = []
|
||||
for child in parent:
|
||||
if child.tag == del_tag and child.get(author_attr) == self.author:
|
||||
if child.tag == del_tag and child in targets:
|
||||
to_process.append((child, list(parent).index(child)))
|
||||
|
||||
for del_elem, del_index in reversed(to_process):
|
||||
@@ -234,8 +287,7 @@ class RedliningValidator:
|
||||
for p_elem in root.findall(f".//{p_tag}"):
|
||||
text_parts = []
|
||||
for t_elem in p_elem.findall(f".//{t_tag}"):
|
||||
if t_elem.text:
|
||||
text_parts.append(t_elem.text)
|
||||
text_parts.append(self._rendered_text(t_elem))
|
||||
paragraph_text = "".join(text_parts)
|
||||
if paragraph_text:
|
||||
paragraphs.append(paragraph_text)
|
||||
|
||||
+168
-44
@@ -3,20 +3,29 @@ Excel Formula Recalculation Script
|
||||
Recalculates all formulas in an Excel file using LibreOffice
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from office.soffice import get_soffice_env
|
||||
from office.soffice import get_soffice_env, run_soffice
|
||||
|
||||
from openpyxl import load_workbook
|
||||
|
||||
MACRO_DIR_MACOS = "~/Library/Application Support/LibreOffice/4/user/basic/Standard"
|
||||
MACRO_DIR_LINUX = "~/.config/libreoffice/4/user/basic/Standard"
|
||||
MACRO_FILENAME = "Module1.xba"
|
||||
SOFFICE_MISSING = "soffice not found on PATH; LibreOffice is required to recalculate"
|
||||
|
||||
MAX_LOCATIONS = 100
|
||||
|
||||
EXTERNAL_REF_RE = re.compile(r"""(?<![\w"\[])'?\[\d+\][^!"\[\]]*'?!""")
|
||||
|
||||
RECALCULATE_MACRO = """<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE script:module PUBLIC "-//OpenOffice.org//DTD OfficeDocument 1.0//EN" "module.dtd">
|
||||
@@ -39,63 +48,168 @@ def has_gtimeout():
|
||||
return False
|
||||
|
||||
|
||||
def setup_libreoffice_macro():
|
||||
macro_dir = os.path.expanduser(
|
||||
MACRO_DIR_MACOS if platform.system() == "Darwin" else MACRO_DIR_LINUX
|
||||
)
|
||||
macro_file = os.path.join(macro_dir, MACRO_FILENAME)
|
||||
def _stamp(path):
|
||||
st = os.stat(path)
|
||||
return st.st_mtime_ns, st.st_size
|
||||
|
||||
if (
|
||||
os.path.exists(macro_file)
|
||||
and "RecalculateAndSave" in Path(macro_file).read_text()
|
||||
):
|
||||
return True
|
||||
|
||||
if not os.path.exists(macro_dir):
|
||||
subprocess.run(
|
||||
["soffice", "--headless", "--terminate_after_init"],
|
||||
def setup_libreoffice_macro(profile_dir: Path, timeout=30):
|
||||
url = profile_dir.as_uri()
|
||||
try:
|
||||
run_soffice(
|
||||
["--headless", "--terminate_after_init", f"-env:UserInstallation={url}"],
|
||||
capture_output=True,
|
||||
timeout=10,
|
||||
env=get_soffice_env(),
|
||||
timeout=timeout,
|
||||
)
|
||||
os.makedirs(macro_dir, exist_ok=True)
|
||||
except FileNotFoundError:
|
||||
return None, SOFFICE_MISSING
|
||||
except subprocess.TimeoutExpired:
|
||||
return None, "LibreOffice timed out creating its profile; formulas were NOT recalculated"
|
||||
|
||||
macro_dir = profile_dir / "user" / "basic" / "Standard"
|
||||
if not macro_dir.exists():
|
||||
return None, "LibreOffice did not create a usable profile; formulas were NOT recalculated"
|
||||
|
||||
try:
|
||||
Path(macro_file).write_text(RECALCULATE_MACRO)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
(macro_dir / MACRO_FILENAME).write_text(RECALCULATE_MACRO)
|
||||
except OSError as e:
|
||||
return None, f"Could not install the recalculation macro: {e}"
|
||||
|
||||
return url, None
|
||||
|
||||
|
||||
def recalc(filename, timeout=30):
|
||||
def external_links_at_risk(filename):
|
||||
try:
|
||||
with zipfile.ZipFile(filename) as archive:
|
||||
names = archive.namelist()
|
||||
except (zipfile.BadZipFile, OSError):
|
||||
return []
|
||||
if not any(n.startswith("xl/externalLinks/") for n in names):
|
||||
return []
|
||||
|
||||
with contextlib.ExitStack() as stack:
|
||||
formulas = load_workbook(filename, data_only=False)
|
||||
stack.callback(formulas.close)
|
||||
values = load_workbook(filename, data_only=True)
|
||||
stack.callback(values.close)
|
||||
|
||||
external_names = [
|
||||
name
|
||||
for name, dn in formulas.defined_names.items()
|
||||
if isinstance(getattr(dn, "value", None), str) and EXTERNAL_REF_RE.search(dn.value)
|
||||
]
|
||||
name_re = (
|
||||
re.compile(r"\b(" + "|".join(re.escape(n) for n in external_names) + r")\b")
|
||||
if external_names
|
||||
else None
|
||||
)
|
||||
|
||||
at_risk = []
|
||||
for sheet in formulas.sheetnames:
|
||||
ws = formulas[sheet]
|
||||
if not hasattr(ws, "iter_rows"):
|
||||
continue
|
||||
cached = values[sheet]
|
||||
for row in ws.iter_rows():
|
||||
for cell in row:
|
||||
v = cell.value
|
||||
if not (isinstance(v, str) and v.startswith("=")):
|
||||
continue
|
||||
reaches_out = EXTERNAL_REF_RE.search(v) or (name_re and name_re.search(v))
|
||||
if reaches_out and cached[cell.coordinate].value is None:
|
||||
at_risk.append(f"{sheet}!{cell.coordinate}")
|
||||
return at_risk
|
||||
|
||||
|
||||
def recalc(filename, timeout=30, force=False):
|
||||
if not Path(filename).exists():
|
||||
return {"error": f"File {filename} does not exist"}
|
||||
|
||||
abs_path = str(Path(filename).absolute())
|
||||
|
||||
if not setup_libreoffice_macro():
|
||||
return {"error": "Failed to setup LibreOffice macro"}
|
||||
if not os.access(abs_path, os.W_OK):
|
||||
return {"error": f"{filename} is not writable; recalculation rewrites the file in place"}
|
||||
|
||||
try:
|
||||
get_soffice_env()
|
||||
except Exception as e:
|
||||
return {"error": f"Could not prepare the LibreOffice environment: {e}"}
|
||||
|
||||
if not force:
|
||||
try:
|
||||
at_risk = external_links_at_risk(filename)
|
||||
except Exception as e:
|
||||
return {"error": f"Could not inspect {filename} for external links: {e}"}
|
||||
if at_risk:
|
||||
shown = at_risk[:MAX_LOCATIONS]
|
||||
return {
|
||||
"error": (
|
||||
"Refusing to recalculate: this workbook links to another workbook, and "
|
||||
f"{len(at_risk)} linked cell(s) have lost their cached value (openpyxl strips "
|
||||
"these on save). Recalculating would resolve them to #NAME? and delete the "
|
||||
"external links for good. Copy those cells' values from the original file "
|
||||
"before saving, or pass --force to accept the loss. Charts and conditional "
|
||||
"formats can hold external references too, so this list may not be exhaustive."
|
||||
),
|
||||
"external_link_cells": shown,
|
||||
"external_link_cells_truncated": max(0, len(at_risk) - len(shown)),
|
||||
}
|
||||
|
||||
with tempfile.TemporaryDirectory(
|
||||
prefix="recalc-lo-profile-", ignore_cleanup_errors=True
|
||||
) as profile_dir:
|
||||
return _recalc_with_profile(filename, abs_path, timeout, Path(profile_dir))
|
||||
|
||||
|
||||
def _recalc_with_profile(filename, abs_path, timeout, profile_dir: Path):
|
||||
started = time.monotonic()
|
||||
profile_url, err = setup_libreoffice_macro(profile_dir, timeout=timeout)
|
||||
if err:
|
||||
return {"error": err}
|
||||
|
||||
timeout = max(5, int(timeout - (time.monotonic() - started)))
|
||||
|
||||
before = _stamp(abs_path)
|
||||
|
||||
cmd = [
|
||||
"soffice",
|
||||
"--headless",
|
||||
"--norestore",
|
||||
f"-env:UserInstallation={profile_url}",
|
||||
"vnd.sun.star.script:Standard.Module1.RecalculateAndSave?language=Basic&location=application",
|
||||
abs_path,
|
||||
]
|
||||
|
||||
if platform.system() == "Linux":
|
||||
if platform.system() == "Linux" and shutil.which("timeout"):
|
||||
cmd = ["timeout", str(timeout)] + cmd
|
||||
elif platform.system() == "Darwin" and has_gtimeout():
|
||||
cmd = ["gtimeout", str(timeout)] + cmd
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, env=get_soffice_env())
|
||||
timed_out = f"LibreOffice timed out after {timeout}s; formulas were NOT recalculated. Re-run with a longer timeout."
|
||||
|
||||
if result.returncode != 0 and result.returncode != 124:
|
||||
error_msg = result.stderr or "Unknown error during recalculation"
|
||||
if "Module1" in error_msg or "RecalculateAndSave" not in error_msg:
|
||||
return {"error": "LibreOffice macro not configured properly"}
|
||||
return {"error": error_msg}
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, env=get_soffice_env(), timeout=timeout + 15
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"error": timed_out}
|
||||
except FileNotFoundError:
|
||||
return {"error": SOFFICE_MISSING}
|
||||
|
||||
if result.returncode == 124:
|
||||
return {"error": timed_out}
|
||||
|
||||
if result.returncode != 0:
|
||||
detail = (result.stderr or "").strip() or f"soffice exited {result.returncode}"
|
||||
return {"error": f"LibreOffice failed to recalculate: {detail}"}
|
||||
|
||||
if _stamp(abs_path) == before:
|
||||
return {
|
||||
"error": (
|
||||
"LibreOffice exited cleanly but never rewrote the file, so nothing was "
|
||||
"recalculated. Check that no other LibreOffice instance is running, then retry."
|
||||
)
|
||||
}
|
||||
|
||||
try:
|
||||
wb = load_workbook(filename, data_only=True)
|
||||
@@ -114,6 +228,8 @@ def recalc(filename, timeout=30):
|
||||
|
||||
for sheet_name in wb.sheetnames:
|
||||
ws = wb[sheet_name]
|
||||
if not hasattr(ws, "iter_rows"):
|
||||
continue
|
||||
for row in ws.iter_rows():
|
||||
for cell in row:
|
||||
if cell.value is not None and isinstance(cell.value, str):
|
||||
@@ -124,8 +240,6 @@ def recalc(filename, timeout=30):
|
||||
total_errors += 1
|
||||
break
|
||||
|
||||
wb.close()
|
||||
|
||||
result = {
|
||||
"status": "success" if total_errors == 0 else "errors_found",
|
||||
"total_errors": total_errors,
|
||||
@@ -134,15 +248,19 @@ def recalc(filename, timeout=30):
|
||||
|
||||
for err_type, locations in error_details.items():
|
||||
if locations:
|
||||
result["error_summary"][err_type] = {
|
||||
"count": len(locations),
|
||||
"locations": locations[:20],
|
||||
}
|
||||
entry = {"count": len(locations), "locations": locations[:MAX_LOCATIONS]}
|
||||
if len(locations) > MAX_LOCATIONS:
|
||||
entry["locations_truncated"] = len(locations) - MAX_LOCATIONS
|
||||
result["error_summary"][err_type] = entry
|
||||
|
||||
wb.close()
|
||||
|
||||
wb_formulas = load_workbook(filename, data_only=False)
|
||||
formula_count = 0
|
||||
for sheet_name in wb_formulas.sheetnames:
|
||||
ws = wb_formulas[sheet_name]
|
||||
if not hasattr(ws, "iter_rows"):
|
||||
continue
|
||||
for row in ws.iter_rows():
|
||||
for cell in row:
|
||||
if (
|
||||
@@ -162,8 +280,11 @@ def recalc(filename, timeout=30):
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python recalc.py <excel_file> [timeout_seconds]")
|
||||
args = [a for a in sys.argv[1:] if a != "--force"]
|
||||
force = "--force" in sys.argv[1:]
|
||||
|
||||
if not args:
|
||||
print("Usage: python recalc.py <excel_file> [timeout_seconds] [--force]")
|
||||
print("\nRecalculates all formulas in an Excel file using LibreOffice")
|
||||
print("\nReturns JSON with error details:")
|
||||
print(" - status: 'success' or 'errors_found'")
|
||||
@@ -171,13 +292,16 @@ def main():
|
||||
print(" - total_formulas: Number of formulas in the file")
|
||||
print(" - error_summary: Breakdown by error type with locations")
|
||||
print(" - #VALUE!, #DIV/0!, #REF!, #NAME?, #NULL!, #NUM!, #N/A")
|
||||
print("\nOn any failure the JSON has an 'error' key and no 'status'.")
|
||||
print("--force recalculates even when it would destroy external links.")
|
||||
sys.exit(1)
|
||||
|
||||
filename = sys.argv[1]
|
||||
timeout = int(sys.argv[2]) if len(sys.argv) > 2 else 30
|
||||
filename = args[0]
|
||||
timeout = int(args[1]) if len(args) > 1 else 30
|
||||
|
||||
result = recalc(filename, timeout)
|
||||
result = recalc(filename, timeout, force=force)
|
||||
print(json.dumps(result, indent=2))
|
||||
sys.exit(1 if "error" in result else 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user