mirror of
https://github.com/anthropics/skills.git
synced 2026-08-02 13:05:28 +08:00
Update docx, xlsx, pdf, pptx skills with latest improvements (#330)
docx: Add commenting and track-changes support. Reorganize OOXML tooling into a shared office/ module. pptx: Streamline SKILL.md, add slide-editing and pptxgenjs guides, bundle html2pptx as a tgz. Reorganize OOXML tooling into a shared office/ module. xlsx: Move recalc script into scripts/ and expand it. Add shared office/ module for OOXML pack/unpack/validate. pdf: Improve form-filling workflow with new form-structure extraction script and updated field-info extraction.
This commit is contained in:
Keith Lazuka
+187
-98
@@ -76,130 +76,219 @@ Then analyze the images to determine the purpose of each form field (make sure t
|
||||
This script will verify that the field IDs and values you provide are valid; if it prints error messages, correct the appropriate fields and try again.
|
||||
|
||||
# Non-fillable fields
|
||||
If the PDF doesn't have fillable form fields, you'll need to visually determine where the data should be added and create text annotations. Follow the below steps *exactly*. You MUST perform all of these steps to ensure that the the form is accurately completed. Details for each step are below.
|
||||
- Convert the PDF to PNG images and determine field bounding boxes.
|
||||
- Create a JSON file with field information and validation images showing the bounding boxes.
|
||||
- Validate the the bounding boxes.
|
||||
- Use the bounding boxes to fill in the form.
|
||||
If the PDF doesn't have fillable form fields, you'll add text annotations. First try to extract coordinates from the PDF structure (more accurate), then fall back to visual estimation if needed.
|
||||
|
||||
## Step 1: Visual Analysis (REQUIRED)
|
||||
- Convert the PDF to PNG images. Run this script from this file's directory:
|
||||
`python scripts/convert_pdf_to_images.py <file.pdf> <output_directory>`
|
||||
The script will create a PNG image for each page in the PDF.
|
||||
- Carefully examine each PNG image and identify all form fields and areas where the user should enter data. For each form field where the user should enter text, determine bounding boxes for both the form field label, and the area where the user should enter text. The label and entry bounding boxes MUST NOT INTERSECT; the text entry box should only include the area where data should be entered. Usually this area will be immediately to the side, above, or below its label. Entry bounding boxes must be tall and wide enough to contain their text.
|
||||
## Step 1: Try Structure Extraction First
|
||||
|
||||
These are some examples of form structures that you might see:
|
||||
Run this script to extract text labels, lines, and checkboxes with their exact PDF coordinates:
|
||||
`python scripts/extract_form_structure.py <input.pdf> form_structure.json`
|
||||
|
||||
*Label inside box*
|
||||
```
|
||||
┌────────────────────────┐
|
||||
│ Name: │
|
||||
└────────────────────────┘
|
||||
```
|
||||
The input area should be to the right of the "Name" label and extend to the edge of the box.
|
||||
This creates a JSON file containing:
|
||||
- **labels**: Every text element with exact coordinates (x0, top, x1, bottom in PDF points)
|
||||
- **lines**: Horizontal lines that define row boundaries
|
||||
- **checkboxes**: Small square rectangles that are checkboxes (with center coordinates)
|
||||
- **row_boundaries**: Row top/bottom positions calculated from horizontal lines
|
||||
|
||||
*Label before line*
|
||||
```
|
||||
Email: _______________________
|
||||
```
|
||||
The input area should be above the line and include its entire width.
|
||||
**Check the results**: If `form_structure.json` has meaningful labels (text elements that correspond to form fields), use **Approach A: Structure-Based Coordinates**. If the PDF is scanned/image-based and has few or no labels, use **Approach B: Visual Estimation**.
|
||||
|
||||
*Label under line*
|
||||
```
|
||||
_________________________
|
||||
Name
|
||||
```
|
||||
The input area should be above the line and include the entire width of the line. This is common for signature and date fields.
|
||||
---
|
||||
|
||||
*Label above line*
|
||||
```
|
||||
Please enter any special requests:
|
||||
________________________________________________
|
||||
```
|
||||
The input area should extend from the bottom of the label to the line, and should include the entire width of the line.
|
||||
## Approach A: Structure-Based Coordinates (Preferred)
|
||||
|
||||
*Checkboxes*
|
||||
```
|
||||
Are you a US citizen? Yes □ No □
|
||||
```
|
||||
For checkboxes:
|
||||
- Look for small square boxes (□) - these are the actual checkboxes to target. They may be to the left or right of their labels.
|
||||
- Distinguish between label text ("Yes", "No") and the clickable checkbox squares.
|
||||
- The entry bounding box should cover ONLY the small square, not the text label.
|
||||
Use this when `extract_form_structure.py` found text labels in the PDF.
|
||||
|
||||
### Step 2: Create fields.json and validation images (REQUIRED)
|
||||
- Create a file named `fields.json` with information for the form fields and bounding boxes in this format:
|
||||
```
|
||||
### A.1: Analyze the Structure
|
||||
|
||||
Read form_structure.json and identify:
|
||||
|
||||
1. **Label groups**: Adjacent text elements that form a single label (e.g., "Last" + "Name")
|
||||
2. **Row structure**: Labels with similar `top` values are in the same row
|
||||
3. **Field columns**: Entry areas start after label ends (x0 = label.x1 + gap)
|
||||
4. **Checkboxes**: Use the checkbox coordinates directly from the structure
|
||||
|
||||
**Coordinate system**: PDF coordinates where y=0 is at TOP of page, y increases downward.
|
||||
|
||||
### A.2: Check for Missing Elements
|
||||
|
||||
The structure extraction may not detect all form elements. Common cases:
|
||||
- **Circular checkboxes**: Only square rectangles are detected as checkboxes
|
||||
- **Complex graphics**: Decorative elements or non-standard form controls
|
||||
- **Faded or light-colored elements**: May not be extracted
|
||||
|
||||
If you see form fields in the PDF images that aren't in form_structure.json, you'll need to use **visual analysis** for those specific fields (see "Hybrid Approach" below).
|
||||
|
||||
### A.3: Create fields.json with PDF Coordinates
|
||||
|
||||
For each field, calculate entry coordinates from the extracted structure:
|
||||
|
||||
**Text fields:**
|
||||
- entry x0 = label x1 + 5 (small gap after label)
|
||||
- entry x1 = next label's x0, or row boundary
|
||||
- entry top = same as label top
|
||||
- entry bottom = row boundary line below, or label bottom + row_height
|
||||
|
||||
**Checkboxes:**
|
||||
- Use the checkbox rectangle coordinates directly from form_structure.json
|
||||
- entry_bounding_box = [checkbox.x0, checkbox.top, checkbox.x1, checkbox.bottom]
|
||||
|
||||
Create fields.json using `pdf_width` and `pdf_height` (signals PDF coordinates):
|
||||
```json
|
||||
{
|
||||
"pages": [
|
||||
{
|
||||
"page_number": 1,
|
||||
"image_width": (first page image width in pixels),
|
||||
"image_height": (first page image height in pixels),
|
||||
},
|
||||
{
|
||||
"page_number": 2,
|
||||
"image_width": (second page image width in pixels),
|
||||
"image_height": (second page image height in pixels),
|
||||
}
|
||||
// additional pages
|
||||
{"page_number": 1, "pdf_width": 612, "pdf_height": 792}
|
||||
],
|
||||
"form_fields": [
|
||||
// Example for a text field.
|
||||
{
|
||||
"page_number": 1,
|
||||
"description": "The user's last name should be entered here",
|
||||
// Bounding boxes are [left, top, right, bottom]. The bounding boxes for the label and text entry should not overlap.
|
||||
"field_label": "Last name",
|
||||
"label_bounding_box": [30, 125, 95, 142],
|
||||
"entry_bounding_box": [100, 125, 280, 142],
|
||||
"entry_text": {
|
||||
"text": "Johnson", // This text will be added as an annotation at the entry_bounding_box location
|
||||
"font_size": 14, // optional, defaults to 14
|
||||
"font_color": "000000", // optional, RRGGBB format, defaults to 000000 (black)
|
||||
}
|
||||
"description": "Last name entry field",
|
||||
"field_label": "Last Name",
|
||||
"label_bounding_box": [43, 63, 87, 73],
|
||||
"entry_bounding_box": [92, 63, 260, 79],
|
||||
"entry_text": {"text": "Smith", "font_size": 10}
|
||||
},
|
||||
// Example for a checkbox. TARGET THE SQUARE for the entry bounding box, NOT THE TEXT
|
||||
{
|
||||
"page_number": 2,
|
||||
"description": "Checkbox that should be checked if the user is over 18",
|
||||
"entry_bounding_box": [140, 525, 155, 540], // Small box over checkbox square
|
||||
"page_number": 1,
|
||||
"description": "US Citizen Yes checkbox",
|
||||
"field_label": "Yes",
|
||||
"label_bounding_box": [100, 525, 132, 540], // Box containing "Yes" text
|
||||
// Use "X" to check a checkbox.
|
||||
"entry_text": {
|
||||
"text": "X",
|
||||
}
|
||||
"label_bounding_box": [260, 200, 280, 210],
|
||||
"entry_bounding_box": [285, 197, 292, 205],
|
||||
"entry_text": {"text": "X"}
|
||||
}
|
||||
// additional form field entries
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Create validation images by running this script from this file's directory for each page:
|
||||
`python scripts/create_validation_image.py <page_number> <path_to_fields.json> <input_image_path> <output_image_path>
|
||||
**Important**: Use `pdf_width`/`pdf_height` and coordinates directly from form_structure.json.
|
||||
|
||||
The validation images will have red rectangles where text should be entered, and blue rectangles covering label text.
|
||||
### A.4: Validate Bounding Boxes
|
||||
|
||||
### Step 3: Validate Bounding Boxes (REQUIRED)
|
||||
#### Automated intersection check
|
||||
- Verify that none of bounding boxes intersect and that the entry bounding boxes are tall enough by checking the fields.json file with the `check_bounding_boxes.py` script (run from this file's directory):
|
||||
`python scripts/check_bounding_boxes.py <JSON file>`
|
||||
Before filling, check your bounding boxes for errors:
|
||||
`python scripts/check_bounding_boxes.py fields.json`
|
||||
|
||||
If there are errors, reanalyze the relevant fields, adjust the bounding boxes, and iterate until there are no remaining errors. Remember: label (blue) bounding boxes should contain text labels, entry (red) boxes should not.
|
||||
This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
|
||||
|
||||
#### Manual image inspection
|
||||
**CRITICAL: Do not proceed without visually inspecting validation images**
|
||||
- Red rectangles must ONLY cover input areas
|
||||
- Red rectangles MUST NOT contain any text
|
||||
- Blue rectangles should contain label text
|
||||
- For checkboxes:
|
||||
- Red rectangle MUST be centered on the checkbox square
|
||||
- Blue rectangle should cover the text label for the checkbox
|
||||
---
|
||||
|
||||
- If any rectangles look wrong, fix fields.json, regenerate the validation images, and verify again. Repeat this process until the bounding boxes are fully accurate.
|
||||
## Approach B: Visual Estimation (Fallback)
|
||||
|
||||
Use this when the PDF is scanned/image-based and structure extraction found no usable text labels (e.g., all text shows as "(cid:X)" patterns).
|
||||
|
||||
### Step 4: Add annotations to the PDF
|
||||
Run this script from this file's directory to create a filled-out PDF using the information in fields.json:
|
||||
`python scripts/fill_pdf_form_with_annotations.py <input_pdf_path> <path_to_fields.json> <output_pdf_path>
|
||||
### B.1: Convert PDF to Images
|
||||
|
||||
`python scripts/convert_pdf_to_images.py <input.pdf> <images_dir/>`
|
||||
|
||||
### B.2: Initial Field Identification
|
||||
|
||||
Examine each page image to identify form sections and get **rough estimates** of field locations:
|
||||
- Form field labels and their approximate positions
|
||||
- Entry areas (lines, boxes, or blank spaces for text input)
|
||||
- Checkboxes and their approximate locations
|
||||
|
||||
For each field, note approximate pixel coordinates (they don't need to be precise yet).
|
||||
|
||||
### B.3: Zoom Refinement (CRITICAL for accuracy)
|
||||
|
||||
For each field, crop a region around the estimated position to refine coordinates precisely.
|
||||
|
||||
**Create a zoomed crop using ImageMagick:**
|
||||
```bash
|
||||
magick <page_image> -crop <width>x<height>+<x>+<y> +repage <crop_output.png>
|
||||
```
|
||||
|
||||
Where:
|
||||
- `<x>, <y>` = top-left corner of crop region (use your rough estimate minus padding)
|
||||
- `<width>, <height>` = size of crop region (field area plus ~50px padding on each side)
|
||||
|
||||
**Example:** To refine a "Name" field estimated around (100, 150):
|
||||
```bash
|
||||
magick images_dir/page_1.png -crop 300x80+50+120 +repage crops/name_field.png
|
||||
```
|
||||
|
||||
(Note: if the `magick` command isn't available, try `convert` with the same arguments).
|
||||
|
||||
**Examine the cropped image** to determine precise coordinates:
|
||||
1. Identify the exact pixel where the entry area begins (after the label)
|
||||
2. Identify where the entry area ends (before next field or edge)
|
||||
3. Identify the top and bottom of the entry line/box
|
||||
|
||||
**Convert crop coordinates back to full image coordinates:**
|
||||
- full_x = crop_x + crop_offset_x
|
||||
- full_y = crop_y + crop_offset_y
|
||||
|
||||
Example: If the crop started at (50, 120) and the entry box starts at (52, 18) within the crop:
|
||||
- entry_x0 = 52 + 50 = 102
|
||||
- entry_top = 18 + 120 = 138
|
||||
|
||||
**Repeat for each field**, grouping nearby fields into single crops when possible.
|
||||
|
||||
### B.4: Create fields.json with Refined Coordinates
|
||||
|
||||
Create fields.json using `image_width` and `image_height` (signals image coordinates):
|
||||
```json
|
||||
{
|
||||
"pages": [
|
||||
{"page_number": 1, "image_width": 1700, "image_height": 2200}
|
||||
],
|
||||
"form_fields": [
|
||||
{
|
||||
"page_number": 1,
|
||||
"description": "Last name entry field",
|
||||
"field_label": "Last Name",
|
||||
"label_bounding_box": [120, 175, 242, 198],
|
||||
"entry_bounding_box": [255, 175, 720, 218],
|
||||
"entry_text": {"text": "Smith", "font_size": 10}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Important**: Use `image_width`/`image_height` and the refined pixel coordinates from the zoom analysis.
|
||||
|
||||
### B.5: Validate Bounding Boxes
|
||||
|
||||
Before filling, check your bounding boxes for errors:
|
||||
`python scripts/check_bounding_boxes.py fields.json`
|
||||
|
||||
This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
|
||||
|
||||
---
|
||||
|
||||
## Hybrid Approach: Structure + Visual
|
||||
|
||||
Use this when structure extraction works for most fields but misses some elements (e.g., circular checkboxes, unusual form controls).
|
||||
|
||||
1. **Use Approach A** for fields that were detected in form_structure.json
|
||||
2. **Convert PDF to images** for visual analysis of missing fields
|
||||
3. **Use zoom refinement** (from Approach B) for the missing fields
|
||||
4. **Combine coordinates**: For fields from structure extraction, use `pdf_width`/`pdf_height`. For visually-estimated fields, you must convert image coordinates to PDF coordinates:
|
||||
- pdf_x = image_x * (pdf_width / image_width)
|
||||
- pdf_y = image_y * (pdf_height / image_height)
|
||||
5. **Use a single coordinate system** in fields.json - convert all to PDF coordinates with `pdf_width`/`pdf_height`
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Validate Before Filling
|
||||
|
||||
**Always validate bounding boxes before filling:**
|
||||
`python scripts/check_bounding_boxes.py fields.json`
|
||||
|
||||
This checks for:
|
||||
- Intersecting bounding boxes (which would cause overlapping text)
|
||||
- Entry boxes that are too small for the specified font size
|
||||
|
||||
Fix any reported errors in fields.json before proceeding.
|
||||
|
||||
## Step 3: Fill the Form
|
||||
|
||||
The fill script auto-detects the coordinate system and handles conversion:
|
||||
`python scripts/fill_pdf_form_with_annotations.py <input.pdf> fields.json <output.pdf>`
|
||||
|
||||
## Step 4: Verify Output
|
||||
|
||||
Convert the filled PDF to images and verify text placement:
|
||||
`python scripts/convert_pdf_to_images.py <output.pdf> <verify_images/>`
|
||||
|
||||
If text is mispositioned:
|
||||
- **Approach A**: Check that you're using PDF coordinates from form_structure.json with `pdf_width`/`pdf_height`
|
||||
- **Approach B**: Check that image dimensions match and coordinates are accurate pixels
|
||||
- **Hybrid**: Ensure coordinate conversions are correct for visually-estimated fields
|
||||
|
||||
Reference in New Issue
Block a user