# One-time: install the build tools (pip, not npm — safe under the hold) pip install usd-core pyinstaller # Build the .exe pyinstaller --onefile --windowed --name DeadlineCleaner --collect-all pxr deadline_cleaner.py # VIDEO - Explanation Video : https://drive.google.com/file/d/1CZJn9puQDk0ywYm9UgEDb7l4ql433b6e/view?usp=sharing - Link to .py : https://drive.google.com/file/d/1yST-Ezu1ctN5EEA77ogf-7y-7tNXSoMe/view?usp=sharing # what I need to flag : - [x] mimap flagging - [x] Aov excess - [x] Memory power - [ ] USD Dependencies - [ ] Non-indexed attribute 'shop_materialpath' string - [ ] invalid mesh - [ ] bad primvar sample size - [ ] displacement shader doesn't modify P - [ ] singular matric detected Also find online answer for those # Display - [ ] Organize error per category # Deadline Log ANalyser — User Guide A tool that inspects Deadline render jobs and tells you **which renders are risky** — the ones that time out, eat too much RAM, load oversized USD scenes, use un-mipmapped textures, or write to the wrong server. It is **read-only**. It looks at render reports and the Deadline job database; it never changes a job, a render, or anything on the servers. --- ## 1. What it's for Student renders sometimes bring the farm — and the `DATA_PFE` server — to its knees. Deadline Cleaner reads the logs Deadline already produced and flags the jobs most likely to be the cause, so you can fix them before they cause another crash. Two typical uses: - **Daily triage** — "show me everything that failed on the farm this week." - **Single-job check** — "why did *my* job fail / why was it so slow?" --- ## 2. Before you start - **Windows**, with the **Deadline Client** installed (you already have it if you submit renders). The tool finds `deadlinecommand.exe` automatically. - That's it for the basic features. For **USD inspection** (texture + scene dependency analysis) the tool also needs the USD libraries — the packaged `.exe` already includes them, so you don't have to install anything. --- ## 3. Launching the tool **Double-click `DeadlineCleaner.exe`.** The window opens straight away. (If you were given the Python script instead of the `.exe`: open a terminal in its folder and run `python deadline_cleaner.py`.) A green line at the top — *"Deadline Client found"* — means the tool can talk to the farm. If it's red, the Deadline Client isn't installed or can't be found; ask a pipeline TD. --- ## 4. The window, control by control | Control | What it does | |---|---| | **Look back** | How far back to search: `24h`, `7d`, `30d`, or `all`. Ignored if you fill in Job IDs. | | **Max jobs to scan** | A slider — `5, 10, 20, 50, 100, … , All`. Limits how many jobs are pulled, newest first. Start small (20–50) for a quick look. | | **Job IDs** | Optional. Paste one or more 24-character job IDs (one per line) to analyse *exactly* those jobs and nothing else. This skips the time search. See §7. | | **Groups** | Tick/untick the student groups (`HYT, JRN, VRT, CNB, PMF, COR, GTE`, and `OTHER`). Unticked groups are left out of the report. | | **Deadline job status** | Which jobs to pull: `Failed`, `Active`, `Completed`, `Suspended`, `Pending`. Defaults to **Failed + Active** — the ones worth worrying about. | | **Inspect USD scenes** | Optional, slower. Opens each job's USD scene to count dependencies and check textures. Leave off for a fast first pass. | | **Output folder** | Where the report files are written. Default `report`. | | **Analyze** | Runs the analysis. Progress shows in the box below; the report opens in your browser when it's done. | --- ## 5. Running your first analysis 1. Leave **Look back** at `7d`. 2. Set **Max jobs** to `20` (quick). 3. Leave **Groups** all ticked, **job status** at Failed + Active. 4. Click **Analyze**. 5. After a few seconds the HTML report opens in your browser. That's the whole loop. Tighten the filters once you know what you're looking for. --- ## 6. Reading the report The report is an HTML page, organised **Group → User → Job**. ### Severity Every job gets a worst-case severity, colour-coded: | Badge | Meaning | |---|---| | 🔴 **critical** | Almost certainly hurting the farm — fix first. | | 🟠 **high** | A real problem; fix soon. | | 🟡 **medium** | Worth correcting; not an emergency. | | 🔵 **low** | Minor / informational. | | ⚪ **info** | Nothing wrong detected. | Groups and users with **critical** or **high** items are expanded automatically; quieter ones stay collapsed. The chips at the top jump you to a group. ### A job card Each card shows the job name, frame, elapsed time, peak RAM, AOV count, texture summary, USD dependency size, output path, and a list of **flags** — the specific things found wrong. It also shows two states: - **task: …** — how that one frame ended (success / timeout / failed / …). - **job: …** — the overall Deadline job status. These differ on purpose: a job marked **Failed** can still have many frames that rendered fine — only the bad frame failed. --- ## 7. Checking one specific job Spotted a red job in **Deadline Monitor**? 1. Right-click it → **Copy Job ID**. 2. Paste it into the tool's **Job IDs** box (you can paste several, one per line). 3. Click **Analyze**. The tool fetches just those jobs — fast — and ignores the Look-back / status filters. --- ## 8. What the flags mean | Flag | Severity | What it means & why it matters | |---|---|---| | `timeout` | critical | Deadline killed the task for running too long. | | `stuck_progress` | critical | Less than 5 % rendered after 30 minutes — the render is not converging. | | `output_to_data_pfe` | critical | The render is writing its EXRs to `V:` (DATA_PFE). Renders must write to `W:` (DATA_FRM). This directly loads the server that crashes. | | `ram_saturation` | critical | Peak RAM came within 1 GB of the worker's total — the machine was about to swap or crash. | | `long_frame` | high | A single frame took over 1 h 30 — our hard ceiling. | | `memory_grower` | high | Karma's memory kept climbing during the render (a leak / runaway scene). | | `usd_load_failed` | high | Husk could not open the USD scene — broken or missing reference. | | `non_zero_exit` | high | The renderer exited with an error and no more specific cause was found. | | `mipmap_missing` | high / medium | Textures are not mipmapped (`UNTILED`). Convert them — `.rat` for Karma, `.tex` for Renderman. | | `aov_excess` | medium | More than 20 AOVs in the render — usually more than the shot needs. | | `wont_fit_32gb` | medium | Peak RAM ≥ 31 GB — this job can't run on a 32 GB machine. | | `env_config_error` | medium | The worker had no `HUSK_PATH` set — the render never started. | | `usd_dep_heavy` | medium | 200+ USD layers pulled in — a very heavy scene graph. | | `wont_fit_16gb` | low | Peak RAM ≥ 15 GB — this job can't run on a 16 GB machine. | | `mipmap_offspec` | low | Textures are tiled but not in the renderer-native `.rat` / `.tex` format. | | `render_warnings` | low | Non-fatal warnings from the renderer (missing primvars, light/displacement notes, etc.). | ### Quick fixes - **`output_to_data_pfe`** → repoint the render output to `W:/` in your render settings. - **`mipmap_missing` / `mipmap_offspec`** → convert textures to `.rat` (Karma) or `.tex` (Renderman) before rendering. - **`aov_excess`** → delete render vars the shot doesn't actually use. - **`long_frame` / `timeout` / `memory_grower`** → lighten the scene: lower samples, reduce volume detail, check for runaway geometry or instances. - **`wont_fit_16gb` / `wont_fit_32gb`** → reduce memory use, or have the job pinned to bigger workers. --- ## 9. The output files In your chosen output folder: | File | Use | |---|---| | `report.html` | The main report — open this in a browser. | | `report.csv` | The same data as a spreadsheet, for sorting/filtering. | | `per_student.md` | A plain-text rollup grouped by group → student. | --- ## 10. Command line (optional) For those comfortable with a terminal: ``` # Failed + Active jobs from the last 7 days python deadline_cleaner.py --deadline --since 7d -o report # Only failed jobs, last 24h, with USD inspection python deadline_cleaner.py --deadline --job-status Failed --since 24h --inspect-usd -o report # Specific jobs by ID python deadline_cleaner.py --deadline --job-ids 6a00cd5b756fadd627ead018,69f9fceda9224a4ba9985021 -o report ``` Run with no arguments to open the GUI. --- ## 11. Troubleshooting | Symptom | Fix | |---|---| | Red "deadlinecommand not found" line | Deadline Client isn't installed / on this machine. Use a workstation that has it. | | "Nothing to analyze" | Widen **Look back**, tick more **job statuses**, or check your Job IDs are valid 24-hex strings. | | USD row says `no_pxr` | USD libraries aren't available — use the packaged `.exe`, which bundles them. | | Textures show "no data" for a Renderman job | Renderman doesn't log texture stats by default. Tick **Inspect USD scenes** to read them from the scene instead. | | Lots of `unreachable` textures | Some texture paths point at drives not mapped on this machine — the job itself is fine, the tool just can't measure those files from here. | --- ## 12. What it can't tell you - It reads **reports of renders that already ran** — it can't predict a job that hasn't rendered yet. - It diagnoses; it does **not** fix jobs or scenes. Use it to produce a report, then correct the scene in Houdini / Prism yourself. - `Nuke` comp jobs and Prism `cleanup` jobs are intentionally skipped — they aren't the renders that crash the farm.