diff --git a/DATA/production/configurations/mapCreationGrid/README.md b/DATA/production/configurations/mapCreationGrid/README.md new file mode 100644 index 000000000..d022fe8f5 --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/README.md @@ -0,0 +1,402 @@ +# TPC Map Creation on the GRID + +Runs the ALICE TPC space-charge distortion map creation chain on the ALICE GRID, as a two-stage +pipeline: a heavy map-creation stage (hours per slot) followed by a fast post-processing stage +(minutes per slot). Ported from the original SLURM/Lustre-based production. + +The four ROOT macros this pipeline runs (`staticMapCreatorCPM.C`, `SmoothingExtrapolate.C`, +`voxResQA.C`, `TPCFastTransformInitCPM.C`) live in the main O2 repository, under +`Detectors/TPC/calibration/SpacePoints/macro/` and `GPU/TPCFastTransformation/macro/` respectively, and +install to `$O2_ROOT/share/macro/` as part of a normal O2 build -- this directory only holds the GRID +submission/orchestration layer (list discovery, manifest staging, `grid_submit.sh` wrappers, per-slot +job scripts). + +## Quickstart + +For a normal production on the official O2 build, with defaults left alone, only two settings usually +need changing: the bad-time-ranges list and how long each map should span. Everything else below can be +left as-is. + +1. **Build the slot lists** (local, once per batch of runs): + + ```bash + ./discoverResiduals.sh + ``` + + `` lists one run per line: ` `, e.g. `LHC26ak 572557`. + +2. **Submit stage 1 (map creation).** In `submitMapCreation.sh`, set: + - `BAD_RANGES_NAME` — the bad-time-ranges file for this data period (keep the default if it still + applies). + - `SLOT_LENGTH_MIN` — how many minutes of data go into one map, e.g. `5`. + - `JOBNAME` — any name that's unique to this submission; you'll need it again in step 3. + + Then: + + ```bash + ./submitMapCreation.sh [path/to/lists] + ``` + + `path/to/lists` is the `./lists` directory step 1 wrote (default: `./lists` in the current directory) + — only pass it explicitly if you're running this from a different directory than step 1. + + This takes hours per map — watch progress on alimonitor and wait for it to finish (or mostly finish) + before continuing. + +3. **Submit stage 2 (smoothing + FastTransform).** In `submitPostProcess.sh`, set `STAGE1_JOBNAME_TAG` + to the exact `JOBNAME` used in step 2, then: + + ```bash + ./submitPostProcess.sh + ``` + + This takes minutes per map and produces the final smoothed map, the `FT_*.root` FastTransform file, + and QA plots. + +Everything else in this document explains what the other settings do and how to debug things when they +don't go as expected — skip ahead to it only if you need to change something beyond the three above. + +## Prerequisites + +- A valid AliEn/JAliEn token (`alien-token-init `) and `alien.py` on your `PATH`. +- `O2DPG_ROOT` set (provides `grid_submit.sh`). +- **bash ≥ 4** for the locally-run scripts (`discoverResiduals.sh`, `makeResidualLists.sh`) — they use + `mapfile`. macOS still ships bash 3.2, so on a Mac install a newer one (`brew install bash`) and make + sure it precedes `/bin` on your `PATH`; `#!/usr/bin/env bash` then picks it up. Note this also makes + `set -e` stricter (an arithmetic `(( ))` evaluating to 0 aborts), which is the behaviour lxplus has. +- GNU `parallel` (used by `discoverResiduals.sh`). +- Write access to your AliEn home directory. The wrappers derive + `alien:///alice/cern.ch/user///MapCreation` automatically from `ALIEN_USER` (falling + back to your local username), the same way `grid_submit.sh` derives `MY_HOMEDIR` — **no per-user edit + needed**. Both wrappers print the resolved directory before doing anything, so check that line on your + first run: if your laptop's local username differs from your AliEn account, `export ALIEN_USER=`, or set `MAPCREATION_ALIEN_DIR` to override the whole path. This directory only holds + the slot manifest, the bad-time-ranges list, and the `stage1Workdir` pointer files -- not the actual + map data (see Step 3) -- and, if `MACRO_SOURCE=alien` (see below), a staged copy of the macros. +- A `--packagespec` tag whose O2 build includes the four macros above, **or** `MACRO_SOURCE=alien` (see + "Where the macros come from" below) if no such tag is published yet. + +## Where the macros come from (`MACRO_SOURCE`) + +Every payload script (`mapCreationJob.sh`, `postProcessJob.sh`) reads a +`MACRO_SOURCE` variable, substituted by its wrapper the same way as `JOBNAME_TAG`: + +- **`official` (default)**: runs the macros straight from the O2 package this job's `--packagespec` + loads (`$O2_ROOT/share/macro/.C`) — no staging or download at all. This is the same idiom + `DATA/production/configurations/asyncCalib/createCorrectionMap.sh` already uses for the older + `staticMapCreator.C`/`TPCFastTransformInit.C` pair. Requires `PACKAGESPEC` to resolve to an O2 build new + enough to include `staticMapCreatorCPM.C` et al. If it doesn't, the job fails fast with a clear error + rather than silently falling back. +- **`alien`**: stages a macro from a local path (`MACRO_LOCAL_PATH` in `submitMapCreation.sh`, + `MACRO_LOCAL_DIR` in `submitPostProcess.sh`) to AliEn, and jobs download it + from there instead. Use this until a `--packagespec` tag with the macros is published — point + `MACRO_LOCAL_PATH`/`MACRO_LOCAL_DIR` at your own O2 checkout's copy of the macro(s). + +## The chain, in order + +``` +discoverResiduals.sh (local) find residual files on AliEn, build per-run slot lists + │ + ▼ +submitMapCreation.sh (submits) STAGE 1: map creation (hours/slot) + │ └─ per-slot payload: mapCreationJob.sh + ▼ (wait for jobs to finish on alimonitor) + │ +submitPostProcess.sh (submits) STAGE 2: smoothing + FastTransform (minutes/slot) + └─ per-slot payload: postProcessJob.sh +``` + +Stage 1 and stage 2 are **independent GRID submissions**, not one script. If some stage-1 slots fail +or are still running, stage 2 simply skips those (see "Partial failure" below) — no need to wait for +100% of stage 1 before submitting stage 2, and a failed stage-2 subjob can be resubmitted on its own +without redoing the expensive stage-1 work for that slot. + +### Step 1 — discover residual files and build slot lists (local, once per batch of runs) + +```bash +./discoverResiduals.sh [--calib-pass cpass2_residuals] +``` + +`` is one run per line: ` `, e.g.: + +``` +LHC26ak 572557 +LHC26ak 572558 +``` + +This queries AliEn (`/alice/data/////`) for `o2tpc_residuals_*.root` +files and writes per-run slot lists under `./lists//`. Runs in parallel (`N_PARALLEL=8` in the +script). Safe to re-run; each run is independent. The period is resolved **per line**, so a batch file +may legitimately span periods; malformed rows are rejected up front rather than as one failed run +mid-batch. + +**One list per run — granularity comes from `SLOT_LENGTH_MIN`, not from the catalog.** Each residual +filename carries the aggregation slot label `___` its reco job +assigned, and those labels are **not** a uniform grid — same-length slots are not guaranteed, and a +run's catalog can hold overlapping/nested labels at different granularities. Example (run 572266): + +``` +0_105425 (22 files) 105426_210851 (6) 0_210851 (35) +210852_316277 (2) 210852_421703 (45) +``` + +Slicing per label would give overlapping slots with wildly uneven statistics — the 2- and 6-file ones +trip `MIN_TRACKS_PER_SLICE` and come back `badCalib_` while the data that would have fixed them sits in +a sibling list. So `makeResidualLists.sh` writes **one list per run** holding every file, and you choose +the map length with `SLOT_LENGTH_MIN` at submission time. It is then uniform across runs and independent +of whatever granularity the upstream catalog happens to have. + +`stageSlotsForGrid.py` cuts each run into windows of `SLOT_LENGTH_MIN` and writes **one manifest line +per window**, carrying that window's bounds and only the files whose label overlaps it. A file's +contents are always a subset of its label's range, so this selection never discards data that could +contribute, and it costs nothing — no file is opened. That matters: the macro's own per-file skip has to +*open* each out-of-window file to read its time range (~7 s each over `alien://`, at ~zero CPU), and +AliEn kills a payload whose CPU stays low for 15 min. For run 572266: + +| `SLOT_LENGTH_MIN` | windows × files per subjob (of 110) | +|---|---| +| `0` (one map per run) | 1 × 110 | +| `10` | 2 × (63, 47) | +| `5` | 4 × (57, 41, 47, 45) | + +A coarse-labelled file spanning several windows is deliberately kept by each of them; the macro's per-TF +`firstTFTime`/`lastTFTime` cut then decides which TFs each map actually uses, so nothing is +double-counted. That is why the per-subjob counts can sum to more than the total. + +A pure overlap test is too generous at window edges, though: the aggregator's coarse labels routinely +begin 1–2 ms before a window ends, and that was enough to drag all 45 of their files into a window whose +map they contribute nothing to (run 572266 at 5 min: window 1 took 88 files to use 41). So a file is +only assigned to a window when the overlap **exceeds `OVERLAP_MARGIN_MS`** (default 1000). The cost is +bounded exactly: if a label overlaps by X ms then at most X ms of that file's data belongs here, so +skipping at X ≤ margin loses at most `margin` ms of coverage per window edge — 1 s out of 300 s. The +margin is clamped to a tenth of the window, and `stageSlotsForGrid.py` verifies that every file still +lands in at least one window, warning loudly if the margin is ever set high enough to orphan one. +With it, run 572266 at 5 min gives 57 / 41 / 47 / 45 instead of 63 / 88 / 47 / 45. + +`mapCreationJob.sh` additionally *orders* each window's files by orbit distance from the window centre, +so whatever still comes along is opened last and `MAX_TRACKS_PER_SLICE` usually stops the job first. + +Neither job script does any time arithmetic: both read the run and the window straight off the same +manifest line, so they cannot disagree about a slot's identity. + +**Previewing the split.** The windows are decided at submission time, not at discovery, so `./lists/` +holds one file per *run*, not per window. To see how a given `SLOT_LENGTH_MIN` actually slices your +runs, run the manifest builder in preview mode — it skips the AliEn upload, so it needs no token and +re-queries nothing; add `--window-lists` to also write one plain file list per window: + +```bash +./stageSlotsForGrid.py --lists-dir ./lists --manifest-name mapSlots.preview.tsv \ + --slot-length-min 5 --preview --window-lists +ls mapSlots.preview.windows/ # one .TDof._.txt per window +``` + +`--preview` skips the upload, so it needs no token and re-queries nothing; `--window-lists` additionally +writes the per-window lists. See `./stageSlotsForGrid.py --help` for the full option list. + +The window boundaries and file counts are printed as it goes, which is all you usually need. To see +*which* files a window got, add `--window-lists`. That is off by default because on a parallel +filesystem each list is a metadata round trip: on Lustre, 120 of them measured **32.8 s**, against +0.03 s for all the actual work. Pair it with `--window-dir /tmp/windows` to keep them on local disk. +`--timing` prints the read/compute/write breakdown. + +The manifest itself is always written locally — one file, and the complete record of the split (one +line per window, sixth column being the comma-separated file list). + +Cheap enough to compare several lengths back to back. For run 572266 (20 min): + +| `SLOT_LENGTH_MIN` | windows | files per window | +|---|---|---| +| `10` | 2 | 63, 47 | +| `5` | 4 | 57, 41, 47, 45 | +| `2` | 10 | 57, 57, 63, 41, 41, 47, 47, 47, 45, 45 | + +Note the counts stop falling below ~5 min: a coarse aggregation label spanning 10 minutes overlaps +*every* 2-minute window inside it, so slicing finer than the aggregator's own slot length buys time +resolution but not less I/O per job. Add `--window-lists` to a normal (non-preview) run to write the +same directory alongside the uploaded manifest, so you can check afterwards what a submission actually +used. + +**`./lists/` accumulates.** A second batch discovered into the same tree does not replace the first — +it adds to it, and Step 2 by default puts *everything* under `./lists/` into the manifest. That +silently recalibrates the earlier batch from scratch at hours per slot. Either use a fresh `listsDir` +per batch, or set `RUNS_FILTER` in the Step-2 wrapper (see below). + +### Step 2 — configure and submit stage 1 (map creation) + +Edit the config block at the top of **`submitMapCreation.sh`**: + +| Variable | Meaning | +|---|---| +| `JOBNAME` | Identifies this production. Tags the AliEn manifest (`mapSlots..tsv`) and the small `stage1Workdir..txt` pointer file (see Step 3) — **must be unique per submission** so concurrent/repeated productions don't clobber each other's manifest or pointer. Stage 2 needs to know this value (see Step 4). | +| `TOPWORKDIR` | Static top-level bucket under your GRID home for this job's working directory (replaces `grid_submit.sh`'s generic `selfjobs` default with something identifiable). | +| `SLOT_LENGTH_MIN` | Target length of **one map**, in minutes. Each run is cut into whole steps of this length; a trailing remainder shorter than half a step is absorbed into the last window rather than left as a runt that would fail `MIN_TRACKS_PER_SLICE`. `0` = one map per run. A 20 min run at `5` gives 4 maps, a 60 min run gives 12 — the count follows the run length instead of being fixed. | +| `PACKAGESPEC` | O2/O2DPG software tag to run under — set this to the async-reco tag matching your data. Also determines whether `MACRO_SOURCE=official` can find the macros (see "Where the macros come from" above). | +| `ASUSER` | AliEn account this job runs/is charged as, e.g. `pwg_pp` (your account must already be a member of that group) instead of your own personal account/quota. Determines both which quota pays for the job and where its `${ALIEN_JDL_OUTPUTDIR}` lands (see Step 3). Verify on alimonitor that the first job is really attributed correctly before relying on it for a big batch. | +| `MACRO_SOURCE` / `MACRO_LOCAL_PATH` | See "Where the macros come from" above. | +| `RUNS_FILTER` | Optional comma-separated run whitelist, e.g. `"572557,572558"`. Empty means every run found under `listsDir` — including runs left over from an earlier batch (see Step 1). Naming a run that has no lists is a hard error, not a silently smaller production. | +| `BAD_RANGES_NAME` | Filename of the bad-time-ranges list under `${MACRO_ALIEN_DIR}/lists/`. Period- and system-specific (the default is the PbPb26 list), so change it when the data changes. It is both the upload destination and the name each job downloads, so the two cannot drift apart. Set to `""` to disable bad-ranges filtering entirely for this submission — a deliberate opt-out (`staticMapCreatorCPM.C` treats an empty filename that way), not the same as forgetting to set it. | +| `BAD_RANGES_LOCAL` | Local path to the bad-time-ranges list, only needed if you want to (re-)upload a new one this submission. Leave empty to reuse whatever is already staged on AliEn under `BAD_RANGES_NAME` — the wrapper checks that it actually exists there before submitting anything, so a wrong `BAD_RANGES_NAME`/`ALIEN_USER` or a never-uploaded file fails fast, locally, once, instead of inside every GRID subjob. | +| `MACRO_ALIEN_DIR_BARE` | AliEn directory (no `alien://` prefix) used for staging the slot manifest and the small `stage1Workdir` pointer file (see Step 3) — and the macro too, when `MACRO_SOURCE=alien`. **Not** the actual map data, which goes to `${ALIEN_JDL_OUTPUTDIR}` instead. Only edit it here — the wrapper substitutes it into the payload script (`mapCreationJob.sh`) automatically before every submission, same mechanism as `JOBNAME_TAG`. Don't hand-edit `MACRO_ALIEN_DIR_BARE`/`MACRO_ALIEN_DIR` inside the payload scripts themselves — it's a placeholder line there, not a real per-user setting. | + +**Physics/processing parameters live in a different file** — edit the config block in +**`mapCreationJob.sh`** itself (not the wrapper; these are not part of the wrapper's +placeholder-substitution mechanism, so a hand-edit here is a normal, permanent change, not something +that gets overwritten per submission): + +| Variable | Meaning | +|---|---| +| `Z2XBINNING`, `Y2XBINNING` | Voxel binning along z/x and y/x. | +| `USE_SMOOTHED` | Passed through to the macro's `useSmoothed`. | +| `CREATE_SPLINE` | Leave `false` — `TPCFastTransformInitCPM.C` is invoked separately in stage 2, not from this macro. | +| `MAX_TRACKS_PER_SLICE` / `MIN_TRACKS_PER_SLICE` | Per-voxel-slice track count cap / floor. Falling short of `MIN_TRACKS_PER_SLICE` is what triggers the `badCalib_` marking (see Step 3). | +| `MAX_DEDX`, `MAX_DEDX_EXP`, `MAX_DEV_DEDX_OVER_EXP` | dE/dx-based track rejection cuts (`-1` = disabled). | +| `SKIP_EDGE_PADS` | Skip edge pads in the calibration. | +| `BAD_RANGE_SELECTION` | Which bad-time-range comment tag(s) to apply (`"ALL"` = every entry in the bad-ranges list). | +| `MAX_Z2X_CUT` | Track `tgl` cut (`SpacePointsCalibConfParam scdcalib.maxZ2X`) — real production value is `1.4`. | +| `MAX_TRACK_WORKERS` | **Set this to match your GRID job's actual allocated CPU core count** (i.e. whatever `--packagespec`/JDL gives you, commonly matching a `multicore_N` tag). Auto-detection via `hardware_concurrency()` does *not* reliably reflect the real GRID allocation — confirmed auto-detecting 32 threads on an 8-core job (4x oversubscription) once. Getting this wrong doesn't fail the job, just wastes/starves CPU. | + +Note: the macro no longer reads IDC scalers from CCDB directly (`useCTPLumi`/`nScalerRanges`/`useScalerRange` +are gone from its signature) — luminosity values are joined back offline by `SmoothingExtrapolate.C` in +stage 2 instead (its own `USE_CTP_LUMI`, see Step 4). + +Then: + +```bash +./submitMapCreation.sh [listsDir] # listsDir defaults to ./lists +``` + +This builds + uploads the slot manifest (and, if `MACRO_SOURCE=alien`, re-stages the macro) and submits +a SINGLE `grid_submit.sh --prodsplit N` production, where N is the total number of windows across all +selected runs. There is no longer one submission per time division: a subjob index identifies a +(run, window) pair on its own, because the windows are baked into the manifest. + +**Local single-slot test before submitting to the GRID** (recommended — much faster feedback loop): + +```bash +SUBJOBID=1 ./mapCreationJob.sh +``` + +Runs subjob 1's slot locally using whatever manifest/macro is already staged on AliEn. + +### Step 3 — monitor and wait + +Watch progress on alimonitor (job status, logs, quota). + +**Where the output actually goes** (per ALICE Grid support): each slot's map is staged +directly to *that job's own* JDL-assigned output directory (`${ALIEN_JDL_OUTPUTDIR}`), **not** a fixed +path under your own AliEn home — writing multi-GB production data into personal space would need its +own separate disk-quota request on top of the CPU one, so this pipeline deliberately doesn't do that. +Concretely, for subjob N: + +``` +/-//voxRes.__.TDof.it0.root +``` + +That parent directory is only known once the job actually runs and is otherwise unrecoverable later +(it includes the real submission timestamp) — so `submitMapCreation.sh` captures it the +moment it submits and uploads it as a small pointer file, `stage1Workdir..txt`, to +your own AliEn space (negligible size — a single line of text, not the actual data). Stage 2 downloads +this pointer and reconstructs each slot's exact output location from it (see Step 4) — you shouldn't +need to look at it directly, but it's what makes stage 2 able to find stage 1's output at all. + +If a slot didn't reach the minimum track-count threshold, its map is staged with a `badCalib_` prefix +instead of the normal name — that's the real production's own convention, replicated here so stage 2 +knows to skip it (see below). This is expected behavior for low-statistics slots, not a failure. + +You do **not** need to wait for every stage-1 subjob to finish before moving on to stage 2 — see +"Partial failure" below. + +**Note**: there's no clean, human-browsable `/...`-style directory tree here (unlike the real +input data's own `/alice/data////...` catalog convention) — that catalog is +populated by a centrally-run, formally-registered production system with dedicated write access, not +something reachable via a generic `grid_submit.sh` submission like this one. If you want a tidy, +reorganized view of a run's output later, that needs a separate finalize/merge step (copying out of +these per-job locations into wherever makes sense) — deliberately not built as part of this pipeline. + +### Step 4 — configure and submit stage 2 (smoothing + FastTransform) + +Edit the config block at the top of **`submitPostProcess.sh`**: + +| Variable | Meaning | +|---|---| +| `JOBNAME` | This stage's *own* job name (just for GRID job monitoring) — **not** the same thing as `STAGE1_JOBNAME_TAG` below. | +| `STAGE1_JOBNAME_TAG` | **Must equal the `JOBNAME` stage 1 was actually submitted with** (Step 2). This is what stage 2 uses to find both the manifest and the `stage1Workdir..txt` pointer file that tells it where stage 1's own jobs staged their maps. A mismatch here doesn't corrupt anything — the wrapper fails loudly up front if the pointer file can't be fetched (see Step 4's submission output), rather than silently producing zero output. | +| `TOPWORKDIR` | Same meaning as stage 1's. Independent value — stage 2 is its own GRID submission with its own working directory, not nested under stage 1's. | +| `PACKAGESPEC` | Same meaning as stage 1; keep in sync. Also determines whether `MACRO_SOURCE=official` can find the macros. | +| `ASUSER` | Same meaning as stage 1's — independent of what stage 1 used (stage 2's own submission). | +| `MACRO_SOURCE` / `MACRO_LOCAL_DIR` | See "Where the macros come from" above (this stage uploads all three macros at once when `MACRO_SOURCE=alien`, via `MACRO_LOCAL_DIR` rather than a single-file path). | + +**Physics/processing parameters**, same as stage 1, live in the payload script +(**`postProcessJob.sh`**), not the wrapper: + +| Variable | Meaning | +|---|---| +| `USE_SMOOTHED` | Should match stage 1's `USE_SMOOTHED` — selects `DO_GAUS_SMOOTH` below. | +| `DO_GAUS_SMOOTH` | Whether `SmoothingExtrapolate.C` applies Gaussian smoothing. | +| `DO_EXTRAPOLATION` | `2` = extrapolate both smoothed and raw values. | +| `A11_MAX_Z2X` | `-1` = disabled. | +| `MASK_IA11` | `0` = disabled. | +| `USE_CTP_LUMI` | Selects `TPCFastTransformInitCPM.C`'s luminosity source. `2` = CTP (matches stage 1's macro not accessing IDC scalers directly at all -- see the note at the end of Step 2). | +| `QA_DRAW_ERRORS` | `voxResQA.C` option — draw error bars. | +| `QA_USE_SMOOTHED` | `voxResQA.C` option — `true` draws the smoothed dXS/dYS/dZS (needs `SmoothingExtrapolate.C`'s output, which is exactly what it runs on); `false` draws raw dX/dY/dZ instead. | +| `QA_Z2X_BIN_SEL` | `voxResQA.C` option — `-1` = all z2x bins. | + +**QA plots**: stage 2 also runs `voxResQA.C` on each slot's smoothed map, producing +`voxResQA_1D.root`/`_2D.root` (canvases) plus individual `.png` quick-look images. QA failure +is non-fatal — it won't fail an otherwise-good slot. + +Then: + +```bash +./submitPostProcess.sh +``` + +This downloads the *existing* manifest stage 1 already uploaded (does not rebuild it — see the comment in +the script for why that matters), re-stages the three macros if `MACRO_SOURCE=alien`, and submits one +production, same slot granularity as stage 1. + +**Local single-slot test:** + +```bash +SUBJOBID=1 ./postProcessJob.sh +``` + +Stage 2's final output (the smoothed map, `FT_*.root` FastTransform file, and the two `voxResQA*.root` +QA files) is staged the same way as stage 1's — directly to *this job's own* `${ALIEN_JDL_OUTPUTDIR}`, +not stage 1's. Since stage 2 is a completely separate GRID submission (its own `JOBNAME`, its own +submission timestamp), this is a **different directory tree** from stage 1's, even for the exact same +logical slot: + +``` +/-// + voxRes.__.TDof.it0.GausSmooth.Extrapolation.root + FT_voxRes.__.TDof.it0.GausSmooth.Extrapolation.root + voxResQA.voxRes.__.TDof.it0.GausSmooth.Extrapolation_1D.root + voxResQA.voxRes.__.TDof.it0.GausSmooth.Extrapolation_2D.root +``` + +Stage 1's and stage 2's output for the same slot are *not* grouped under one tidy parent directory — +that's the direct cost of letting the JDL own output placement instead of managing it ourselves (see +the note at the end of Step 3). Same disk-quota reasoning as stage 1: explicit, synchronous `alien.py cp` +straight there, not the passive `#JDL_OUTPUT@disk=N` auto-upload (which only happens during the job's +asynchronous SAVING phase, after this script has already exited). + +## Partial failure — this is by design, not a workaround + +- **Some stage-1 slots fail or haven't finished**: stage 2, for that slot, finds nothing at the + reconstructed `${ALIEN_JDL_OUTPUTDIR}` location, logs a warning, and exits 0 (success/no-op) — it does + not fail the whole production. Resubmit just the missing stage-1 slot(s) later, then resubmit the + corresponding stage-2 subjob(s). +- **A stage-1 slot is marked `badCalib_`** (too few tracks — see Step 3): stage 2 recognizes the + `badCalib_` prefix specifically and logs that the calibration was intentionally marked bad, rather + than the generic "not found" message. Also a no-op, not a failure. +- **Some stage-2 subjobs fail** (e.g. a transient GRID/network issue): only those need resubmitting — + stage 1's expensive output is untouched and doesn't need to be redone. + +## Known gotchas + +- **`STAGE1_JOBNAME_TAG` is the only thing stage 2 has to get right.** It selects both the manifest and the `stage1Workdir` pointer. There is no time-window arithmetic to keep in sync any more — both stages read each slot's window straight off the same manifest line. A wrong tag fails loudly at the pointer fetch; a *stale* pointer under a reused tag would still no-op silently, so keep `JOBNAME` unique per submission. +- **Don't hand-edit `STAGE1_OUTPUTDIR_BASE`/`COUNTERWIDTH`/`MACRO_ALIEN_DIR_BARE`/`BAD_RANGES_NAME` inside the payload scripts** — like `JOBNAME_TAG`, these are placeholder lines the wrapper substitutes automatically per submission, not real per-user settings. +- **A window split can, rarely, produce a half with zero real data** if the split point lands past the run's actual end-of-run (a residual file's nominal timestamp range can pad past the true last data). That slot correctly gets marked `badCalib_` — it's an expected edge case, not something to debug further; the macro now also fails fast on it instead of wasting the CPU-time quota downloading data it will discard. +- **AliEn `wc -l`-derived counts can come back whitespace-padded**, which `grid_submit.sh --prodsplit` rejects outright (`Production split must be a positive integer (got ' 70')`) — already fixed in the checked-in scripts, just noting it in case you copy this pattern elsewhere. +- **Job success rate around 80-90% is normal** for a first production pass — check individual failed-job logs on alimonitor before assuming something is systemically broken. diff --git a/DATA/production/configurations/mapCreationGrid/discoverResiduals.sh b/DATA/production/configurations/mapCreationGrid/discoverResiduals.sh new file mode 100755 index 000000000..b2d714c4b --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/discoverResiduals.sh @@ -0,0 +1,201 @@ +#!/usr/bin/env bash +# discoverResiduals.sh +# +# Runs locally. For each run in a batch file, calls makeResidualLists.sh +# with --alien to query AliEn for o2tpc_residuals*.root files and build +# per-period residual lists under lists//. +# +# Runs are processed in parallel using GNU parallel. +# +# Usage: +# ./discoverResiduals.sh [--calib-pass ] +# +# Batch file format (one run per line, whitespace-separated): +# +# e.g.: LHC26ak 572557 +# +# PERIOD is derived from the first field by stripping from the first '.' onward +# (e.g. LHC26ab.b5p → LHC26ab — a trailing '.xyz' suffix is tolerated but not +# required). +# YEAR is derived from the two-digit year in the period token +# (e.g. LHC26ab → 2026). +# +# Dependencies: alien.py, parallel, makeResidualLists.sh + +set -euo pipefail + +# --------------------------------------------------------------------------- +# Parse arguments +# --------------------------------------------------------------------------- +BATCH_FILE="" +CALIB_PASS="cpass2_residuals" + +while [[ $# -gt 0 ]]; do + case "$1" in + --calib-pass) + [[ $# -lt 2 ]] && { echo "ERROR: --calib-pass requires an argument"; exit 1; } + CALIB_PASS="$2"; shift 2 ;; + -*) + echo "ERROR: Unknown option: $1"; exit 1 ;; + *) + [[ -n "${BATCH_FILE}" ]] && { echo "ERROR: unexpected extra argument: $1"; exit 1; } + BATCH_FILE="$1"; shift ;; + esac +done + +[[ -z "${BATCH_FILE}" ]] && { echo "Usage: $0 [--calib-pass ]"; exit 1; } +[[ -f "${BATCH_FILE}" ]] || { echo "ERROR: batch file not found: ${BATCH_FILE}"; exit 1; } + +# --------------------------------------------------------------------------- +# Parse the batch file into one " " entry per line +# --------------------------------------------------------------------------- +# The period is resolved PER LINE, not once from the first line: the batch format carries a period +# token on every row, so a batch legitimately spanning two periods must not silently inherit the +# first row's period for all of its runs (that produced a valid-looking /alice/data//// query, which just finds nothing and degrades into a per-run warning). +mapfile -t ENTRIES < <(awk 'NF > 0 && !/^[[:space:]]*#/ {print $1, $2}' "${BATCH_FILE}") +[[ ${#ENTRIES[@]} -gt 0 ]] || { echo "ERROR: no runs found in ${BATCH_FILE}"; exit 1; } + +# Validate every entry up front -- a malformed row should stop the batch here, not surface as one +# failed run out of fifty after the parallel fan-out has already started. +RUNS=() +PERIODS_SEEN=() +for entry in "${ENTRIES[@]}"; do + read -r _tok _run <<< "${entry}" + [[ -n "${_tok}" && -n "${_run}" ]] \ + || { echo "ERROR: malformed line in ${BATCH_FILE} (need ' '): '${entry}'"; exit 1; } + [[ "${_run}" =~ ^[0-9]+$ ]] \ + || { echo "ERROR: run number is not numeric in ${BATCH_FILE}: '${entry}'"; exit 1; } + # Field 1 example: LHC26ab.b5p → PERIOD=LHC26ab, YEAR=2026 + [[ "${_tok%%.*}" =~ LHC([0-9]{2}) ]] \ + || { echo "ERROR: cannot extract year from period token '${_tok}' (line: '${entry}')"; exit 1; } + RUNS+=("${_run}") + PERIODS_SEEN+=("${_tok%%.*}") +done + +mapfile -t PERIODS_UNIQ < <(printf '%s\n' "${PERIODS_SEEN[@]}" | sort -u) + +# --------------------------------------------------------------------------- +# Fixed configuration +# --------------------------------------------------------------------------- +# Maximum number of runs to process in parallel (0 = one per CPU core) +N_PARALLEL=8 + +# Root of the output tree (relative or absolute) +WORK_DIR="./lists" + +# Directory containing this script +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Path to makeResidualLists.sh (co-located here) +MAKE_LISTS_SCRIPT="${SCRIPT_DIR}/makeResidualLists.sh" + +# --------------------------------------------------------------------------- + +echo "Batch file : ${BATCH_FILE}" +echo "Period(s) : ${PERIODS_UNIQ[*]}" +echo "Calib pass : ${CALIB_PASS}" +echo "Runs (${#RUNS[@]}): ${RUNS[*]}" +echo + +mkdir -p "${WORK_DIR}" + +# --------------------------------------------------------------------------- +# per_run — all work for a single run, called by parallel +# +# On success writes /run_completed.txt containing the absolute +# path of RUN_DIR so the parent shell can collect completed runs after +# parallel finishes. +# --------------------------------------------------------------------------- +per_run() { + local PERIOD_TOKEN="$1" # field 1 of this batch-file row + local RUN="$2" # field 2 of this batch-file row + local CALIB_PASS="$3" + local WORK_DIR="$4" + local MAKE_LISTS_SCRIPT="$5" + + # Period/year come from THIS row, so a multi-period batch resolves each run against its own period. + # Already validated in the caller's pre-flight loop, so a parse failure here would be a bug. + local PERIOD YEAR + PERIOD="${PERIOD_TOKEN%%.*}" # strip suffix after first '.' + [[ "${PERIOD}" =~ LHC([0-9]{2}) ]] + YEAR="20${BASH_REMATCH[1]}" + + local RUN_DIR="${WORK_DIR}/${RUN}" + mkdir -p "${RUN_DIR}" + + local ALIEN_PATH="/alice/data/${YEAR}/${PERIOD}/${RUN}/${CALIB_PASS}/" + + echo "========================================" + echo "Processing run ${RUN} (period ${PERIOD}, year ${YEAR})" + echo "========================================" + + echo " [${RUN}] Calling makeResidualLists.sh (alien path: ${ALIEN_PATH})..." + ( + cd "${RUN_DIR}" + run="${RUN}" "${MAKE_LISTS_SCRIPT}" --subdir "${CALIB_PASS}" --alien --alien-path "${ALIEN_PATH}" + ) || { + echo " [${RUN}] WARNING: makeResidualLists.sh failed, skipping." + return 0 + } + + echo " [${RUN}] Done." + + # Signal success to the parent by writing the absolute run dir path + realpath "${RUN_DIR}" > "${RUN_DIR}/run_completed.txt" +} + +# Export everything parallel needs to call per_run in a subshell +export -f per_run + +# --------------------------------------------------------------------------- +# Run all runs in parallel +# --------------------------------------------------------------------------- +echo "========================================" +echo "Launching ${#RUNS[@]} runs with N_PARALLEL=${N_PARALLEL}..." +echo "========================================" + +# Fan out over the full " " entries, not bare run numbers, so each job carries its +# own period. --colsep splits each entry into {1}=period token and {2}=run; note that under --colsep +# a bare {} means column 1 ONLY, so both columns must be passed explicitly. --tagstring uses {2} to +# keep the log prefix to just the run number. +# shellcheck disable=SC1083 # {1}/{2} are GNU parallel replacement strings, not brace expansions +parallel \ + --jobs "${N_PARALLEL}" \ + --line-buffer \ + --colsep ' ' \ + --tagstring "[{2}]" \ + per_run {1} {2} "${CALIB_PASS}" "${WORK_DIR}" "${MAKE_LISTS_SCRIPT}" \ + ::: "${ENTRIES[@]}" + +# --------------------------------------------------------------------------- +# Collect completed runs (sentinel files written by per_run on success) +# --------------------------------------------------------------------------- +COMPLETED_RUNS=() +for RUN in "${RUNS[@]}"; do + SENTINEL="${WORK_DIR}/${RUN}/run_completed.txt" + if [[ -f "${SENTINEL}" ]]; then + COMPLETED_RUNS+=("$(cat "${SENTINEL}")") + rm -f "${SENTINEL}" + fi +done + +echo "========================================" +echo "All runs processed. ${#COMPLETED_RUNS[@]}/${#RUNS[@]} succeeded." +echo "========================================" + +if (( ${#COMPLETED_RUNS[@]} == 0 )); then + echo "No runs completed successfully." + exit 0 +fi + +# Sort for deterministic output order. Must come AFTER the empty check: expanding "${arr[@]}" on an +# empty array is an "unbound variable" error under `set -u` in bash <= 4.3. +IFS=$'\n' COMPLETED_RUNS=($(printf '%s\n' "${COMPLETED_RUNS[@]}" | sort)) +unset IFS + +echo "Per-run slot-file indexes (input for stageSlotsForGrid.py):" +for RUN_DIR in "${COMPLETED_RUNS[@]}"; do + RUN="$(basename "${RUN_DIR}")" + echo " ${RUN_DIR}/${RUN}.residuals_lists.txt" +done diff --git a/DATA/production/configurations/mapCreationGrid/makeResidualLists.sh b/DATA/production/configurations/mapCreationGrid/makeResidualLists.sh new file mode 100755 index 000000000..cbdb6fd51 --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/makeResidualLists.sh @@ -0,0 +1,280 @@ +#!/usr/bin/env bash +# makeResidualLists.sh +# Find o2tpc_residuals*.root files (local or alien) and split them into +# per-period lists plus a combined all-runs list. +# +# Single-run usage: +# run= ./makeResidualLists.sh [--subdir ] [--alien] [--dry-run] +# +# Multi-run usage (loops over all run directories automatically): +# ./makeResidualLists.sh --all [--subdir ] [--alien] [--dry-run] +# +# Arguments: +# --all loop over all NNN*/ subdirs in the current directory +# --subdir subdirectory within each run dir to search in (default: cpass1_residuals) +# --alien use alien.py instead of local find +# --alien-path

explicit AliEn directory to search (overrides --year/--period/--subdir) +# --year year component of the AliEn path (required with --alien unless --alien-path) +# --period period component of the AliEn path (required with --alien unless --alien-path) +# --dry-run print what would be done without writing files +# In the normal chain you never pass --year/--period by hand: discoverResiduals.sh derives them per +# run from the batch file and passes a fully-formed --alien-path instead. +# +# Environment (single-run mode): +# run : run number (optional; used in output filenames) + +set -euo pipefail + +# ── colours ─────────────────────────────────────────────────────────────────── +RED='\033[0;31m'; YELLOW='\033[1;33m'; GREEN='\033[0;32m' +CYAN='\033[0;36m'; BOLD='\033[1m'; RESET='\033[0m' + + +# Safe exit: use return when sourced, exit when run directly +_finish() { local code=${1:-0}; [[ "${BASH_SOURCE[0]}" != "${0}" ]] && return "$code" || exit "$code"; } +error() { echo -e "${RED}[ERROR]${RESET} $*" >&2; _finish 1; } +warn() { echo -e "${YELLOW}[WARN]${RESET} $*"; } +info() { echo -e "${CYAN}[INFO]${RESET} $*"; } +ok() { echo -e "${GREEN}[OK]${RESET} $*"; } + +# ── settings ────────────────────────────────────────────────────────────────── +: ${run:=""} +USE_ALIEN=0 +DRY_RUN=0 +ALL_MODE=0 +SUBDIR="cpass1_residuals" +ALIEN_PATH_OVERRIDE="" +YEAR="" +PERIOD="" + +# ── parse arguments ─────────────────────────────────────────────────────────── +while [[ $# -gt 0 ]]; do + case "$1" in + --alien) USE_ALIEN=1 ; shift ;; + --dry-run) DRY_RUN=1 ; shift ;; + --all) ALL_MODE=1 ; shift ;; + --subdir) + [[ $# -lt 2 ]] && error "--subdir requires an argument" + SUBDIR="$2" ; shift 2 ;; + --alien-path) + [[ $# -lt 2 ]] && error "--alien-path requires an argument" + ALIEN_PATH_OVERRIDE="$2" ; shift 2 ;; + --year) + [[ $# -lt 2 ]] && error "--year requires an argument" + YEAR="$2" ; shift 2 ;; + --period) + [[ $# -lt 2 ]] && error "--period requires an argument" + PERIOD="$2" ; shift 2 ;; + *) error "Unknown argument: $1" ;; + esac +done + +# AliEn search path, e.g. /alice/data/2024/LHC24as/560334/cpass4_residuals/ +# +# There is deliberately NO default year/period here any more. There used to be a hardcoded +# YEAR=2026 / PERIOD=LHC26ab pair, which was harmless in the normal chain (discoverResiduals.sh always +# passes --alien-path, which overrides it) but became live in --all mode, where it silently queried +# LHC26ab for every run regardless of what the caller actually asked for. Requiring the caller to be +# explicit turns that into an up-front error instead of an empty result set. +ALIEN_PATH="" +if [[ -n "${ALIEN_PATH_OVERRIDE}" ]]; then + ALIEN_PATH="${ALIEN_PATH_OVERRIDE}" +elif [[ $USE_ALIEN -eq 1 ]]; then + [[ -n "$YEAR" && -n "$PERIOD" ]] \ + || error "--alien needs either --alien-path , or both --year and --period " + ALIEN_PATH="/alice/data/${YEAR}/${PERIOD}/${run}/${SUBDIR}/" +fi + +# ── multi-run mode ──────────────────────────────────────────────────────────── +# Re-invokes this script once per run directory, then collects all_list files. +if [[ $ALL_MODE -eq 1 ]]; then + SCRIPT="$(readlink -f "${BASH_SOURCE[0]}")" + TOP="$PWD" + + echo + echo -e "${BOLD}╔══════════════════════════════════════════════════════════════╗${RESET}" + echo -e "${BOLD}║ TPC Residual List Builder [multi-run mode] ║${RESET}" + echo -e "${BOLD}╚══════════════════════════════════════════════════════════════╝${RESET}" + echo + info "Base directory : $TOP" + info "Subdir : $SUBDIR" + [[ $DRY_RUN -eq 1 ]] && warn "DRY-RUN mode — no files will be written" + echo + + RUNS=( $(ls -d [0-9]*/ 2>/dev/null | sed 's|/$||') ) + [[ ${#RUNS[@]} -eq 0 ]] && error "No run directories (NNN*/) found in $TOP" + info "Found ${#RUNS[@]} run director(ies): ${RUNS[*]}" + echo + + FAILED=() + for r in "${RUNS[@]}"; do + WORK_DIR="$TOP/$r/$SUBDIR" + if [[ ! -d "$WORK_DIR" ]]; then + warn "Skipping run $r — subdir not found: $WORK_DIR" + FAILED+=("$r") + continue + fi + echo -e "${BOLD}════ Run $r ════${RESET}" + ( + cd "$WORK_DIR" + # Propagate every option that affects WHERE we look, not just the two mode flags. --subdir + # was previously dropped here: the parent used it to build WORK_DIR while the child fell + # back to the "cpass1_residuals" default, so the two disagreed about which calib pass was + # being listed. Array, not a string, so a value containing spaces cannot re-split. + EXTRA_FLAGS=("--subdir" "$SUBDIR") + [[ $USE_ALIEN -eq 1 ]] && EXTRA_FLAGS+=("--alien") + [[ $DRY_RUN -eq 1 ]] && EXTRA_FLAGS+=("--dry-run") + [[ -n "$YEAR" ]] && EXTRA_FLAGS+=("--year" "$YEAR") + [[ -n "$PERIOD" ]] && EXTRA_FLAGS+=("--period" "$PERIOD") + run="$r" bash "$SCRIPT" "${EXTRA_FLAGS[@]}" + ) || { warn "Run $r failed — continuing"; FAILED+=("$r"); } + echo + done + + # Collect the per-run index files back to the top directory + if [[ $DRY_RUN -eq 0 ]]; then + info "Collecting *.residuals_lists.txt → $TOP" + find "$TOP" -mindepth 3 -maxdepth 3 -name "*.residuals_lists.txt" \ + -exec cp {} "$TOP/" \; + else + info "[DRY-RUN] Would copy *.residuals_lists.txt to $TOP" + fi + + echo + if [[ ${#FAILED[@]} -gt 0 ]]; then + warn "Finished with ${#FAILED[@]} skipped/failed run(s): ${FAILED[*]}" + else + ok "All ${#RUNS[@]} run(s) processed successfully." + fi + _finish 0 +fi + +# ── single-run mode ─────────────────────────────────────────────────────────── +# Safety check: refuse to run in single-run mode from a directory that contains +# numeric run subdirectories — that almost certainly means --all was forgotten. +if [[ -z "$run" ]] && ls -d [0-9][0-9][0-9][0-9][0-9][0-9]*/ &>/dev/null; then + error "Current directory contains run subdirectories but no run number is set.\n" \ + " Did you mean: $0 --all\n" \ + " Or: run= $0" +fi + +FILE_ALL="residuals.all.txt" +RUN_PREFIX="${run:+${run}_}" + +echo +echo -e "${BOLD}╔══════════════════════════════════════════════════════════════╗${RESET}" +echo -e "${BOLD}║ TPC Residual List Builder ║${RESET}" +echo -e "${BOLD}╚══════════════════════════════════════════════════════════════╝${RESET}" +echo +[[ -n "$run" ]] && info "Run number : ${run}" || warn "No run number set — output filenames will omit run" +info "Working dir: $PWD" +info "Source : $([ $USE_ALIEN -eq 1 ] && echo "alien ($ALIEN_PATH)" || echo "local")" +[[ $DRY_RUN -eq 1 ]] && warn "DRY-RUN mode — no files will be written" +echo + +# ── clean up stale output files from previous run ───────────────────────────── +mapfile -t STALE < <(ls "$FILE_ALL" "residuals.${RUN_PREFIX}"*.txt \ + "${run:-NOMATCH}.residuals_lists.txt" \ + "${run:-NOMATCH}.residuals_all_list.txt" 2>/dev/null || true) + +if [[ ${#STALE[@]} -gt 0 ]]; then + warn "Removing ${#STALE[@]} stale file(s) from previous run:" + for f in "${STALE[@]}"; do + echo " rm $f" + [[ $DRY_RUN -eq 0 ]] && rm -f "$f" + done + echo +fi + +# ── collect residual file list ──────────────────────────────────────────────── +info "Searching for o2tpc_residuals*.root files..." + +if [[ $DRY_RUN -eq 1 ]]; then + if [[ $USE_ALIEN -eq 1 ]]; then + info "[DRY-RUN] Would run: alien.py find $ALIEN_PATH o2tpc_residuals*.root | sed 's|^|alien://|' > $FILE_ALL" + else + COUNT=$(find "$PWD" -name "o2tpc_residuals*.root" | wc -l) + info "[DRY-RUN] Would write $COUNT file path(s) to: $FILE_ALL" + find "$PWD" -name "o2tpc_residuals*.root" \ + | awk -F'/' '{print $NF";"$0}' | sort | awk -F';' '{print $NF}' \ + | head -5 | sed 's/^/ /' || true # `|| true`: head -5 SIGPIPEs the upstream find, + # which under pipefail would abort this dry run + [[ $COUNT -gt 5 ]] && echo " ... ($COUNT total)" + fi + echo + info "[DRY-RUN] Would rename to residuals.all.${RUN_PREFIX}___.txt" + info "[DRY-RUN] Would split into per-period lists residuals.${RUN_PREFIX}.txt" + info "[DRY-RUN] Would write index file ${run:-norunnumber}.residuals_lists.txt" + echo + ok "Dry-run complete — no files written." + _finish 0 +fi + +# ── real mode from here ─────────────────────────────────────────────────────── +if [[ $USE_ALIEN -eq 1 ]]; then + alien.py find "$ALIEN_PATH" o2tpc_residuals*.root \ + | sed 's|^|alien://|' > "$FILE_ALL" +else + find "$PWD" -name "o2tpc_residuals*.root" \ + | awk -F'/' '{print $NF";"$0}' \ + | sort \ + | awk -F';' '{print $NF}' \ + > "$FILE_ALL" +fi + +if [[ ! -s "$FILE_ALL" ]]; then + warn "No o2tpc_residuals*.root files found — all output lists will be empty." + _finish 0 +fi +ok "Found $(wc -l < "$FILE_ALL" | tr -d '[:space:]') file(s)." + +# ── extract timestamp range ─────────────────────────────────────────────────── +read -r firstTS1 lastTS1 firstTF1 lastTF1 <<< \ + "$(head -1 "$FILE_ALL" | sed -rn 's|.*_([0-9]{13})_([0-9]{13})_([0-9]+)_([0-9]+).*|\1 \2 \3 \4|p')" +read -r firstTS2 lastTS2 firstTF2 lastTF2 <<< \ + "$(tail -1 "$FILE_ALL" | sed -rn 's|.*_([0-9]{13})_([0-9]{13})_([0-9]+)_([0-9]+).*|\1 \2 \3 \4|p')" + +# ── MC fallback (no timestamps) ─────────────────────────────────────────────── +if [[ -z "${firstTS1:-}" || -z "${lastTS2:-}" ]]; then + warn "No timestamp info found — assuming MC input." + FILE_ALL_RANGE="residuals.all.${run}.txt" + mv "$FILE_ALL" "$FILE_ALL_RANGE" + ls "$PWD/${FILE_ALL_RANGE}" > "${run}.residuals_lists.txt" + ok "Done (MC mode)." + _finish 0 +fi + +# ── rename with timestamp range ─────────────────────────────────────────────── +FILE_ALL_RANGE="residuals.all.${RUN_PREFIX}${firstTS1}_${lastTS2}_${firstTF1}_${lastTF2}.txt" +info "Renaming: $FILE_ALL → $FILE_ALL_RANGE" +mv "$FILE_ALL" "$FILE_ALL_RANGE" + +# ── write the run's slot list ───────────────────────────────────────────────── +# ONE list per run, holding every residual file found. Slot granularity is deliberately NOT taken from +# the aggregation labels in the filenames -- it is chosen at submission time via SLOT_LENGTH_MIN, and +# mapCreationJob.sh prefilters each subjob's copy of this list down to the files that can actually +# contribute to its own time window. +# +# Why not group by label: each residual filename carries a slot label +# ___, and those labels are NOT a uniform grid -- same-length slots +# are not guaranteed, and a run's catalog can hold overlapping/nested labels at different granularities +# (why doesn't matter here, just that it happens), e.g. run 572266: +# 0_105425 (22 files) 105426_210851 (6) 0_210851 (35) +# 210852_316277 (2) 210852_421703 (45) +# Slicing per distinct label would give overlapping slots with wildly uneven statistics -- the 2- and +# 6-file ones trip minTracksPerSlice and come back badCalib_ while the data that would have fixed them +# sits in a sibling list. Keeping one list per run makes granularity a deliberate choice instead of an +# artefact of the upstream catalog's granularity, and keeps it uniform across runs. +# The run's slot list IS the all-files list -- there is exactly one per run, so it is not copied to a +# second name. Both index files below therefore point at this one file. +info "Slot list: $FILE_ALL_RANGE ($(wc -l < "$FILE_ALL_RANGE" | tr -d '[:space:]') file(s))" +info "Windowing happens at submission time (SLOT_LENGTH_MIN), not here -- this is the whole run." + +# ── write index files ───────────────────────────────────────────────────────── +# One index per run, holding the single path to that run's slot list. +LISTS_INDEX="${run:-norunnumber}.residuals_lists.txt" +ls "$PWD/${FILE_ALL_RANGE}" > "$LISTS_INDEX" + +echo +ok "Done. 1 slot list written." +ok "Slot list index : $LISTS_INDEX" \ No newline at end of file diff --git a/DATA/production/configurations/mapCreationGrid/mapCreationJob.sh b/DATA/production/configurations/mapCreationGrid/mapCreationJob.sh new file mode 100755 index 000000000..0e0d2c65c --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/mapCreationJob.sh @@ -0,0 +1,296 @@ +#!/usr/bin/env bash +# mapCreationJob.sh — TPC space-charge map creation, one time slot per GRID job. +# +# This is STAGE 1 of a two-stage pipeline: runs staticMapCreatorCPM.C directly to produce the raw +# per-slot voxRes.*.it0.root -- the heavy step, hours per slot. Stages its output to this job's own +# JDL-assigned ${ALIEN_JDL_OUTPUTDIR} (see below) so STAGE 2 (postProcessJob.sh: +# SmoothingExtrapolate.C + TPCFastTransformInitCPM.C, minutes per slot) can find it independently and +# be resubmitted on its own if it fails, without re-running this stage. Everything further downstream +# (iteration chains beyond it0, finalize/QA) stays local for now -- deliberately out of scope here. +# +# Submitted via: +# ${O2DPG_ROOT}/GRID/utils/grid_submit.sh --script mapCreationJob.sh \ +# --prodsplit --packagespec O2PDPSuite:: ... +# +# --prodsplit gives every subjob nothing but a plain integer index +# (ALIEN_O2DPG_GRIDSUBMIT_SUBJOBID, exported by grid_submit.sh -- real GRID only). This script +# downloads the mapSlots..tsv manifest that stageSlotsForGrid.py uploaded (one line +# per slot: "\t") -- tagged by JOBNAME_TAG +# so a different-named submission's manifest doesn't overwrite this one's (see JOBNAME_TAG below) -- +# picks its own line by index, and reconstructs a local slot-list file. --prodsplit is used instead of +# --split-on-collection: the latter needs one already-registered AliEn catalog file per split unit +# (N slow alien.py round trips, N tiny objects cluttering one AliEn directory), while a single manifest +# upload avoids both. +# +# Slot windowing: this script does NO time arithmetic. stageSlotsForGrid.py decides the windows at +# submission time (from SLOT_LENGTH_MIN) and writes one manifest line per window, carrying the window's +# bounds and exactly the files that can contribute to it. Stage 2 reads the same line, so the two +# stages cannot disagree about a slot's identity -- they used to each recompute it from +# TIME_DIVISION/TIME_DIVISION_INDEX, and any mismatch silently produced zero output. +# +# JOBNAME_TAG: tags the slot manifest name (mapSlots..tsv) so a different-named submission's +# manifest doesn't silently overwrite this one's. Substituted from the wrapper's own JOBNAME. +# +# MACRO_SOURCE: "official" (default) runs the macro straight from the O2 package already loaded by this +# job's --packagespec ($O2_ROOT/share/macro/staticMapCreatorCPM.C -- the same idiom +# DATA/production/configurations/asyncCalib/createCorrectionMap.sh already uses for staticMapCreator.C), +# no staging/download needed. Requires a packagespec built from an O2 tag that includes +# staticMapCreatorCPM.C (Detectors/TPC/calibration/SpacePoints/macro/) -- until such a tag is published, +# use MACRO_SOURCE=alien instead, which downloads a copy staged ahead of time under MACRO_ALIEN_DIR (see +# submitMapCreation.sh's MACRO_LOCAL_PATH). +# +# MACRO_ALIEN_DIR_BARE: same placeholder mechanism, substituted from the wrapper's own +# MACRO_ALIEN_DIR_BARE -- if you change that in submitMapCreation.sh (e.g. to point at a +# different AliEn user/directory), it now propagates here automatically instead of needing a second, +# easy-to-forget manual edit of this file. Used for the bad-time-ranges list and the slot manifest +# regardless of MACRO_SOURCE; only used for the macro itself when MACRO_SOURCE=alien. +JOBNAME_TAG=default +MACRO_SOURCE=official +MACRO_ALIEN_DIR_BARE=${MACRO_ALIEN_DIR_BARE:-} +# +# BAD_RANGES_NAME: filename of the bad-time-ranges list to pull from ${MACRO_ALIEN_DIR}/lists/. Same +# placeholder mechanism -- substituted from the wrapper's own BAD_RANGES_NAME, which is also what the +# wrapper uploads to when BAD_RANGES_LOCAL is set, so the uploaded name and the downloaded name are the +# same value by construction. The list is period/system-specific (the default below is PbPb26), so +# expect to change it in the wrapper when the data changes -- not here. Empty ("") is a supported value +# meaning "skip bad-ranges filtering entirely for this submission" -- see BAD_RANGES_ARG below. +BAD_RANGES_NAME=${BAD_RANGES_NAME:-Merge_BadIntervals_LHC26_PbPb_DCA-0P300.txt} +#JDL_OUTPUT=*.log@disk=1,*.txt@disk=1,*.info@disk=1 +#JDL_REQUIRE={member(other.GridPartitions,"multicore_8") && (other.CE != "ALICE::NIHAM::PBS64") && (other.CE != "ALICE::KISTI_GSDC::LCG") && (other.CE != "ALICE::SaoPaulo::LCG")}; +# Site blacklist: these three CEs excluded from this pipeline's jobs. Setting #JDL_REQUIRE overrides +# grid_submit.sh's own default Requirements entirely (it's read verbatim, not merged -- see +# grid_submit.sh's REQUIRESPEC handling), so the multicore_8 GridPartitions clause is repeated here +# rather than dropped. +# voxRes.*.root deliberately NOT in this pattern -- it's staged explicitly, once, via the synchronous +# alien.py cp below (to a fixed, predictable location stage 2 can find). Having it ALSO match here would +# make the automatic per-job upload duplicate the same (100+ MB) file a second time, to the ephemeral +# per-job dir, for no benefit now that the fixed-location copy is the reliable one. See "Two-stage +# pipeline" in README.md. + +set -e + +# xrootd client tuning -- retry/timeout behavior for alien://xrootd reads. Set before any AliEn +# transfer below (both our own alien.py cp calls and the macro's own residual-file reads go through +# this same client). Without it, a stalled read can hang until AliEn's own watchdog kills the payload +# for staying below its CPU-usage threshold too long (TJAlienFile "Operation expired"). +# +# CONNECTIONRETRY=2, CONNECTIONWINDOW=30: patiently retrying a sick storage server for many minutes is +# fine on a machine you own, but fatal on the GRID -- AliEn's job agent kills any payload whose CPU +# stays below threshold for 15 minutes, and a stalled bulk read burns that budget at ~0% CPU. A higher +# CONNECTIONRETRY would allow correspondingly longer silent reconnection per request before an error is +# ever raised; at 2 x 30s the worst case is 60 s, so the macro's own "could not load entry -> skip file" +# path (which handles a FAILED read perfectly well -- the problem is only ever that xrootd does not +# fail, it waits) gets a chance to run before the watchdog does. +# +# Note this is a mitigation, not a guarantee: none of these knobs bound a transfer that is trickling +# rather than dead, since each sub-request completes before its own timeout while the overall read +# crawls. That case is caught earlier instead, by the unhealthy-replica open-time gate in +# staticMapCreatorCPM.C (SCDCALIB_MAX_FILE_OPEN_SEC). +export XRD_TIMEOUTRESOLUTION=15 +export XRD_CONNECTIONWINDOW=30 +export XRD_CONNECTIONRETRY=2 +export XRD_REQUESTTIMEOUT=120 +export XRD_STREAMTIMEOUT=120 +export XRD_STREAMERRORWINDOW=0 + +# --- AliEn path holding the bad-time-ranges list / mapSlots.tsv (and staticMapCreatorCPM.C too, when +# MACRO_SOURCE=alien). grid_submit.sh only ships this one script; everything else is staged here once +# and pulled down per job. +# +# Normally substituted above by submitMapCreation.sh (which derives it from your AliEn +# account), so it is empty only when this script is run by hand -- in which case say so explicitly +# rather than building "alien:///mapSlots...tsv" and failing later with a confusing catalog error. +: "${MACRO_ALIEN_DIR_BARE:?MACRO_ALIEN_DIR_BARE is empty -- the submission wrapper normally substitutes it. For a standalone/local run, export it first, e.g. MACRO_ALIEN_DIR_BARE=/alice/cern.ch/user///MapCreation}" +MACRO_ALIEN_DIR="alien://${MACRO_ALIEN_DIR_BARE}" + +if [[ "${MACRO_SOURCE}" == "official" ]]; then + : "${O2_ROOT:?O2_ROOT is not set -- MACRO_SOURCE=official needs the O2 package loaded via --packagespec}" + MACRO_PATH="${O2_ROOT}/share/macro/staticMapCreatorCPM.C" + if [[ ! -f "${MACRO_PATH}" ]]; then + echo "ERROR: MACRO_SOURCE=official but ${MACRO_PATH} does not exist -- this job's --packagespec does not include staticMapCreatorCPM.C yet. Use MACRO_SOURCE=alien until a packagespec with it is published." >&2 + exit 1 + fi +else + alien.py cp -f "${MACRO_ALIEN_DIR}/staticMapCreatorCPM.C" file:./staticMapCreatorCPM.C + MACRO_PATH="./staticMapCreatorCPM.C" +fi +# BAD_RANGES_NAME="" is a deliberate, supported value -- see the wrapper's own comment on it -- meaning +# "skip bad-ranges filtering entirely" (staticMapCreatorCPM.C treats an empty badRangeList argument that +# way, see its `if (badRangeList.length() > 0)` guard). BAD_RANGES_ARG is what actually gets passed to +# the macro below; empty there, not "badRanges.dat", is what selects that mode. +BAD_RANGES_ARG="" +if [[ -n "${BAD_RANGES_NAME}" ]]; then + alien.py cp -f "${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME}" file:./badRanges.dat + BAD_RANGES_ARG="badRanges.dat" +fi +alien.py cp -f "${MACRO_ALIEN_DIR}/mapSlots.${JOBNAME_TAG}.tsv" file:./mapSlots.tsv + +# --- Determine this subjob's index into mapSlots.tsv --- +# ALIEN_O2DPG_GRIDSUBMIT_SUBJOBID is set by grid_submit.sh on the real GRID (--prodsplit mode) before +# it runs this script. For --local testing without it, pass SUBJOBID explicitly, e.g.: +# SUBJOBID=3 grid_submit.sh --local ... +SUBJOBID="${ALIEN_O2DPG_GRIDSUBMIT_SUBJOBID:-${SUBJOBID:-}}" +: "${SUBJOBID:?SUBJOBID must be set (via ALIEN_O2DPG_GRIDSUBMIT_SUBJOBID on GRID, or the SUBJOBID env var for --local testing)}" + +LINE=$(sed -n "${SUBJOBID}p" mapSlots.tsv) +if [[ -z "${LINE}" ]]; then + echo "ERROR: mapSlots.tsv has no line ${SUBJOBID}" >&2 + exit 1 +fi + +# Manifest line: \t \t \t \t \t +# The window bounds and the file selection are decided once, at submission time, by +# stageSlotsForGrid.py -- this script does no time arithmetic of its own. That is deliberate: stage 2 +# reads the SAME manifest line, so the two stages cannot disagree about a slot's identity (they +# previously each recomputed it and a mismatch silently produced zero output). +IFS=$'\t' read -r RUN WIN_IDX WIN_COUNT firstTSIter lastTSIter PATHS <<< "${LINE}" +for v in RUN WIN_IDX WIN_COUNT firstTSIter lastTSIter; do + if [[ ! "${!v}" =~ ^[0-9]+$ ]]; then + echo "ERROR: manifest line ${SUBJOBID} field ${v}='${!v}' is not numeric -- expected 6 tab-separated fields " >&2 + exit 1 + fi +done +[[ -n "${PATHS}" ]] || { echo "ERROR: manifest line ${SUBJOBID} carries no file paths" >&2; exit 1; } + +SLOT_LIST="slotlist_${SUBJOBID}.txt" +tr ',' '\n' <<< "${PATHS}" > "${SLOT_LIST}" +nSlotFiles=$(wc -l < "${SLOT_LIST}" | tr -d '[:space:]') # tr: wc right-justifies its count +echo "Processing slot ${SUBJOBID}: run ${RUN}, window ${WIN_IDX}/${WIN_COUNT} [${firstTSIter},${lastTSIter}] ($(( (lastTSIter - firstTSIter) / 1000 ))s), ${nSlotFiles} file(s)" + +# --- Process the files in the order that reaches this window's data soonest --- +# WHY: the macro's per-file skip still has to OPEN every out-of-window file to read its records tree -- +# ~7 s each over alien://, at ~zero CPU -- and AliEn's job agent kills any payload whose CPU stays +# below threshold for 15 minutes (observed for real, run 572246: 47 files skipped in 6 min, killed +# before reaching in-window data). +# +# stageSlotsForGrid.py already dropped every file whose aggregation label cannot overlap this window, +# but that filter can only be as sharp as the labels are. A coarse label produced by the aggregator +# merging an underpopulated slot forward can span several windows, so overlapping this window by as +# little as 1 ms pulls in all of its files. Ordering fixes what filtering cannot: each file's CTF +# directory carries an _orbit field, which IS that file's true position in time, so we sort by +# distance from this window's centre. The files that can actually contribute are opened first and +# maxTracksPerSlice usually stops the job before it ever reaches the far ones. +# +# Only the ORDER depends on this, never which files are considered, so the orbit->time estimate below +# is allowed to be approximate: it is interpolated from the list's own extremes. +if awk -F'_orbit' 'NF < 2 {exit 1}' "${SLOT_LIST}"; then + awk -F'_orbit' -v lo="${firstTSIter}" -v hi="${lastTSIter}" ' + { + line[NR] = $0 + # orbit from the CTF directory name + split($2, o, "_"); orb[NR] = o[1] + 0 + # Time anchor: the EXTREMES of the aggregation labels present, not their midpoints. In nearly + # every window all files carry a single label (one aggregator slot spans several windows), and + # identical midpoints collapse the interpolation to frac=0 -- which sorts ascending no matter + # where the window actually sits, the exact order that gets a late window killed by the CPU + # watchdog. Extremes stay meaningful with one label: [labelFirst,labelLast] maps onto + # [oMin,oMax]. + if (match($0, /o2tpc_residuals_[0-9]{13}_[0-9]{13}_/)) { + split(substr($0, RSTART + 16, 27), t, "_") + if (!seen || (t[1] + 0) < mMin) mMin = t[1] + 0 + if (!seen || (t[2] + 0) > mMax) mMax = t[2] + 0 + seen = 1 + } + if (n == 0 || orb[NR] < oMin) oMin = orb[NR] + if (n == 0 || orb[NR] > oMax) oMax = orb[NR] + n = NR + } + END { + centre = (lo + hi) / 2 + # map the window centre onto the orbit axis using this list own extremes + frac = (seen && mMax > mMin) ? (centre - mMin) / (mMax - mMin) : 0 + if (frac < 0) frac = 0 + if (frac > 1) frac = 1 + target = oMin + frac * (oMax - oMin) + for (i = 1; i <= n; i++) { + d = orb[i] - target; if (d < 0) d = -d + printf "%d\t%s\n", d, line[i] + } + }' "${SLOT_LIST}" \ + | sort -n -k1,1 | cut -f2- > "${SLOT_LIST}.ord" + if [[ -s "${SLOT_LIST}.ord" ]]; then + mv "${SLOT_LIST}.ord" "${SLOT_LIST}" + echo "File order: sorted by orbit distance from the window centre (closest first)" + else + rm -f "${SLOT_LIST}.ord" + echo "WARNING: could not order ${SLOT_LIST} by orbit -- leaving the file order untouched" >&2 + fi +else + # No _orbit field in at least one path -- cannot establish a time order, so leave the list alone + # rather than reorder it on a key that is not there. Costs the optimization, never correctness. + echo "WARNING: not every path in ${SLOT_LIST} carries an _orbit field -- leaving the file order untouched (no time-ordering possible)" >&2 +fi + +# --- Config (production values; createSpline is the one deliberate exception, see CREATE_SPLINE +# below: TPCFastTransformInitCPM.C runs separately in stage 2 here instead) --- +# NOTE: nScalerRanges/useScalerRange/useCTPLumi are gone from the macro's signature. It no longer reads +# IDC scalers from CCDB at all; the values are joined in offline by SmoothingExtrapolate.C in stage 2. +Z2XBINNING="28" +Y2XBINNING="20" +USE_SMOOTHED=2 +CREATE_SPLINE=false # TPCFastTransformInitCPM.C runs in stage 2 instead; leave false here +MAX_TRACKS_PER_SLICE=10000 +MIN_TRACKS_PER_SLICE=3000 +MAX_DEDX=-1 +MAX_DEDX_EXP=-1 +MAX_DEV_DEDX_OVER_EXP=-1 +SKIP_EDGE_PADS=1 +BAD_RANGE_SELECTION="ALL" +MAX_Z2X_CUT=1.4 # SpacePointsCalibConfParam scdcalib.maxZ2X override -- real + # production value, vs. the struct's own compiled-in default of 1.0 +MAX_TRACK_WORKERS=8 # matches the JDL's CPUCores/multicore_8 allocation (grid_submit.sh's + # default) -- hardware_concurrency() does NOT reliably reflect the + # real GRID allocation (confirmed 2026-07-25: auto-detected 32 on an + # 8-core job, 4x oversubscription), so pass the known real value. + +OUTPUT_STEM="voxRes.${RUN}_${firstTSIter}_${lastTSIter}.TD${WIN_IDX}of${WIN_COUNT}.it0" +OUTPUT_FILE="${OUTPUT_STEM}.root" + +export alien_CLOSE_SE="ALICE::CERN::EOS" +cmd="(time root.exe -b -q -l -x '${MACRO_PATH}+O(\"${SLOT_LIST}\", ${RUN}, \"${OUTPUT_FILE}\", \"\", \"ITS-TPC\", \"${Z2XBINNING}\", \"${Y2XBINNING}\", ${USE_SMOOTHED}, ${CREATE_SPLINE}, ${MAX_TRACKS_PER_SLICE}, ${MIN_TRACKS_PER_SLICE}, \"${BAD_RANGES_ARG}\", ${firstTSIter}, ${lastTSIter}, ${MAX_DEDX}, ${MAX_DEDX_EXP}, ${MAX_DEV_DEDX_OVER_EXP}, ${SKIP_EDGE_PADS}, \"${BAD_RANGE_SELECTION}\", ${MAX_Z2X_CUT}, ${MAX_TRACK_WORKERS})')" +echo "$(date) ${cmd}" +eval "${cmd}" + +if [[ ! -s "${OUTPUT_FILE}" ]]; then + echo "ERROR: ${OUTPUT_FILE} was not produced -- this slot's map creation failed" >&2 + exit 1 +fi + +# Bad-calib handling: the macro creates an extensionless badCalib. marker file when +# minTracksPerSlice wasn't reached. Stage under a badCalib_-prefixed +# name instead of the normal one -- stage 2, looking for the normal (unprefixed) name, simply won't +# find it and will hit the same graceful-skip path already used for "stage 1 hasn't produced this yet", +# with no extra logic needed there. The marker file itself has no extension and matches none of our +# JDL output patterns, so it doesn't survive on its own -- this rename is what actually carries the +# bad-calib status forward. +STAGE_NAME="${OUTPUT_FILE}" +if [[ -f "badCalib.${OUTPUT_STEM}" ]]; then + STAGE_NAME="badCalib_${OUTPUT_FILE}" + echo "Calibration marked bad for this slot (found badCalib.${OUTPUT_STEM}) -- staging as ${STAGE_NAME} instead of ${OUTPUT_FILE}" +fi + +# --- Two-stage pipeline: stage the produced map to THIS job's own JDL-assigned output directory +# (per ALICE Grid support -- do not write production data to a personal-space path; that needs its own +# disk-quota request on top of the CPU one, unlike ${ALIEN_JDL_OUTPUTDIR}, which is already covered by +# however this job's quota/ownership is set up). ALIEN_JDL_OUTPUTDIR is JAliEn's own per-JDL-field env +# var (case-sensitive, same convention as ALIEN_JDL_CPUCORES etc.); fall back to ALIEN_JOB_OUTPUTDIR +# (grid_submit.sh's own equivalent, computed the same way from the same JDL field -- see that script) in +# case the former isn't exported in some execution context. Explicit, synchronous cp rather than relying +# on the passive #JDL_OUTPUT@disk=N auto-upload -- that only happens during the job's asynchronous +# SAVING phase after this script exits, which can take minutes and would race against a later reader +# (stage 2 doesn't try to read this immediately, but no reason to introduce that fragility when an +# explicit cp is just as easy and finishes before this script does). +# +# NOTE: this directory is only known once the job actually runs -- see +# submitMapCreation.sh for how the post-processing stage (postProcessJob.sh, +# SmoothingExtrapolate.C + TPCFastTransformInitCPM.C) finds it later without needing an in-job breadcrumb. +ALIEN_OUTPUT_DIR="alien://${ALIEN_JDL_OUTPUTDIR:-${ALIEN_JOB_OUTPUTDIR:-}}" +if [[ "${ALIEN_OUTPUT_DIR}" == "alien://" ]]; then + echo "ERROR: neither ALIEN_JDL_OUTPUTDIR nor ALIEN_JOB_OUTPUTDIR is set -- don't know where to stage ${OUTPUT_FILE}" >&2 + exit 1 +fi +if ! alien.py cp -f "file:${OUTPUT_FILE}" "${ALIEN_OUTPUT_DIR}/${STAGE_NAME}"; then + echo "ERROR: failed to stage ${OUTPUT_FILE} to ${ALIEN_OUTPUT_DIR} -- the post-processing stage won't be able to find this slot's map" >&2 + exit 1 +fi +echo "Staged ${OUTPUT_FILE} to ${ALIEN_OUTPUT_DIR}/${STAGE_NAME}" diff --git a/DATA/production/configurations/mapCreationGrid/postProcessJob.sh b/DATA/production/configurations/mapCreationGrid/postProcessJob.sh new file mode 100755 index 000000000..676bcf459 --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/postProcessJob.sh @@ -0,0 +1,281 @@ +#!/usr/bin/env bash +# postProcessJob.sh — STAGE 2 of the two-stage pipeline. +# +# Runs the fast (minutes, not hours) part of the chain on an already-produced map: +# SmoothingExtrapolate.C -> TPCFastTransformInitCPM.C +# +# Independent of stage 1 (mapCreationJob.sh): does NOT re-run map creation, does NOT even download the +# (multi-GB) residual files. It only needs the small manifest (mapSlots.tsv, reused as-is from stage 1, +# NOT rebuilt) to work out which map stage 1 should have produced for this same subjob index. Both +# stages read the run/window straight off the SAME manifest line, so they cannot disagree about a +# slot's identity. +# +# If stage 1 hasn't produced (or failed to produce) this slot's map yet, this script logs a warning +# and exits 0 (success/no-op) rather than failing -- so a partial stage-1 completion (some slots ok, +# some failed) doesn't turn every stage-2 subjob into a failure. Once stage 1 is resubmitted for the +# missing slots, just resubmit the corresponding stage-2 subjob(s). +# +# Slot windowing: none happens here. stageSlotsForGrid.py fixed the windows at submission time and +# wrote them into the manifest; this script just reads its line. +# +# STAGE1_JOBNAME_TAG: which stage-1 production's manifest/output to look for -- NOT this stage's own job +# name, but stage 1's JOBNAME_TAG (mapCreationJob.sh), since that's what determined the manifest +# name AND (via the STAGE1_OUTPUTDIR_BASE pointer file it uploaded) where stage 1's own jobs staged their +# maps. Must match exactly, or this will always hit the graceful-skip path even though stage 1 succeeded. +# +# STAGE1_OUTPUTDIR_BASE / COUNTERWIDTH (per ALICE Grid support): stage 1 stages its map to its own +# ${ALIEN_JDL_OUTPUTDIR} rather than a fixed personal-space path (avoids a separate disk-quota request), +# so there's no static location to just look up. STAGE1_OUTPUTDIR_BASE is stage 1's MY_JOBWORKDIR (that +# submission's timestamped parent directory, captured by submitMapCreation.sh right when it +# ran grid_submit.sh -- otherwise unrecoverable later) and COUNTERWIDTH is grid_submit.sh's own +# alien_counter zero-padding width -- together they let this script reconstruct stage 1's exact per-slot +# ${ALIEN_JDL_OUTPUTDIR} the same way grid_submit.sh itself derives OutputDir (see below), without any +# in-job breadcrumb from stage 1 or an extra AliEn round trip. +# +# MACRO_SOURCE: same meaning and default ("official") as stage 1 -- see mapCreationJob.sh. Applies +# to all three macros used here (SmoothingExtrapolate.C, TPCFastTransformInitCPM.C, voxResQA.C), all +# installed to the same $O2_ROOT/share/macro/ once the O2 PR is merged into this job's packagespec. +# +# MACRO_ALIEN_DIR_BARE: same placeholder mechanism, substituted from the wrapper's own +# MACRO_ALIEN_DIR_BARE -- must match stage 1's. Used for the manifest/stage1Workdir pointer regardless of +# MACRO_SOURCE; only used for the macros themselves when MACRO_SOURCE=alien. +STAGE1_JOBNAME_TAG=default +STAGE1_OUTPUTDIR_BASE= +COUNTERWIDTH=3 +MACRO_SOURCE=official +MACRO_ALIEN_DIR_BARE=${MACRO_ALIEN_DIR_BARE:-} +#JDL_OUTPUT=*.log@disk=1,*.txt@disk=1,*.png@disk=1 +#JDL_REQUIRE={member(other.GridPartitions,"multicore_8") && (other.CE != "ALICE::NIHAM::PBS64") && (other.CE != "ALICE::KISTI_GSDC::LCG") && (other.CE != "ALICE::SaoPaulo::LCG")}; +# Site blacklist -- same three CEs excluded as stage 1, see that script's comment for why the +# GridPartitions clause has to be repeated here (grid_submit.sh replaces its whole default Requirements, +# doesn't merge with it). +# voxRes.*.root/FT_*.root/voxResQA*.root deliberately NOT in this pattern -- same reasoning as stage 1 +# (mapCreationJob.sh): they're staged explicitly, once, to this job's own ${ALIEN_JDL_OUTPUTDIR} +# below, so having them ALSO match here would duplicate the upload to the same place for no benefit. +# The QA step's individual *.png canvases ARE left in this pattern -- they're small, convenience-only +# quick-look images, not worth a second explicit-staging code path for. + +set -e + +# xrootd client tuning -- see mapCreationJob.sh for the reasoning; same values, same rationale, +# applies equally here since this script does just as much alien://xrootd transfer (macros, manifest, +# the fixed-location map download) even though it doesn't touch the multi-GB residual files. +export XRD_TIMEOUTRESOLUTION=15 +export XRD_CONNECTIONWINDOW=30 +export XRD_CONNECTIONRETRY=4 +export XRD_REQUESTTIMEOUT=120 +export XRD_STREAMTIMEOUT=120 +export XRD_STREAMERRORWINDOW=0 + +# --- AliEn path holding mapSlots.tsv / the stage1Workdir pointer file (and the three macros too, when +# MACRO_SOURCE=alien). Same root dir as stage 1's MACRO_ALIEN_DIR. +# +# Normally substituted above by submitPostProcess.sh; empty only on a standalone/local run -- see the +# same guard in mapCreationJob.sh. +: "${MACRO_ALIEN_DIR_BARE:?MACRO_ALIEN_DIR_BARE is empty -- the submission wrapper normally substitutes it. For a standalone/local run, export it first, e.g. MACRO_ALIEN_DIR_BARE=/alice/cern.ch/user///MapCreation}" +MACRO_ALIEN_DIR="alien://${MACRO_ALIEN_DIR_BARE}" + +if [[ "${MACRO_SOURCE}" == "official" ]]; then + : "${O2_ROOT:?O2_ROOT is not set -- MACRO_SOURCE=official needs the O2 package loaded via --packagespec}" + SMOOTHING_MACRO_PATH="${O2_ROOT}/share/macro/SmoothingExtrapolate.C" + FASTTRANSFORM_MACRO_PATH="${O2_ROOT}/share/macro/TPCFastTransformInitCPM.C" + QA_MACRO_PATH="${O2_ROOT}/share/macro/voxResQA.C" + for f in "${SMOOTHING_MACRO_PATH}" "${FASTTRANSFORM_MACRO_PATH}" "${QA_MACRO_PATH}"; do + if [[ ! -f "${f}" ]]; then + echo "ERROR: MACRO_SOURCE=official but ${f} does not exist -- this job's --packagespec does not include it yet. Use MACRO_SOURCE=alien until a packagespec with it is published." >&2 + exit 1 + fi + done +else + alien.py cp -f "${MACRO_ALIEN_DIR}/SmoothingExtrapolate.C" file:./SmoothingExtrapolate.C + alien.py cp -f "${MACRO_ALIEN_DIR}/TPCFastTransformInitCPM.C" file:./TPCFastTransformInitCPM.C + alien.py cp -f "${MACRO_ALIEN_DIR}/voxResQA.C" file:./voxResQA.C + SMOOTHING_MACRO_PATH="./SmoothingExtrapolate.C" + FASTTRANSFORM_MACRO_PATH="./TPCFastTransformInitCPM.C" + QA_MACRO_PATH="./voxResQA.C" +fi +# Tagged manifest name -- must match whatever stage 1 actually uploaded (mapSlots..tsv). +alien.py cp -f "${MACRO_ALIEN_DIR}/mapSlots.${STAGE1_JOBNAME_TAG}.tsv" file:./mapSlots.tsv + +# --- Determine this subjob's index into mapSlots.tsv (same convention as stage 1) --- +SUBJOBID="${ALIEN_O2DPG_GRIDSUBMIT_SUBJOBID:-${SUBJOBID:-}}" +: "${SUBJOBID:?SUBJOBID must be set (via ALIEN_O2DPG_GRIDSUBMIT_SUBJOBID on GRID, or the SUBJOBID env var for --local testing)}" + +LINE=$(sed -n "${SUBJOBID}p" mapSlots.tsv) +if [[ -z "${LINE}" ]]; then + echo "ERROR: mapSlots.tsv has no line ${SUBJOBID}" >&2 + exit 1 +fi + +# Manifest line: \t \t \t \t \t +# Stage 2 does NO time arithmetic: it reads the window straight off the same manifest line stage 1 +# used, so OUTPUT_STEM below is identical to stage 1's by construction. Previously both stages +# recomputed the window from TIME_DIVISION/TIME_DIVISION_INDEX and any disagreement meant stage 2 +# silently found nothing and no-opped for every slot. +# The 6th field (the file list) is deliberately discarded: stage 2 never touches the residual +# files, it only needs the run and the window. It still has to be READ, or lastTSIter would +# swallow the rest of the line. +IFS=$'\t' read -r RUN WIN_IDX WIN_COUNT firstTSIter lastTSIter _ <<< "${LINE}" +for v in RUN WIN_IDX WIN_COUNT firstTSIter lastTSIter; do + if [[ ! "${!v}" =~ ^[0-9]+$ ]]; then + echo "ERROR: manifest line ${SUBJOBID} field ${v}='${!v}' is not numeric -- expected 6 tab-separated fields " >&2 + exit 1 + fi +done +echo "Slot ${SUBJOBID}: run ${RUN}, window ${WIN_IDX}/${WIN_COUNT} [${firstTSIter},${lastTSIter}]" + +# --- Config (production defaults) --- +USE_SMOOTHED=2 # matches stage 1's USE_SMOOTHED -- selects doGausSmooth=true below +DO_GAUS_SMOOTH=true +DO_EXTRAPOLATION=2 # smoothed AND raw values extrapolated +A11_MAX_Z2X=-1 # disabled +MASK_IA11=0 # disabled +USE_CTP_LUMI=2 # matches stage 1's USE_CTP_LUMI + +# voxResQA.C config -- needs SmoothingExtrapolate.C's DS[] (smoothed dX/dY/dZ) already filled in, +# hence run from here, on OUTPUT_MAP_SMOOTH, not the raw stage-1 map. +QA_DRAW_ERRORS=false +QA_USE_SMOOTHED=true # true: draw dXS/dYS/dZS (needs SmoothingExtrapolate.C's output); false: raw dX/dY/dZ +QA_Z2X_BIN_SEL=-1 # -1: all z2x bins + +# --- Expected stage-1 output filename (must match mapCreationJob.sh's OUTPUT_STEM exactly) --- +OUTPUT_STEM="voxRes.${RUN}_${firstTSIter}_${lastTSIter}.TD${WIN_IDX}of${WIN_COUNT}.it0" +STAGE1_OUTPUT_FILE="${OUTPUT_STEM}.root" + +# Reconstruct stage 1's exact per-slot ${ALIEN_JDL_OUTPUTDIR} -- same formula grid_submit.sh itself uses +# to derive OutputDir (MY_JOBWORKDIR/#alien_counter_0Wi#): STAGE1_OUTPUTDIR_BASE is stage 1's +# MY_JOBWORKDIR, and this slot's own subjob index (SUBJOBID, same manifest line/counter convention as +# stage 1) zero-padded to COUNTERWIDTH gives the exact subdirectory stage 1's own job for this slot wrote +# its output to. +SUBJOB_COUNTER_DIR=$(printf "%0${COUNTERWIDTH}d" "${SUBJOBID}") +STAGE1_OUTPUT_LOCATION="alien://${STAGE1_OUTPUTDIR_BASE}/${SUBJOB_COUNTER_DIR}" + +# Local name deliberately does NOT look like our own output (voxRes.*.root/FT_*.root) -- both are now +# staged explicitly below rather than picked up via a JDL_OUTPUT glob, but naming this distinctly still +# avoids any confusion between "the map we downloaded as input" and "the map we produced" (the exact bug +# found+fixed earlier for badRanges.txt/badRanges.dat). +INPUT_MAP="input_map.root" + +echo "Looking for stage-1 output at ${STAGE1_OUTPUT_LOCATION}/${STAGE1_OUTPUT_FILE}" +if ! alien.py cp -f "${STAGE1_OUTPUT_LOCATION}/${STAGE1_OUTPUT_FILE}" "file:${INPUT_MAP}" || [[ ! -s "${INPUT_MAP}" ]]; then + rm -f "${INPUT_MAP}" + # Not there under the normal name -- check whether stage 1 actually ran but renamed it away because + # the calibration was marked bad (same convention as the real production, see mapCreationJob.sh), + # rather than assuming it just hasn't run yet. + if alien.py cp -f "${STAGE1_OUTPUT_LOCATION}/badCalib_${STAGE1_OUTPUT_FILE}" "file:${INPUT_MAP}" 2>/dev/null && [[ -s "${INPUT_MAP}" ]]; then + echo "Stage 1 marked this slot's calibration as bad (found badCalib_${STAGE1_OUTPUT_FILE} instead of ${STAGE1_OUTPUT_FILE}) -- skipping post-processing on purpose, not an error." >&2 + else + echo "WARNING: stage-1 output '${STAGE1_OUTPUT_FILE}' not found at ${STAGE1_OUTPUT_LOCATION} -- this slot's map creation likely hasn't run yet or failed. Skipping (not an error) -- resubmit this subjob once stage 1 has produced it." >&2 + fi + rm -f "${INPUT_MAP}" + exit 0 +fi + +# --- [1/2] Gaussian smoothing + extrapolation --- +GAUSSMOOTH_ADD="" +[[ "${DO_GAUS_SMOOTH}" == "true" ]] && GAUSSMOOTH_ADD+=".GausSmooth" +[[ "${DO_EXTRAPOLATION}" -gt 0 ]] && GAUSSMOOTH_ADD+=".Extrapolation" +OUTPUT_MAP_SMOOTH="${OUTPUT_STEM}${GAUSSMOOTH_ADD}.root" + +cmd="(time root.exe -b -q -l -x '${SMOOTHING_MACRO_PATH}+Og(\"${INPUT_MAP}\", \"${OUTPUT_MAP_SMOOTH}\", ${DO_GAUS_SMOOTH}, ${DO_EXTRAPOLATION}, ${A11_MAX_Z2X}, ${MASK_IA11})')" +echo "$(date) ${cmd}" +eval "${cmd}" + +if [[ ! -s "${OUTPUT_MAP_SMOOTH}" ]]; then + echo "ERROR: ${OUTPUT_MAP_SMOOTH} was not produced -- SmoothingExtrapolate.C failed" >&2 + exit 1 +fi + +# --- Resolve this job's own JDL-assigned output directory (per ALICE Grid support -- see +# mapCreationJob.sh for the full reasoning: production data belongs in ${ALIEN_JDL_OUTPUTDIR}, +# not a personal-space path). ALIEN_JDL_OUTPUTDIR is JAliEn's own per-JDL-field env var; fall back to +# ALIEN_JOB_OUTPUTDIR (grid_submit.sh's equivalent, computed from the same JDL field). +ALIEN_OUTPUT_DIR="alien://${ALIEN_JDL_OUTPUTDIR:-${ALIEN_JOB_OUTPUTDIR:-}}" +if [[ "${ALIEN_OUTPUT_DIR}" == "alien://" ]]; then + echo "ERROR: neither ALIEN_JDL_OUTPUTDIR nor ALIEN_JOB_OUTPUTDIR is set -- don't know where to stage this slot's output" >&2 + exit 1 +fi + +# Stage the smoothed map NOW, before the QA and FastTransform steps rather than after them. Those steps +# can fail hard on input this script cannot control -- notably TPCFastTransformInitCPM.C's +# LOGP(fatal, "meanCTP ... not set!"), which fires for any slot whose stage-1 map carries meanCTP=0 in +# its UserInfo (useCTPLumi=2 exempts meanIDC from that check, but NOT meanCTP). Staging last meant such +# a slot lost its perfectly good smoothed map as well, forcing a resubmit to redo the smoothing. +if ! alien.py cp -f "file:${OUTPUT_MAP_SMOOTH}" "${ALIEN_OUTPUT_DIR}/${OUTPUT_MAP_SMOOTH}"; then + echo "ERROR: failed to stage ${OUTPUT_MAP_SMOOTH} to ${ALIEN_OUTPUT_DIR}" >&2 + exit 1 +fi +echo "Staged ${OUTPUT_MAP_SMOOTH} to ${ALIEN_OUTPUT_DIR}" + +# --- Provenance pointer: unlike stage 1 -> stage 2 (which has the stage1Workdir..txt pointer), +# stage 2's own output directory otherwise carries no trace of which exact stage-1 output it was built +# from. Record it as a one-line pointer, same idea, staged right alongside the smoothed map. +STAGE1_SOURCE_FILE="stage1Source.txt" +echo "${STAGE1_OUTPUT_LOCATION}/${STAGE1_OUTPUT_FILE}" > "${STAGE1_SOURCE_FILE}" +if ! alien.py cp -f "file:${STAGE1_SOURCE_FILE}" "${ALIEN_OUTPUT_DIR}/${STAGE1_SOURCE_FILE}"; then + echo "WARNING: failed to stage ${STAGE1_SOURCE_FILE} -- provenance pointer only, not fatal" >&2 +fi + +# --- QA plots (voxResQA.C) -- runs on just this one slot's smoothed map, same per-slot granularity as +# everything else in this pipeline. Non-fatal on failure: QA is a diagnostic nice-to-have, not a blocking +# deliverable like the smoothed map/FastTransform output -- a QA hiccup shouldn't fail an otherwise-good +# slot. +QA_SUFFIX=".${OUTPUT_STEM}${GAUSSMOOTH_ADD}" +cmd="(time root.exe -b -q -l -x '${QA_MACRO_PATH}+Og(\"ls ${OUTPUT_MAP_SMOOTH}\", \"${QA_SUFFIX}\", ${QA_DRAW_ERRORS}, ${QA_USE_SMOOTHED}, ${QA_Z2X_BIN_SEL})')" +echo "$(date) ${cmd}" +eval "${cmd}" || echo "WARNING: voxResQA.C failed for this slot -- QA plots will be missing, but this doesn't affect ${OUTPUT_MAP_SMOOTH} itself" >&2 + +QA_FILE_1D="voxResQA${QA_SUFFIX}_1D.root" +QA_FILE_2D="voxResQA${QA_SUFFIX}_2D.root" + +# --- [2/2] Final O2 FastTransform --- +OUTPUT_FT="FT_${OUTPUT_MAP_SMOOTH}" + +cmd="(time root.exe -b -q -l -x '${FASTTRANSFORM_MACRO_PATH}+Og(\"${OUTPUT_MAP_SMOOTH}\", \"${OUTPUT_FT}\", ${USE_SMOOTHED}, ${USE_CTP_LUMI})')" +echo "$(date) ${cmd}" +eval "${cmd}" + +if [[ ! -s "${OUTPUT_FT}" ]]; then + echo "ERROR: ${OUTPUT_FT} was not produced -- TPCFastTransformInitCPM.C failed" >&2 + exit 1 +fi + +# --- Stage the FastTransform output to this job's own JDL-assigned output directory (the smoothed map +# was already staged above, right after it was produced). Explicit, synchronous cp rather than relying on +# the passive #JDL_OUTPUT@disk=N auto-upload, which only happens during the job's asynchronous SAVING +# phase after this script exits. +if ! alien.py cp -f "file:${OUTPUT_FT}" "${ALIEN_OUTPUT_DIR}/${OUTPUT_FT}"; then + echo "ERROR: failed to stage ${OUTPUT_FT} to ${ALIEN_OUTPUT_DIR}" >&2 + exit 1 +fi +echo "Staged ${OUTPUT_FT} to ${ALIEN_OUTPUT_DIR}" + +# QA plots -- non-fatal staging, same reasoning as the non-fatal QA run itself above: if voxResQA.C +# failed or produced nothing, there's nothing to stage, and that shouldn't fail this otherwise-good slot. +for qaFile in "${QA_FILE_1D}" "${QA_FILE_2D}"; do + if [[ -s "${qaFile}" ]]; then + alien.py cp -f "file:${qaFile}" "${ALIEN_OUTPUT_DIR}/${qaFile}" || \ + echo "WARNING: failed to stage ${qaFile} -- non-fatal, just loses this slot's QA plots" >&2 + else + echo "WARNING: ${qaFile} was not produced -- voxResQA.C likely failed above, skipping its staging" >&2 + fi +done + +# TPCFastTransformInitCPM.C's own debug/summary trees (produced unconditionally alongside OUTPUT_FT, +# named after its stem -- see createFastTransform() in that macro) were never staged anywhere: the +# #JDL_OUTPUT pattern above only matches *.log/*.txt/*.png (no *.root), and the explicit staging block +# above only names OUTPUT_MAP_SMOOTH/OUTPUT_FT -- so these were produced on the worker and then silently +# discarded when the job finished. Non-fatal staging, same reasoning as the QA plots: diagnostic, not a +# blocking deliverable. +OUTPUT_FT_DEBUG="${OUTPUT_FT%.root}.debug.root" +OUTPUT_FT_SUMMARY="${OUTPUT_FT%.root}.summary.root" +for ftDebugFile in "${OUTPUT_FT_DEBUG}" "${OUTPUT_FT_SUMMARY}"; do + if [[ -s "${ftDebugFile}" ]]; then + alien.py cp -f "file:${ftDebugFile}" "${ALIEN_OUTPUT_DIR}/${ftDebugFile}" || \ + echo "WARNING: failed to stage ${ftDebugFile} -- non-fatal, just loses this slot's FT debug/summary tree" >&2 + else + echo "WARNING: ${ftDebugFile} was not produced -- skipping its staging" >&2 + fi +done + +echo "Stage 2 complete: ${OUTPUT_MAP_SMOOTH}, ${OUTPUT_FT}" diff --git a/DATA/production/configurations/mapCreationGrid/stageSlotsForGrid.py b/DATA/production/configurations/mapCreationGrid/stageSlotsForGrid.py new file mode 100755 index 000000000..eae34838d --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/stageSlotsForGrid.py @@ -0,0 +1,340 @@ +#!/usr/bin/env python3 +"""Build and upload the slot manifest for the TPC map-creation GRID pipeline. + +Bridge between the local Step-1 list discovery (discoverResiduals.sh) and GRID submission. + +Cuts each run into time windows of ~--slot-length-min and writes ONE manifest line per window: + + \t \t \t \t \t + +paired with `grid_submit.sh --prodsplit `. Every subjob gets nothing but a plain integer +index (ALIEN_O2DPG_GRIDSUBMIT_SUBJOBID), downloads this one manifest and picks its own line, so a +subjob index identifies a (run, window) pair on its own. Both job scripts read the window straight off +that line, which is why they cannot disagree about a slot's identity. + +Why the windows are not simply the aggregation labels +----------------------------------------------------- +Each residual filename carries the slot label ___ that its reco +job assigned, and those labels are NOT a uniform grid -- same-length slots are not guaranteed, and a +run's catalog can hold overlapping/nested labels at different granularities (why doesn't matter here, +just that it happens). Slicing per label gives overlapping slots with wildly uneven statistics. +Choosing the window length here instead makes granularity a deliberate decision, uniform across runs. + +A file is assigned to a window when its label overlaps it by MORE than --overlap-margin-ms. A bare +overlap test is too generous at the edges: coarse labels routinely begin 1-2 ms before a window ends, +which would drag all of their files into a window whose map they contribute nothing to. The bound is +exact -- if a label overlaps by X ms then at most X ms of that file's data can belong to the window, +and the macro's own per-TF cut discards the rest regardless -- so skipping at X <= margin costs at +most `margin` ms of coverage per window edge. + +This was a bash+awk script. It moved to Python because the work is arithmetic over 13-digit +millisecond timestamps and every bug it produced was a property of the shell rather than of the +problem: `(( x += 0 ))` returning 1 under `set -e`, `sed | sort | head` aborting the script by SIGPIPE +under `pipefail`, `<<<` herestrings writing temp files onto Lustre, awk's %d truncating a millisecond +timestamp through a 32-bit int on mawk, and awk holding one output file open per window. Python has +arbitrary-precision ints and none of those failure modes. + +Prints the number of manifest lines (windows) to stdout on its own -- pass that as --prodsplit N to +grid_submit.sh. All other output goes to stderr, so `N=$(./stageSlotsForGrid.py ...)` captures just the +count. +""" + +import argparse +import os +import re +import shutil +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +# o2tpc_residuals____.root, anchored: the timestamps are always +# exactly 13 digits (ResidualAggregator.cxx formats them from the slot's start/end in ms). +SIG_RE = re.compile(r"o2tpc_residuals_(\d{13})_(\d{13})_\d+_\d+\.root$") + + +def log(msg): + print(msg, file=sys.stderr) + + +def parse_args(argv): + p = argparse.ArgumentParser( + description="Build and upload the TPC map-creation slot manifest.", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + p.add_argument("--lists-dir", default="./lists", + help="root lists dir (discoverResiduals.sh's WORK_DIR), containing one / " + "subdir per run, each with a .residuals_lists.txt index") + p.add_argument("--alien-dir", default="", + help="AliEn directory to upload the manifest into. Ignored with --preview.") + p.add_argument("--manifest-name", default="mapSlots.tsv", + help="manifest filename. Tag it per submission (mapSlots..tsv) so " + "concurrent productions do not overwrite each other's.") + p.add_argument("--runs", default="", + help="comma-separated run whitelist. Empty means EVERY run under --lists-dir, " + "including ones left from an earlier batch, which would be recalibrated from " + "scratch at hours per slot. Naming a run that has no lists is an error.") + p.add_argument("--slot-length-min", type=float, default=0, + help="target length of ONE map, in minutes. 0 = one window per run.") + p.add_argument("--overlap-margin-ms", type=int, default=1000, + help="a file whose label overlaps a window by <= this is not assigned to it") + p.add_argument("--window-lists", action="store_true", + help="also write one plain file list per window, for inspecting which files a " + "window got. OFF by default: on Lustre each is a metadata round trip and 120 " + "of them cost ~33 s, against ~0.03 s for all the actual work. The manifest " + "already holds every window in full, one line each.") + p.add_argument("--window-dir", default="", + help="where --window-lists writes. Default: .windows. Point it at " + "local disk (e.g. /tmp/windows) on a parallel filesystem.") + p.add_argument("--timing", action="store_true", default=bool(os.environ.get("STAGE_TIMING")), + help="print a read/compute/write breakdown to stderr (also via STAGE_TIMING=1)") + p.add_argument("--preview", action="store_true", + default=bool(os.environ.get("PREVIEW_ONLY")), + help="write the per-window lists but do not upload (also via PREVIEW_ONLY=1)") + return p.parse_args(argv) + + +def read_slot_files(index_path): + """The per-run index lists the file(s) holding that run's residual paths (normally exactly one).""" + out = [] + for line in index_path.read_text().splitlines(): + line = line.strip() + if line: + out.append(Path(line)) + return out + + +def plan_windows(run_first, run_last, slot_ms, margin_ms): + """Whole steps of slot_ms across [run_first, run_last]. + + A trailing remainder shorter than half a step is absorbed into the last window rather than left as + a runt that would fail minTracksPerSlice. Returns [(winFirst, winLast, effectiveMargin)]. + """ + span = run_last - run_first + if slot_ms <= 0 or span <= 0: + n_win = 1 + else: + n_win = max(1, span // slot_ms) + if (span - n_win * slot_ms) * 2 >= slot_ms: + n_win += 1 + + wins = [] + for w in range(n_win): + if n_win == 1: + ws, we = run_first, run_last + else: + ws = run_first + w * slot_ms + we = run_last if w + 1 == n_win else run_first + (w + 1) * slot_ms - 1 + # Clamp the margin to a tenth of the window so a short window can never be emptied by it. + wins.append((ws, we, min(margin_ms, (we - ws) // 10))) + return wins + + +def assign(paths, wins, run_first, slot_ms): + """Fan every file out to the windows its label overlaps by more than that window's margin. + + Only the windows a label can possibly touch are examined, derived directly from the label bounds, + rather than scanning all of them per file: with 10-minute labels and 5-minute windows that is 2-3 + candidates instead of the whole run's worth. Returns (buckets, grazed, orphans). + """ + n_win = len(wins) + buckets = [[] for _ in range(n_win)] + grazed = [0] * n_win + orphans = [] + + for lo, hi, line in paths: + if slot_ms > 0 and n_win > 1: + # Windows are contiguous steps of slot_ms from run_first, so a label maps to an index + # range. +/-1 guards the boundaries and the last window, which extends to run_last. + first = max(0, (lo - run_first) // slot_ms - 1) + last = min(n_win - 1, (hi - run_first) // slot_ms + 1) + else: + first, last = 0, n_win - 1 + + hit = False + for i in range(first, last + 1): + ws, we, margin = wins[i] + if hi < ws or lo > we: + continue + if min(hi, we) - max(lo, ws) > margin: + buckets[i].append(line) + hit = True + else: + grazed[i] += 1 + if not hit: + orphans.append(line) + + return buckets, grazed, orphans + + +def main(argv): + args = parse_args(argv) + + if not args.preview and not args.alien_dir: + log("ERROR: --alien-dir is required unless --preview is given") + return 1 + + lists_dir = Path(args.lists_dir) + index_files = sorted(lists_dir.glob("*/*.residuals_lists.txt")) + if not index_files: + log(f"ERROR: no *.residuals_lists.txt files found under {lists_dir}") + return 1 + + slot_ms = int(round(args.slot_length_min * 60_000)) + wanted = [r for r in args.runs.split(",") if r] if args.runs else [] + + # Rebuilt from scratch each time so it always reflects THIS invocation's slot length and margin + # rather than a stale earlier split. + window_dir = Path(args.window_dir) if args.window_dir else \ + Path(Path(args.manifest_name).with_suffix("").name + ".windows") + t_rm = time.monotonic() + if args.window_lists: + if window_dir.exists(): + shutil.rmtree(window_dir) + window_dir.mkdir(parents=True) + t_rm = time.monotonic() - t_rm + t_read = t_calc = t_write = 0.0 + + manifest_lines = [] + runs_included = [] + empty_runs = [] + n_missing = 0 + n_orphan_total = 0 + + for idx in index_files: + run = idx.name[: -len(".residuals_lists.txt")] + if wanted and run not in wanted: + log(f"Run {run}: not in runs filter, skipping") + continue + runs_included.append(run) + log(f"Run {run}: adding slots from {idx.name}") + + n_slots_run = 0 + for slot_file in read_slot_files(idx): + if not slot_file.is_file(): + log(f" WARNING: missing local file {slot_file}, skipping") + n_missing += 1 + continue + + _t = time.monotonic() + _text = slot_file.read_text() + t_read += time.monotonic() - _t + _t = time.monotonic() + parsed = [] + for line in _text.splitlines(): + m = SIG_RE.search(line) + if m: + parsed.append((int(m.group(1)), int(m.group(2)), line)) + if not parsed: + log(f" WARNING: no parseable residual signature in {slot_file}, skipping") + continue + + t_calc += time.monotonic() - _t + _t = time.monotonic() + run_first = min(p[0] for p in parsed) + run_last = max(p[1] for p in parsed) + wins = plan_windows(run_first, run_last, slot_ms, args.overlap_margin_ms) + buckets, grazed, orphans = assign(parsed, wins, run_first, slot_ms) + n_win = len(wins) + t_calc += time.monotonic() - _t + + for w, ((ws, we, _), files) in enumerate(zip(wins, buckets)): + if grazed[w]: + log(f" (skipped {grazed[w]} file(s) grazing this window)") + if not files: + log(f" WARNING: run {run} window {w}/{n_win} [{ws},{we}] has no files, skipping") + continue + if args.window_lists: + _t = time.monotonic() + out = window_dir / f"{run}.TD{w}of{n_win}.{ws}_{we}.txt" + out.write_text("\n".join(files) + "\n") + t_write += time.monotonic() - _t + manifest_lines.append(f"{run}\t{w}\t{n_win}\t{ws}\t{we}\t{','.join(files)}") + log(f" window {w}/{n_win} [{ws},{we}] {(we - ws) // 1000}s {len(files)} file(s)") + n_slots_run += 1 + + # Safety net for the overlap margin: every file must still land in at least one window. It + # always should -- an aggregation label is minutes long, so it overlaps some window by far + # more than the margin -- but a file in no window at all is silent data loss. + if orphans: + log(f" WARNING: {len(orphans)} file(s) of run {run} landed in NO window -- " + f"their data is not in any map.") + log(f" --overlap-margin-ms ({args.overlap_margin_ms}) is too large for a " + f"slot length of {args.slot_length_min} min.") + for o in orphans[:3]: + log(f" {o}") + n_orphan_total += len(orphans) + + log(f" {n_slots_run} slot(s) added for run {run}") + if n_slots_run == 0: + empty_runs.append(run) + + # A run named in --runs but absent from --lists-dir is a typo or a missed discovery step, not a + # request for a smaller production. + missing_runs = [r for r in wanted if r not in runs_included] + if missing_runs: + log(f"ERROR: run(s) requested in --runs have no *.residuals_lists.txt under {lists_dir}: " + f"{' '.join(missing_runs)}") + log(" Run discoverResiduals.sh for them first, or drop them from --runs.") + return 1 + + if n_missing or empty_runs or n_orphan_total: + log("") + log(" ****************************************************************") + log(" ** INCOMPLETE INPUT -- this manifest is smaller than expected **") + if n_missing: + log(f" ** {n_missing} slot file(s) referenced by an index but missing on disk") + if empty_runs: + log(f" ** run(s) contributing 0 slots: {' '.join(empty_runs)}") + if n_orphan_total: + log(f" ** {n_orphan_total} file(s) landed in no window -- lower --overlap-margin-ms") + log(" ** -> re-run discoverResiduals.sh for those runs, or accept") + log(" ** that they are simply absent from this production.") + log(" ****************************************************************") + log("") + + manifest_text = "".join(l + "\n" for l in manifest_lines) + # The manifest is always written locally, as a single file -- it is the complete record of the + # split and costs one create. + _t = time.monotonic() + Path(args.manifest_name).write_text(manifest_text) + t_write += time.monotonic() - _t + n_total = len(manifest_lines) + + if args.timing: + log(f"timing: clear={t_rm:.2f}s read={t_read:.2f}s compute={t_calc:.2f}s " + f"write={t_write:.2f}s ({n_total} window file(s))") + + if args.preview: + log("PREVIEW_ONLY set -- manifest NOT uploaded.") + log(f"Manifest written to: {args.manifest_name} ({n_total} window(s))") + if args.window_lists: + log(f"Per-window file lists in: {window_dir}/") + else: + log("Pass --window-lists to also dump one file list per window " + "(slow on Lustre; add --window-dir /tmp/... )") + print(n_total) + return 0 + + with tempfile.NamedTemporaryFile("w", suffix=".tsv", delete=False) as fh: + fh.write(manifest_text) + tmp_manifest = fh.name + try: + dest = f"alien://{args.alien_dir.rstrip('/')}/{args.manifest_name}" + subprocess.run(["alien.py", "cp", "-f", f"file:{tmp_manifest}", dest], + check=True, stdout=sys.stderr.fileno()) + log(f"Manifest uploaded: {dest} ({n_total} slot(s) across {len(runs_included)} run(s))") + finally: + os.unlink(tmp_manifest) + + log(f"Manifest also written locally to: {args.manifest_name}") + if args.window_lists: + log(f"Per-window file lists in: {window_dir}/") + log(f"Runs in this manifest: {' '.join(runs_included)}") + print(n_total) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/DATA/production/configurations/mapCreationGrid/submitMapCreation.sh b/DATA/production/configurations/mapCreationGrid/submitMapCreation.sh new file mode 100755 index 000000000..72d80cc6b --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/submitMapCreation.sh @@ -0,0 +1,222 @@ +#!/usr/bin/env bash +# submitMapCreation.sh +# +# Submits one grid_submit.sh production-split job, one subjob per time-window slot found in +# (built beforehand by discoverResiduals.sh -- run that first). +# +# Before each submission, this re-stages the slot manifest (mapSlots.tsv, built here via +# stageSlotsForGrid.py) -- and, when MACRO_SOURCE=alien, the macro (staticMapCreatorCPM.C) too -- to +# the same AliEn dir mapCreationJob.sh downloads from. Both must be fresh: a stale macro, or +# stageSlotsForGrid.py pointed at a different destination than the job script expects, would silently +# desync what a running job downloads from what this wrapper thinks it uploaded. The bad-time-ranges list +# is treated differently -- see BAD_RANGES_LOCAL below -- it's occasionally-updated reference data, not +# something edited every session like the macro, so forcing a fresh re-upload on every submission is +# unnecessary friction. +# +# Slot windowing: stageSlotsForGrid.py cuts each run into windows of ~SLOT_LENGTH_MIN and writes ONE +# manifest line per window, carrying the window bounds and exactly the files that can contribute to it. +# A subjob index therefore identifies a (run, window) pair on its own, so this is a SINGLE submission -- +# no loop over TimeDivisionIndex, no TIME_DIVISION placeholders substituted into the payload, and one +# stage1Workdir pointer instead of one per index. (With a fixed window length, runs of different length +# produce different window counts, so the old "one production per index x prodsplit runs" rectangle +# does not exist any more.) Both stages read the window off the same manifest line, which removes the +# old failure mode where stage 2 recomputed it and any mismatch silently produced zero output. +# +# Output location (2026-07-26, per ALICE Grid support): the actual map data (voxRes.*.root) is staged by +# mapCreationJob.sh directly to ${ALIEN_JDL_OUTPUTDIR} (the JDL-assigned per-subjob output dir -- +# see that script), NOT to a fixed personal-space path -- avoids a separate disk-quota request on top of +# the CPU one, and is the standard mechanism support asked us to use. That directory is only known once +# the job actually runs, and its parent (MY_JOBWORKDIR, timestamped at THIS submission) is otherwise +# unrecoverable later -- grid_submit.sh prints it once, right here, before any GRID execution starts: +# pok "Your job's working directory will be $MY_JOBWORKDIR" +# So this wrapper captures that one line, once per submission, and uploads it as a small pointer file +# (stage1Workdir..txt, negligible size, fine to keep in personal space) -- submitPostProcess.sh +# downloads it and reconstructs each slot's exact +# ${ALIEN_JDL_OUTPUTDIR} from MY_JOBWORKDIR + the same alien_counter formula grid_submit.sh itself uses +# (see that script), without needing any in-job breadcrumb or querying AliEn after the fact. +# +# ASUSER / quota: setting ASUSER="pwg_pp" charges this job's CPU usage to that working group instead of +# the personal account, per ALICE Grid support's instruction (the account must already be a member of +# that group). grid_submit.sh's --asuser sets MY_USER=${ASUSER}, which determines BOTH the role active +# when its own internal transaction issues `submit` (so JAliEn attributes the job to that role) AND +# MY_HOMEDIR="/alice/cern.ch/user/${MY_USER:0:1}/${MY_USER}" -- so OutputDir automatically lands under +# that group's own writable space, with zero extra redirection needed. MACRO_ALIEN_DIR_BARE (the small +# manifest/macro staging area this wrapper itself uploads to, read-only from the job's point of view) +# stays fixed under the submitting account regardless of ASUSER -- read access there is unaffected by +# role. +# +# Usage: +# ./submitMapCreation.sh [listsDir] +# (listsDir defaults to ./lists -- discoverResiduals.sh's WORK_DIR, run from wherever you ran that +# script) + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LISTS_DIR="${1:-./lists}" + +# --------------------------------------------------------------------------- +# Config -- edit per submission +# --------------------------------------------------------------------------- +JOBNAME="mapcreation-v1" # must be unique per submission -- bump the suffix for each new production +TOPWORKDIR="MapCreation" # replaces grid_submit.sh's generic default "selfjobs" bucket (on top of $HOME) + # with something identifiable -- matches MACRO_ALIEN_DIR_BARE's own naming + # below. Still just a static string per submission, not a per-run/period path + # like the real input data's own convention (.../2026/LHC26ak/572244/...) -- + # that needs a real post-processing/organize step, deliberately deferred to a + # later iteration of this pipeline. +PACKAGESPEC="O2::daily-20260729-0000-1" +ASUSER="pwg_pp" # your own account = personal quota. "pwg_pp" = charged to that working group's quota AND + # moves MY_JOBWORKDIR/OutputDir to pwg_pp's own space automatically -- see the + # long comment above for why this one value now does everything. NOTE support + # hedged on whether pwg_pp behaves like the other major accounts -- confirm on + # alimonitor that usage is really charged to the group before a large batch. +MACRO_SOURCE="official" # "official" (default): GRID jobs run staticMapCreatorCPM.C straight from the + # O2 package loaded via PACKAGESPEC ($O2_ROOT/share/macro/), no macro upload + # needed here at all -- requires PACKAGESPEC to resolve to a tag built from an O2 + # version that includes it (Detectors/TPC/calibration/SpacePoints/macro/). "alien": + # stage MACRO_LOCAL_PATH to AliEn and have jobs download it instead -- use this + # until such a tag is published. +MACRO_LOCAL_PATH="" # only used when MACRO_SOURCE=alien: local path to staticMapCreatorCPM.C to + # upload, e.g. your O2 checkout's + # Detectors/TPC/calibration/SpacePoints/macro/staticMapCreatorCPM.C. + # Staging area for the manifest + stage1Workdir pointer (and the macro, when + # MACRO_SOURCE=alien). No alien:// prefix. Derived from your own AliEn account the + # same way grid_submit.sh derives MY_HOMEDIR (ALIEN_USER, else the local + # username), so this file needs no per-user edit. Stays under the SUBMITTING + # account regardless of ASUSER -- read access there is unaffected by role, see the + # comment above. If your laptop's local username differs from your AliEn account + # (common), export ALIEN_USER, or just set MAPCREATION_ALIEN_DIR outright. +ALIEN_USER_NAME="${ALIEN_USER:-$(whoami)}" +MACRO_ALIEN_DIR_BARE="${MAPCREATION_ALIEN_DIR:-/alice/cern.ch/user/${ALIEN_USER_NAME:0:1}/${ALIEN_USER_NAME}/MapCreation}" +BAD_RANGES_NAME="Merge_BadIntervals_LHC26_PbPb_DCA-0P300.txt" + # Filename of the bad-time-ranges list, under ${MACRO_ALIEN_DIR}/lists/. Period- + # and system-specific (this one is PbPb26), so it changes when the data does -- + # hence a variable rather than a constant baked into three places. Substituted + # into mapCreationJob.sh the same way as JOBNAME_TAG, so + # set it only here. Used both as the upload destination (when BAD_RANGES_LOCAL is + # set) and as the name each job downloads -- the two cannot drift apart. + # Set to "" to disable bad-ranges filtering entirely for this submission + # (staticMapCreatorCPM.C treats an empty badRangeList argument that way) -- a + # deliberate opt-out, not the same as forgetting to set it. +BAD_RANGES_LOCAL="" # optional: local path to the bad-time-ranges file to (re-)upload this run (real + # source for the PbPb26 list above: + # /lustre/alice/tpcdata/Run3/SCDprodTests/lists/badRange/PbPb26/). Leave empty to + # skip the upload and just use the copy already at + # ${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME}. +RUNS_FILTER="" # optional comma-separated run whitelist, e.g. "572557,572558". Empty = every run + # found under listsDir. discoverResiduals.sh appends to the same ./lists tree + # across batches, so leaving this empty after a second discovery round silently + # folds the EARLIER batch's runs into this submission too -- recalibrating them + # from scratch at hours per slot. Set it when ./lists holds more than you want. + # A run named here with no lists under listsDir is a hard error. +OVERLAP_MARGIN_MS=1000 # a file is assigned to a window only if its aggregation label overlaps that + # window by MORE than this. Coarse labels routinely begin 1-2 ms before a window + # ends, and a bare overlap test drags all of their files into a window whose map + # they contribute nothing to. Bounded cost: at most this many ms of coverage per + # window edge, since the macro's per-TF cut discards the rest anyway. +SLOT_LENGTH_MIN=5 # target length of ONE map, in minutes. Each run is cut into whole steps of this + # length; a trailing remainder shorter than half a step is absorbed into the last + # window rather than left as a runt that would fail MIN_TRACKS_PER_SLICE. 0 means + # one map per run (no splitting). This replaces the old TIME_DIVISION count: with + # a fixed length, a 20 min run gives 4 maps and a 60 min run gives 12, instead of + # the same count regardless of run length. Stage 2 does not need this value -- it + # reuses stage 1's manifest, which already carries the windows. +# --------------------------------------------------------------------------- + +MACRO_ALIEN_DIR="alien://${MACRO_ALIEN_DIR_BARE}" +echo "== Staging area (manifest + stage1Workdir pointer): ${MACRO_ALIEN_DIR_BARE} ==" +echo " (derived from AliEn user '${ALIEN_USER_NAME}' -- export ALIEN_USER or MAPCREATION_ALIEN_DIR if that is wrong)" + +if [[ "${MACRO_SOURCE}" == "alien" ]]; then + [[ -n "${MACRO_LOCAL_PATH}" && -f "${MACRO_LOCAL_PATH}" ]] || { echo "ERROR: MACRO_SOURCE=alien needs MACRO_LOCAL_PATH set to a real staticMapCreatorCPM.C (got '${MACRO_LOCAL_PATH}')" >&2; exit 1; } + echo "== Re-staging macro from ${MACRO_LOCAL_PATH} to ${MACRO_ALIEN_DIR} ==" + alien.py cp -f "file:${MACRO_LOCAL_PATH}" "${MACRO_ALIEN_DIR}/staticMapCreatorCPM.C" +else + echo "== MACRO_SOURCE=official -- skipping macro upload, jobs will use \${O2_ROOT}/share/macro/staticMapCreatorCPM.C from PACKAGESPEC ==" +fi + +if [[ -z "${BAD_RANGES_NAME}" ]]; then + # Deliberate opt-out, not a mistake: an empty BAD_RANGES_NAME propagates through to + # mapCreationJob.sh as BAD_RANGES_ARG="" -- staticMapCreatorCPM.C's own + # `if (badRangeList.length() > 0)` guard treats that as "skip bad-ranges filtering entirely" for + # this submission. Nothing to upload or verify. + echo "== BAD_RANGES_NAME is empty -- bad-time-ranges filtering disabled for this submission ==" +elif [[ -n "${BAD_RANGES_LOCAL}" ]]; then + [[ -f "${BAD_RANGES_LOCAL}" ]] || { echo "ERROR: BAD_RANGES_LOCAL not found: ${BAD_RANGES_LOCAL}" >&2; exit 1; } + echo "== Re-staging bad-ranges list from ${BAD_RANGES_LOCAL} to ${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME} ==" + alien.py cp -f "file:${BAD_RANGES_LOCAL}" "${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME}" +else + # No local file to (re-)upload -- the job will download whatever is already staged under + # BAD_RANGES_NAME, so check now, once, locally, that it actually exists. Left unchecked, a wrong + # BAD_RANGES_NAME/ALIEN_USER or a never-uploaded file would only surface inside + # mapCreationJob.sh's own alien.py cp, in every single subjob, with a raw/confusing AliEn error + # instead of one clear message before anything is submitted. + echo "== BAD_RANGES_LOCAL not set -- checking ${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME} already exists on AliEn ==" + if ! alien.py stat "${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME}" &>/dev/null; then + echo "ERROR: ${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME} not found on AliEn, and BAD_RANGES_LOCAL is not set to (re-)upload one." >&2 + echo " Either set BAD_RANGES_LOCAL to a local file to upload, set BAD_RANGES_NAME=\"\" to disable bad-ranges filtering, or upload the file to that path by hand." >&2 + exit 1 + fi + echo "== Found existing ${MACRO_ALIEN_DIR}/lists/${BAD_RANGES_NAME} ==" +fi + +# Tagged by JOBNAME: a shared, untagged manifest name would let a second submission (different +# lists/config) silently overwrite the first's manifest, breaking any already-running job that re-reads +# it and desyncing stage 2's later subjob-index -> slot lookup. +MANIFEST_NAME="mapSlots.${JOBNAME}.tsv" +echo "== Building + uploading slot manifest from ${LISTS_DIR} (as ${MANIFEST_NAME}) ==" +N=$("${SCRIPT_DIR}/stageSlotsForGrid.py" \ + --lists-dir "${LISTS_DIR}" \ + --alien-dir "${MACRO_ALIEN_DIR_BARE}" \ + --manifest-name "${MANIFEST_NAME}" \ + --runs "${RUNS_FILTER}" \ + --slot-length-min "${SLOT_LENGTH_MIN}" \ + --overlap-margin-ms "${OVERLAP_MARGIN_MS}") +echo "N=${N} slot(s) == map(s) to be produced" +[[ "${N}" -gt 0 ]] || { echo "ERROR: no slots found under ${LISTS_DIR}" >&2; exit 1; } + +TMP_SCRIPT="$(mktemp)" +trap 'rm -f "${TMP_SCRIPT}"' EXIT + +# ONE submission, one subjob per manifest line. There is no longer a loop over TimeDivisionIndex: the +# windows are baked into the manifest by stageSlotsForGrid.py, so a subjob index alone identifies a +# (run, window) pair. That also means one stage1Workdir pointer instead of one per index, and no +# TIME_DIVISION/TIME_DIVISION_INDEX placeholders to substitute -- with a fixed SLOT_LENGTH_MIN, +# different-length runs yield different window counts, so the old rectangular +# "one production per index x prodsplit runs" grid does not exist any more. +sed \ + -e "s/^JOBNAME_TAG=.*/JOBNAME_TAG=${JOBNAME}/" \ + -e "s/^MACRO_SOURCE=.*/MACRO_SOURCE=${MACRO_SOURCE}/" \ + -e "s|^MACRO_ALIEN_DIR_BARE=.*|MACRO_ALIEN_DIR_BARE=${MACRO_ALIEN_DIR_BARE}|" \ + -e "s|^BAD_RANGES_NAME=.*|BAD_RANGES_NAME=${BAD_RANGES_NAME}|" \ + "${SCRIPT_DIR}/mapCreationJob.sh" > "${TMP_SCRIPT}" + +GRID_SUBMIT_ARGS=( + --script "${TMP_SCRIPT}" + --jobname "${JOBNAME}" + --topworkdir "${TOPWORKDIR}" + --prodsplit "${N}" + --packagespec "${PACKAGESPEC}" + --asuser "${ASUSER}" +) + +echo "== Submitting ${JOBNAME} (prodsplit=${N}, asuser=${ASUSER}) ==" +# Capture grid_submit.sh's own stderr (its "pok" status messages go there) while still showing it live, +# so we can pull out the one line that tells us where this submission's data will actually land. +SUBMIT_LOG=$("${O2DPG_ROOT}/GRID/utils/grid_submit.sh" "${GRID_SUBMIT_ARGS[@]}" 2>&1 | tee /dev/stderr) + +# Strip ANSI color codes (pok() wraps its output in \033[32m...\033[m) before parsing. +MY_JOBWORKDIR=$(echo "${SUBMIT_LOG}" | sed -E 's/\x1b\[[0-9;]*m//g' | sed -n "s/^Your job's working directory will be //p" | tail -1) +if [[ -z "${MY_JOBWORKDIR}" ]]; then + echo "ERROR: could not find \"Your job's working directory will be ...\" in grid_submit.sh's output -- stage 2 won't be able to find this submission's output. Aborting." >&2 + exit 1 +fi +echo "== MY_JOBWORKDIR = ${MY_JOBWORKDIR} ==" + +WORKDIR_POINTER_NAME="stage1Workdir.${JOBNAME}.txt" +WORKDIR_POINTER_TMP="$(mktemp)" +echo "${MY_JOBWORKDIR}" > "${WORKDIR_POINTER_TMP}" +alien.py cp -f "file:${WORKDIR_POINTER_TMP}" "${MACRO_ALIEN_DIR}/${WORKDIR_POINTER_NAME}" +rm -f "${WORKDIR_POINTER_TMP}" +echo "== Uploaded ${WORKDIR_POINTER_NAME} ==" diff --git a/DATA/production/configurations/mapCreationGrid/submitPostProcess.sh b/DATA/production/configurations/mapCreationGrid/submitPostProcess.sh new file mode 100755 index 000000000..dce0500b7 --- /dev/null +++ b/DATA/production/configurations/mapCreationGrid/submitPostProcess.sh @@ -0,0 +1,145 @@ +#!/usr/bin/env bash +# submitPostProcess.sh +# +# STAGE 2 submission wrapper: submits one grid_submit.sh production-split job, one subjob per slot -- +# same granularity as stage 1 (submitMapCreation.sh), so a failed +# stage-2 subjob can be resubmitted on its own without touching stage 1's (hours-long) map creation. +# +# Unlike stage 1, this does NOT rebuild the slot manifest (mapSlots.tsv) -- it reuses the exact one +# stage 1 already uploaded (downloaded here only to count lines for --prodsplit N). Rebuilding it here +# would risk a mismatch if the local lists/ dir changed between the two submissions, which would +# silently desync stage 2's subjob-index -> slot mapping from stage 1's. +# +# IMPORTANT: STAGE1_JOBNAME_TAG must match whatever the stage-1 submission this is post-processing +# actually used -- it selects both the manifest and the stage1Workdir pointer. There is nothing else to +# keep in sync: the slot windows come from stage 1's own manifest, which this stage reuses verbatim, so +# the two stages cannot disagree about a slot's identity the way they could when each recomputed it. +# +# Output location (per ALICE Grid support -- see submitMapCreation.sh): stage 1 stages its map to its +# own ${ALIEN_JDL_OUTPUTDIR}, not a fixed personal-space path, so there's no static location +# to just look up. Stage 1's wrapper captured MY_JOBWORKDIR (that submission's timestamped parent +# directory) at submission time and uploaded it as a small pointer file -- +# downloaded below and substituted into the payload script (STAGE1_OUTPUTDIR_BASE) alongside COUNTERWIDTH +# (grid_submit.sh's own alien_counter zero-padding width -- deterministic from N, which is shared between +# both stages since this wrapper reuses stage 1's manifest, so it's recomputed here rather than needing +# its own pointer file). The payload script reconstructs each slot's exact ${ALIEN_JDL_OUTPUTDIR} from +# those two values the same way grid_submit.sh itself derives OutputDir. +# +# ASUSER / quota: see submitMapCreation.sh's own comment for the full story -- ASUSER +# determines both the role JAliEn attributes the job to and where MY_HOMEDIR (and therefore OutputDir) +# lands. Must match whatever STAGE1_JOBNAME_TAG's submission actually used, since STAGE1_OUTPUTDIR_BASE +# below (from stage 1's pointer file) already reflects wherever stage 1's own ASUSER put it -- this +# script doesn't need its own OutputDir logic, it just needs to read/write consistently with whatever +# stage 1 did. +# +# Usage: +# ./submitPostProcess.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# --------------------------------------------------------------------------- +# Config -- edit per submission. STAGE1_JOBNAME_TAG must name the stage-1 submission this +# post-processes; everything else about the slot layout comes from that submission's manifest. +# --------------------------------------------------------------------------- +JOBNAME="mapcreation-postprocess-v1" # this stage's OWN job name (GRID job monitoring identity) -- + # NOT the same thing as STAGE1_JOBNAME_TAG below. +STAGE1_JOBNAME_TAG="mapcreation-v1" # must equal the JOBNAME stage 1 (submitMapCreation.sh) was + # actually submitted with -- that's what determined where + # stage 1 staged its map. +TOPWORKDIR="MapCreation" +PACKAGESPEC="O2::daily-20260729-0000-1" +ASUSER="pwg_pp" # see the long comment above -- your own account for personal quota, "pwg_pp" for group + # quota (this stage's own submission; independent of what stage 1 used). +MACRO_SOURCE="official" # same meaning/default as stage 1 (submitMapCreation.sh) -- applies + # to all three macros used here. +MACRO_LOCAL_DIR="" # only used when MACRO_SOURCE=alien: local directory containing + # SmoothingExtrapolate.C, TPCFastTransformInitCPM.C, voxResQA.C to upload, e.g. + # your O2 checkout's Detectors/TPC/calibration/SpacePoints/macro/ + + # GPU/TPCFastTransformation/macro/ (voxResQA.C and SmoothingExtrapolate.C live in + # the former, TPCFastTransformInitCPM.C in the latter -- symlink/copy them into one + # local dir first if they're not already colocated). + # Same staging root as stage 1, derived the same way (see that script's comment): + # ALIEN_USER, else the local username; override with MAPCREATION_ALIEN_DIR. Must + # resolve to whatever stage 1 actually used, since that's where its manifest and + # stage1Workdir pointer live. +ALIEN_USER_NAME="${ALIEN_USER:-$(whoami)}" +MACRO_ALIEN_DIR_BARE="${MAPCREATION_ALIEN_DIR:-/alice/cern.ch/user/${ALIEN_USER_NAME:0:1}/${ALIEN_USER_NAME}/MapCreation}" +# --------------------------------------------------------------------------- + +MACRO_ALIEN_DIR="alien://${MACRO_ALIEN_DIR_BARE}" +echo "== Staging area (must match stage 1's): ${MACRO_ALIEN_DIR_BARE} ==" +echo " (derived from AliEn user '${ALIEN_USER_NAME}' -- export ALIEN_USER or MAPCREATION_ALIEN_DIR if that is wrong)" + +if [[ "${MACRO_SOURCE}" == "alien" ]]; then + [[ -n "${MACRO_LOCAL_DIR}" && -f "${MACRO_LOCAL_DIR}/SmoothingExtrapolate.C" && -f "${MACRO_LOCAL_DIR}/TPCFastTransformInitCPM.C" && -f "${MACRO_LOCAL_DIR}/voxResQA.C" ]] || { echo "ERROR: MACRO_SOURCE=alien needs MACRO_LOCAL_DIR set to a directory containing all three macros (got '${MACRO_LOCAL_DIR}')" >&2; exit 1; } + echo "== Re-staging SmoothingExtrapolate.C / TPCFastTransformInitCPM.C / voxResQA.C from ${MACRO_LOCAL_DIR} to ${MACRO_ALIEN_DIR} ==" + alien.py cp -f "file:${MACRO_LOCAL_DIR}/SmoothingExtrapolate.C" "${MACRO_ALIEN_DIR}/SmoothingExtrapolate.C" + alien.py cp -f "file:${MACRO_LOCAL_DIR}/TPCFastTransformInitCPM.C" "${MACRO_ALIEN_DIR}/TPCFastTransformInitCPM.C" + alien.py cp -f "file:${MACRO_LOCAL_DIR}/voxResQA.C" "${MACRO_ALIEN_DIR}/voxResQA.C" +else + echo "== MACRO_SOURCE=official -- skipping macro upload, jobs will use \${O2_ROOT}/share/macro/ from PACKAGESPEC ==" +fi + +# Tagged manifest name -- must match whatever stage 1 actually uploaded (mapSlots..tsv), +# hence STAGE1_JOBNAME_TAG rather than this stage's own JOBNAME. +MANIFEST_NAME="mapSlots.${STAGE1_JOBNAME_TAG}.tsv" +echo "== Fetching existing slot manifest from ${MACRO_ALIEN_DIR}/${MANIFEST_NAME} (NOT rebuilt) ==" +TMP_MANIFEST="$(mktemp)" +alien.py cp -f "${MACRO_ALIEN_DIR}/${MANIFEST_NAME}" "file:${TMP_MANIFEST}" +N=$(wc -l < "${TMP_MANIFEST}" | tr -d '[:space:]') # wc -l right-justifies its count with leading + # whitespace even without a filename arg; grid_submit.sh's + # --prodsplit does a strict positive-integer check that + # rejects that -- confirmed real, 2026-07-26 ("Production + # split must be a positive integer (got ' 70')"). +echo "N=${N} slot(s)" +[[ "${N}" -gt 0 ]] || { echo "ERROR: ${MANIFEST_NAME} has no lines -- was stage 1 ever submitted with this JOBNAME?" >&2; exit 1; } + +# Same formula grid_submit.sh itself uses to derive its alien_counter zero-padding width -- deterministic +# from N, which is identical between stage 1 and stage 2 (same manifest, same line count), so this will +# match whatever stage 1's own submission actually used without needing to record it separately. +COUNTERWIDTH=${#N} +[[ "${COUNTERWIDTH}" -lt 3 ]] && COUNTERWIDTH=3 + +TMP_SCRIPT="$(mktemp)" +TMP_WORKDIR_POINTER="$(mktemp)" +trap 'rm -f "${TMP_MANIFEST}" "${TMP_SCRIPT:-}" "${TMP_WORKDIR_POINTER}"' EXIT + +# ONE submission, mirroring stage 1: the windows live in the manifest, so a subjob index identifies a +# (run, window) pair by itself and there is nothing to loop over. +WORKDIR_POINTER_NAME="stage1Workdir.${STAGE1_JOBNAME_TAG}.txt" +echo "== Fetching ${WORKDIR_POINTER_NAME} ==" +# Truncate first: an alien.py cp that returns 0 without writing would otherwise leave stale content +# here and point this whole submission at the wrong stage-1 workdir, silently no-opping every slot. +: > "${TMP_WORKDIR_POINTER}" +if ! alien.py cp -f "${MACRO_ALIEN_DIR}/${WORKDIR_POINTER_NAME}" "file:${TMP_WORKDIR_POINTER}" || [[ ! -s "${TMP_WORKDIR_POINTER}" ]]; then + echo "ERROR: could not fetch ${WORKDIR_POINTER_NAME} -- was stage 1 actually submitted under STAGE1_JOBNAME_TAG=${STAGE1_JOBNAME_TAG}?" >&2 + exit 1 +fi +STAGE1_OUTPUTDIR_BASE=$(head -1 "${TMP_WORKDIR_POINTER}") +if [[ "${STAGE1_OUTPUTDIR_BASE}" != /alice/* ]]; then + echo "ERROR: ${WORKDIR_POINTER_NAME} does not contain an absolute /alice/... path: '${STAGE1_OUTPUTDIR_BASE}'" >&2 + exit 1 +fi +echo "== STAGE1_OUTPUTDIR_BASE = ${STAGE1_OUTPUTDIR_BASE} ==" + +sed \ + -e "s/^STAGE1_JOBNAME_TAG=.*/STAGE1_JOBNAME_TAG=${STAGE1_JOBNAME_TAG}/" \ + -e "s/^MACRO_SOURCE=.*/MACRO_SOURCE=${MACRO_SOURCE}/" \ + -e "s|^MACRO_ALIEN_DIR_BARE=.*|MACRO_ALIEN_DIR_BARE=${MACRO_ALIEN_DIR_BARE}|" \ + -e "s|^STAGE1_OUTPUTDIR_BASE=.*|STAGE1_OUTPUTDIR_BASE=${STAGE1_OUTPUTDIR_BASE}|" \ + -e "s/^COUNTERWIDTH=.*/COUNTERWIDTH=${COUNTERWIDTH}/" \ + "${SCRIPT_DIR}/postProcessJob.sh" > "${TMP_SCRIPT}" + +GRID_SUBMIT_ARGS=( + --script "${TMP_SCRIPT}" + --jobname "${JOBNAME}" + --topworkdir "${TOPWORKDIR}" + --prodsplit "${N}" + --packagespec "${PACKAGESPEC}" + --asuser "${ASUSER}" +) + +echo "== Submitting ${JOBNAME} (prodsplit=${N}, asuser=${ASUSER}) ==" +"${O2DPG_ROOT}/GRID/utils/grid_submit.sh" "${GRID_SUBMIT_ARGS[@]}"