light event building

class module0_flow.reco.light.adc64_event_generator.LightADC64EventGenerator(**params)

Bases: h5flow.core.H5FlowGenerator

Light system event builder - converts multiple ADC64-formatted light files to an event-packed h5flow-readable format. Uses the adc64format library to synchronize and align the events in multiple files.

Parameters:

wvfm_dset_name : str, required, path to dataset to store raw waveforms
sn_table : list of int, required, serial number of each ADC (determines order of the ADCs in the output data type)
n_adcs : int, number of ADC serial numbers (default = 2)
n_channels : int, number of channels per ADC (default = 64)
sync_channel : int, channel index to use for identifying sync events (default = 32)
sync_threshold : int, threshold for identifying sync events (default = 5000) [ADC]
sync_buffer : int, optional, number of events to scan to find the first sync for each event builder process, only relevant if using MPI
clock_timestamp_factor : float, tick size for tai_ns in raw data [ns] (default = 0.625)
batch_size : int, optional, number of events to buffer before initiating next loop iteration (default = 128)
utime_ms_window: float, optional, DAQ unix time window to consider for event matching [ms] (default = 1000)
tai_ns_window: float, optional, event timestamp window to consider for event matching [ns] (default = 1000)

Generates a lightweight “event” dataset along with a dataset containing event-packed raw waveforms.

Example config:

flow:
    source: light_event_generator
    stages: []

light_event_generator:
    classname: LightADC64EventGenerator
    path: module0_flow.reco.light.hdf5_event_generator
    dset_name: 'light/events'
    params:
        wvfm_dset_name: 'light/wvfm'
        n_adcs: 2
        n_channels: 64
        sync_channel: 32
        sn_table:
          - 175780172
          - 175854781
        batch_size: 128
        utime_ms_window: 1000
        tai_ns_window: 1000

events datatype:

id          u8,                     unique identifier per event
event       i4,                     event number from source ROOT file
sn          i4(n_adcs,),            serial number of adc
ch          u1(n_adcs,n_channels),  channel id
utime_ms    u8(n_adcs,n_channels),  unix time since epoch [ms]
tai_ns      u8(n_adcs,n_channels),  time since PPS [ns]
wvfm_valid  u1(n_adcs,n_channels),  boolean indicator if channel is present in event

wvfm datatype:

samples     i2(n_adc,n_channels,n_samples), sample 10-bit ADC value (lowest 6 bits are not used)

defaults

event_dtype()

finish()

finish()

init()

next()

static valid_array(arr)

wvfm_dtype()

class module0_flow.reco.light.mc_event_generator.LightEventGenerator(**params)

Bases: h5flow.core.H5FlowGenerator

Light system event builder - converts rwf_XX.root files to an event-packed h5flow-readable format

Parameters:

wvfm_dset_name : str, required, path to dataset to store raw waveforms
n_adcs : int, number of ADC serial numbers
n_channels : int, number of channels per ADC
n_samples : int, number of samples in waveform, optional if RunData resource exists
chunk_size : int, optional, number of events to buffer before initiating loop

Generates a lightweight “event” dataset along with a dataset containing event-packed raw waveforms.

Requires RunData resource in workflow.

Example config:

flow:
    source: light_event_generator
    stages: []

light_event_generator:
    classname: LightEventGenerator
    dset_name: 'light/events'
    params:
        wvfm_dset_name: 'light/wvfm'
        n_adcs: 2
        n_channels: 64
        n_samples: 256
        chunk_size: 128
        utime_ms_window: 1000
        tai_ns_window: 1000

events datatype:

id          u8,                     unique identifier per event
event       i4,                     event number from source ROOT file
sn          i4(n_adcs,),            serial number of adc
ch          u1(n_adcs,n_channels),  channel id
utime_ms    u8(n_adcs,n_channels),  unix time since epoch [ms]
tai_ns      u8(n_adcs,n_channels),  time since PPS [ns]
wvfm_valid  u1(n_adcs,n_channels),  boolean indicator if channel is present in event

wvfm datatype:

samples     i2(n_adc,n_channels,n_samples), sample 10-bit ADC value (lowest 6 bits are not used)

buffer_dtype()

default_chunk_size = 128

default_n_adcs = 2

default_n_channels = 64

default_tai_ns_mod = 1000000000

default_tai_ns_window = 1000

default_utime_ms_window = 1000

event_dtype()

finish()

init()

next()

store_entry(): Convert TTree entry into a numpy type - this is heckin slow….

store_event(event_number)

Pull from event buffers and assemble into event (fills event and wvfm arrays)

Returns:: event_number : event_number will be incremented when full event has been assembled

wvfm_dtype()

class module0_flow.reco.light.mc_event_generator.LightEventGeneratorMC(**params)

Bases: h5flow.core.H5FlowGenerator

Light system event builder for simulation only. Converts the light waveforms, triggers, and truth info into the same format as generated by the LightEventGenerator

Parameters:

wvfm_dset_name : str, required, path to dataset to store raw waveforms
n_adcs : int, number of ADC serial numbers
n_channels : int, number of channels per ADC
adc_sn : list of int, serial number of each ADC
channel_map: list of list of int, mapping from simulation optical detector to adc, channel, -1 values indicate channel is not connected
busy_channel: list of int, channel used for busy signal on each ADC (if relevant)
busy_delay: int, number of ticks prior to busy signal for each trigger
disabled_channels: list of (adc_idx, channel_idx), channels to zero out (optional)

Requires RunData resource in workflow.

Example config:

flow:
    source: light_event_generator_mc
    stages: []

light_event_generator_mc:
    classname: LightEventGeneratorMC
    dset_name: 'light/events'
    params:
        wvfm_dset_name: 'light/wvfm'
        n_adcs: 2
        n_channels: 64
        adc_sn:
         - 0
         - 1
        channel_map:
         - [0,1,2,3,4,5,6,7,8,9,
            10,11,12,13,14,15,16,17,18,19,20,
            21,22,23,24,25,26,27,28,29,30,
            31,32,33,34,35,36,37,38,39,40,
            41,42,43,44,45,46,47,48,49,50,
            51,52,53,54,55,56,57,58,59,60,
            61,62,63]
         - [0,1,2,3,4,5,6,7,8,9,
            10,11,12,13,14,15,16,17,18,19,20,
            21,22,23,24,25,26,27,28,29,30,
            31,32,33,34,35,36,37,38,39,40,
            41,42,43,44,45,46,47,48,49,50,
            51,52,53,54,55,56,57,58,59,60,
            61,62,63]
        busy_channel:
         - 0
         - 0
        busy_delay: 120
        busy_ampl: 10000

See LightEventGenerator for events and wvfm datatypes.

defaults

finish()

init()

next()

class module0_flow.reco.light.raw_event_generator.LightEventGenerator(**params)

Bases: h5flow.core.H5FlowGenerator

Light system event builder - converts rwf_XX.root files to an event-packed h5flow-readable format

Parameters:

wvfm_dset_name : str, required, path to dataset to store raw waveforms
n_adcs : int, number of ADC serial numbers
n_channels : int, number of channels per ADC
n_samples : int, number of samples in waveform, optional if RunData resource exists
chunk_size : int, optional, number of events to buffer before initiating loop

Generates a lightweight “event” dataset along with a dataset containing event-packed raw waveforms.

Requires RunData resource in workflow.

Example config:

flow:
    source: light_event_generator
    stages: []

light_event_generator:
    classname: LightEventGenerator
    dset_name: 'light/events'
    params:
        wvfm_dset_name: 'light/wvfm'
        n_adcs: 2
        n_channels: 64
        n_samples: 256
        chunk_size: 128
        utime_ms_window: 1000
        tai_ns_window: 1000

events datatype:

id          u8,                     unique identifier per event
event       i4,                     event number from source ROOT file
sn          i4(n_adcs,),            serial number of adc
ch          u1(n_adcs,n_channels),  channel id
utime_ms    u8(n_adcs,n_channels),  unix time since epoch [ms]
tai_ns      u8(n_adcs,n_channels),  time since PPS [ns]
wvfm_valid  u1(n_adcs,n_channels),  boolean indicator if channel is present in event

wvfm datatype:

samples     i2(n_adc,n_channels,n_samples), sample 10-bit ADC value (lowest 6 bits are not used)

buffer_dtype()

default_chunk_size = 128

default_n_adcs = 2

default_n_channels = 64

default_tai_ns_mod = 1000000000

default_tai_ns_window = 1000

default_utime_ms_window = 1000

event_dtype()

finish()

init()

next()

store_entry(): Convert TTree entry into a numpy type - this is heckin slow….

store_event(event_number)

Pull from event buffers and assemble into event (fills event and wvfm arrays)

Returns:: event_number : event_number will be incremented when full event has been assembled

wvfm_dtype()

light hit reconstruction

waveform noise filter

class module0_flow.reco.light.wvfm_noise_filter.WaveformNoiseFilter(**params)

Bases: h5flow.core.H5FlowStage

Applies a custom noise filter algorithm across specified waveform channels, looping on light event data.

Coherent noise filter averages every modulo_param-th sample from filter_samples[0]->filter_samples[1], e.g. avg[i] = 1/N * (sample[i] + sample[i+1*modulo_param] + sample[i+2*modulo_param] + ...). Then applies a subtraction across the waveform of filtered[i] = sample[i] - avg[i % modulo_param].

Finally a pedestal subtraction is applied as:

filtered[i] = filtered[i] - filtered[filter_samples[0]:filter_samples[1]].mean()

Parameters:

fwvfm_dset_name : str, required, output dataset path
wvfm_dset_name : str, required, input dataset path for waveforms
filter_channels : list of int, optional, list of channels to apply filter to (others are copied to output dataset)
filter_samples : list of int, length of 2, min and max sample to use for filter
modulo_param : int, repeat template after this number of samples (starting with filter_samples[0])

wvfm_dset_name is required in the data cache.

Example config:

wvfm_noise_filter:
    classname: WaveformNoiseFilter
    requires:
        - 'light/events'
        - 'light/wvfm'
    params:
        fwvfm_dset_name: 'light/fwvfm'
        wvfm_dset_name: 'light/wvfm'
        filter_channels: [ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63 ]
        filter_samples: [ 0, 80 ]
        modulo_param: 10
        keep_noise: True
        noise_dset_name: 'light/fwvfm_noise'

Uses the same dtype as the input waveform dataset except with 'samples' converted to floats.

class_version = 1.0.0

default_filter_samples = (0, 80)

default_keep_noise = False

default_modulo_param = 10

default_noise_dset_name = light/fwvfm_noise

fwvfm_dtype(nadc, nchannels, nsamples)

init(source_name)

run(source_name, source_slice, cache)

waveform deconvolution

class module0_flow.reco.light.wvfm_deconv.WaveformDeconvolution(**params)

Bases: h5flow.core.H5FlowStage

Applies a Wiener deconvolution filter to each waveform based on an electronics impulse response function, an input noise power spectrum, and an input signal plus noise power spectrum:

wvfm_fft = FFT(wvfm)
filtered_wvfm_fft = wvfm_fft * CONJ(response_fft) * |sig_fft|^2 / (|sig_fft|^2 * |response_fft|^2 + |noise_fft|^2)
filtered_wvfm = IFFT(filtered_wvfm_fft)

Or applies an inverse filter to each waveform base on the electronics response:

wvfm_fft = FFT(wvfm)
filtered_wvfm_fft = wvfm_fft * CONJ(response_fft) / |response_fft|^2

Also can generate the signal and noise power spectra from non-PPS and PPS light triggers, respectively.

Finally, can also generate an impulse response function from non-PPS light triggers using the rising-edge aligned waveform average

Parameters:

wvfm_dset_name: waveform dataset to analyze, required
filter_channels: list of channels to apply filter on, others are copied, required
deconv_dset_name: output dataset name, required
pps_channel: channel to detect PPS events, default=32
pps_threshold: threshold to detect PPS events, default=0
noise_strategy: noise estimation strategy (for noise extraction only), either pps or slice, default=``pps``
noise_slice: samples to use for noise estimation if noise strategy is slice
signal_amplitude: bounds for waveform amplitude to include in signal and impulse function extraction, default=``(-inf,inf)``
gen_noise_spectrum: flag to produce a noise spectrum .npz file from analyzed waveforms, default=``False``
gen_signal_spectrum: flag to produce a signal spectrum .npz file from analyzed waveforms, default=``False``
gen_signal_impulse: flag to produce an impulse .npz file from analyzed waveforms, default=``False``
impulse_alignment_oversampling: factor to increase samples for aligning waveforms, default=10
do_filtering: flag to produce inverse/wiener filtered dataset from input .npz files, , default=``True``
filter_type: set inverse filtering strategy, either wiener, inverse, or matched, default=``wiener``
gaus_filter_width: set gaussian filter width applied to filtered waveforms, a value of 0 does not apply a gaussian filter, default=``0``
noise_spectrum_filename: filename for input/output noise spectrum .npz, default=``wvfm_deconv_noise_power.npz``
signal_spectrum_filename: filename for input/output signal spectrum .npz, default=``wvfm_deconv_signal_power.npz``
signal_impulse_filename: filename for input/output signal impulse .npz, default=``wvfm_deconv_signal_impulse.npz``

Example config:

# configuration for impulse, signal, and noise extraction
light_deconv:
    classname: WaveformDeconvolution
    requires:
        - 'light/fwvfm'
    params:
        wvfm_dset_name: 'light/fwvfm' # use pedestal+noise subtracted waveforms
        deconv_dset_name: 'light/deconv_wvfm'
        gen_noise_spectrum: True
        gen_signal_spectrum: True
        gen_signal_impulse: True
        do_filtering: False
        filter_type: Wiener #, Inverse, or Matched
        gaus_filter_width: 2 # use a gaussian filter to reduce HF noise
        noise_strategy: PPS # or slice
        noise_slice: [-256, null] # last 256 samples

FILT_INVERSE = inverse

FILT_MATCHED = matched

FILT_WIENER = wiener

NOISE_PPS = pps

NOISE_SLICE = slice

class_version = 0.0.1

default_gaus_filter_width = 0

default_noise_spectrum_filename = wvfm_deconv_noise_power.npz

default_signal_amplitude = (0,)

default_signal_impulse_filename = wvfm_deconv_signal_impulse.npz

default_signal_spectrum_filename = wvfm_deconv_signal_power.npz

finish(source_name)

init(source_name)

run(source_name, source_slice, cache)

write_spectrum_or_impulse(name, data, spectrum=False, impulse=False, **attrs)

waveform summation

class module0_flow.reco.light.wvfm_sum.WaveformSum(**params)

Bases: h5flow.core.H5FlowStage

Sums the signal across light detector SiPM channels, while applying a gain correction to each SiPM.

Parameters:

wvfm_dset_name : str, required, input dataset path
swvfm_dset_name : str, required, output dataset path
gain: dict of dict of <adc #>: <channel #>: <gain correction> where each gain correction converts the ADC value to visible energy
gain_mc: same as gain, but only applied if datafile is simulation

wvfm_dset_name along with {wvfm_dset_name}/alignment are required in the data cache.

The Geometry resource is required in the workflow.

Example config:

wvfm_sum:
    classname: WaveformSum
    requires:
        - 'light/events'
        - 'light/deconv'
    params:
        wvfm_dset_name: 'light/deconv'
        swvfm_dset_name: 'light/swvfm'
        gain:
            default: 1.0

Uses the same dtype as the input waveform dataset(s) except with (nadc, nchannel) resized to be (ntpc, ndet).

align_dtype(ntpc, ndet)

class_version = 1.0.0

default_detector_channels

init(source_name)

run(source_name, source_slice, cache)

swvfm_dtype(ntpc, ndet, nsamples)

waveform alignment

class module0_flow.reco.light.wvfm_align.WaveformAlign(**params)

Bases: h5flow.core.H5FlowStage

Calculates the relative aligment of each ADC trigger within the event.

Parameters:

wvfm_dset_name : str, required, input dataset path
busy_channel: dict of int of ``<adc #>: <channel number>
align_dset_name : str, optional, output dataset path

wvfm_dset_name is required in the data cache.

Example config:

wvfm_align:
    classname: WaveformAlign
    requires:
        - 'light/deconv'
    params:
        wvfm_dset_name: 'light/deconv'
        busy_channel:
            All: 0

Saves the alignment data to {wvfm_dset_name}/alignment

alignment datatype:

ns          f8            timestamp corresponding to trigger edge
sample_idx  f4(n_adc,),   sample offset relative to the first trigger in event

align_dtype(nadc)

class_version = 0.0.0

default_busy_channel

init(source_name)

run(source_name, source_slice, cache)

waveform hit finding

class module0_flow.reco.light.hit_finder.WaveformHitFinder(**params)

Bases: h5flow.core.H5FlowStage

Extracts “hits” from waveforms. A hit is defined as a local maxima above a defined threshold. Stores the nearest ±N samples around the hit, along with timing information and some summary information.

To most precisely reconstruct the time of a given hit, use the following:

(hits['ns'] + hits['busy_ns'] + hits['ns_spline']) * units.ns

Parameters:

wvfm_dset_name: str, path to input waveforms
t_ns_dset_name: str, path to corrected light PPS timestamps
hits_dset_name: str, path to output hits dataset
near_samples: int, number of neighboring samples to keep
threshold: dict of dict containing sets of tpc_index: {channel_index: threshold, ...} used for hit finding. A fixed global value can also be specified with a single float value
mask: list of int, detectors to ignore when finding hits

Both wvfm_dset_name, {wvfm_dset_name}/alignment, and t_ns_dset_name are required in the cache.

Requires RunData resource in workflow.

hits datatype:

id          u4,             unique identifier
tpc         u1,             tpc index
det         u1,             detector index
sample_idx  u2,             sample index of peak within waveform
ns          f8,             PPS timestamp of waveform [ns]
busy_ns     f8,             timestamp of peak relative to busy rising edge (aka when the waveform was triggered) [ns]
samples     f4(2*near+1,),  sample adc value around peak
sum         f4,             sum of sample adc values (out to ±near_samples)
max         f4,             peak adc value
sum_spline  f4,             integral of spline around peak (out to ±near_samples)
max_spline  f4,             maximum of spline around peak
ns_spline   f4,             offset from center sample for maximum of spline [ns]
rising_spline f4,           projection of spline to rising edge zero-crossing (offset from center sample) [ns]
rising_err_spline f4,       an estimate of the error on the rising edge zero-crossing [ns]
fwhm_spline f4,             spline FWHM [ns]

class_version = 2.0.0

default_global_threshold = 2000

default_hits_dset_name = light/hits

default_interpolation = 256

default_mask = []

default_near_samples = 3

default_threshold(global_threshold)

static find_outlier_mask(arr)

Find outlier mask using median absolute deviation. An outlier is defined as:

|arr - median(arr, axis=-1)| >
    median(|arr - median(arr, axis=-1)|, axis=-1)

Parameters:: arr – 2D masked array of points, shape: (N,M)
Returns:: 2D boolean masked array of outliers, shape: (N,M), True == outlier

hits_dtype(near_samples)

init(source_name)

run(source_name, source_slice, cache)

other light modules

waveform summary

class module0_flow.reco.light.wvfm_summary.WaveformSummary(**params)

Bases: h5flow.core.H5FlowStage

Extracts summary parameters from light waveforms

Parameters:

pretrigger_window : tuple of first sample and last sample to use for calculating pre-trigger values
wvfm_dset_name : str dataset path to waveforms to process
wvfm_summ_dset_name : str, optional, output dataset name, defaults to {wvfm_dset_name}_summ

{}_summ datatype:

id              u8, unique identifier per waveform
event_id        i8, unique identifier for event
pre_std         f8, std of pretrigger samples
pre_mean        f8, mean of pretrigger samples
post_sum        f8, sum of posttrigger samples
post_max        f8, max of posttrigger samples
post_rising     f8, sample index of max derivative
ch              u4, channel id
sn              u4, serial number of adc
adc             u4, adc index 1:1 w/ serial number

class_version = 0.0.0

default_pretrigger_window = (0, 80)

default_wvfm_dset_name = light/wvfm

default_wvfm_summ_dset_fmt = {}_summ

dtype

init(source_name)

run(source_name, source_slice, cache)

light detector calibration

class module0_flow.misc.light_calib.LightCalibration(**params)

Bases: h5flow.core.H5FlowStage

Generates calibration coefficients for SiPM + detector modules based on the charge information from the event.

Applies the following data quality selection:

only 1 light triggers present in event
no charge signal within fid_cut of light detectors

Calibration is based on the maximum waveform amplitude within the specified sample window. A cut can be placed to exclude waveforms with small expected visible energy (vis_energy_cut).

Parameters:

calib_dset_name: str, path to output dataset within HDF5 file
larpix_gain: float, larpix gain in e/mV
fid_cut: float, fiducial cut away from light detectors
vis_energy_cut: float, do not collect calibration data on waveforms with less than this amount of expected energy
gain_prefactor: dict of dict of <adc #>: <channel #>: <prefactor value> adjusts the gain correction on each channel by this amount (e.g. to account for multiple SiPM per module)
sample_window: search between [<min sample>, <max sample>] for the maximum ADC
light_event_dset_name: str, path to light event dataset
wvfm_dset_name: str, path to input waveform dataset
hit_drift_dset_name: str, path to charge hit drift data
hits_dset_name: str, path to input charge hits dataset

All of wvfm_dset_name, hits_dset_name, and hit_drift_dset_name are required in the cache.

Requires Geometry, RunData, and LArData resources in workflow.

calib datatype (1:1 with event):

id          u4,                 unique identifier
vis_charge  f8(nadc,nchannel),  visible charge in e-
vis_energy  f8(nadc,nchannel),  visible energy in keV
sig         f8(nadc,nchannel),  maximum ADC value within sample window

calib_dtype(nadc, nchannel)

class_version = 0.0.0

defaults

finish(source_name)

init(source_name)

run(source_name, source_slice, cache)