Repository navigation
Conversation
Working notes on the migration of BAT to MeasureBase 0.15: what goes and what stays, variate shapes and ValueShapes, the conventions adopted from MeasureBase, the accelerator plan and the open questions. They accompany the migration and will be removed before it is merged. Assisted-By: generative AI
GelmanRubinConvergence and BrooksGelmanConvergence passed the value of
max(R²) as the name parameter of their ValueAndThreshold results instead
of the symbol `:max_Rsqr`. Both return a `ValueAndThreshold{:max_Rsqr}`
now, which is tested.
Assisted-By: generative AI
`maximum(maximum(logds))` took the maximum of the lexicographically largest walker's log-densities, not the maximum over all walkers. Take the maximum over the log-densities of all walkers instead. Assisted-By: generative AI
The docstrings of `InitFromSamples` and `InitFromIID` were swapped, and the `DensitySample` docstring named `DensitySampleVector` as its constructor. The `MCMCIterator` docstring named functions and types that don't exist (or no longer do), such as `getproposal` for chain states, `MCMCIteratorInfo`, `isvalidchain` and a `DensitySampleVector` constructor for chains. Assisted-By: generative AI
The histograms had a hard-coded 40 bins between the minimum and maximum, ignoring `nbins`, and the maximum fell outside the last left-closed bin. StatsBase now chooses edges that cover all values for the given number of bins. Assisted-By: generative AI
`calculate_levels` left a level undefined when a single bin took the cumulative probability past an interval, which raised an UndefRefError. Each level is now the weight of the bin at which the cumulative probability of the bins, in descending order, reaches the interval. Assisted-By: generative AI
A ternary `cond ? subplot := 3 :` took the next attribute line as its else branch, so marginal plots dropped a marker attribute and other plots set it only by accident. Also accept a Dict for `std`, as for the other estimators. Assisted-By: generative AI
BAT will store batches of variates in the layout of `rand(μ^n)`: vectors of numbers, nested arrays and struct vectors of named tuples, with `Fill` columns for constant components. The internal batch layer provides the variate shape of a batch (`batch_varshape`), conversion to and from the flat standard-variate layout (`flatbatch`, `unflatbatch`), leafwise mapping, reshaping and concatenation of batches, and `_own_batch`, which brings `rand` and transport outputs into this layout and checks constant components against their point masses. All of it works with device arrays. Nothing uses it yet, its tests need MeasureBase 0.15 and come with the structured sample storage. Assisted-By: generative AI
MeasureBase 0.15 provides everything BAT's own measure layer used to duplicate, so BAT's measure types subtype `MeasureBase.AbstractMeasure` directly, BAT requires MeasureBase 0.15, HeterogeneousComputing 0.3 and ValueShapes 0.11.11, and DistributionsAD and ForwardDiffPullbacks are no longer used. Removed are the `BATMeasure` supertype, the wrappers `BATDistMeasure`, `BATPwrMeasure`, `BATPushFwdMeasure`, `BATWeightedMeasure` and `BATSuperpositionMeasure` (use `asmeasure`, `powermeasure`, `pushfwd`, `weightedmeasure` and `superpose`), BAT's standard uniform and normal distributions (use `StdUniform()`, `StdNormal()` and their powers) and the distribution transforms (replaced by `transport_to`). Posteriors are MeasureBase measures with the prior as base measure and the likelihood as density, `batmeasure` canonicalizes inputs to measures, and `UniformBased` and `NormalBased` transport to `StdUniform()^n` resp. `StdNormal()^n`. Samples of named-tuple measures are still stored as `ShapedAsNTArray`s, and the tests of the removed types go. Assisted-By: generative AI
`MeasureBase.mbind` is the general form of what `HierarchicalDistribution` provided: a primary measure, a transition kernel and a combination function, with densities, draws and transports derived structurally. The distribution and its `UnshapedHDist` flat view go away. `distbind` and `lbqintegral` are stable API, so they stay as deprecated wrappers for `mbind` resp. `mintegrate`/`mintegrate_exp`. Assisted-By: generative AI
The MCMC samplers evaluated the target density walker by walker, now `BAT.checked_logdensities(target, X)` hands the batch of walker positions to `MeasureBase.logdensities` and applies BAT's NaN/+Inf policy to the results. The zero-prior short-circuit of posterior measures moves from an override of `logdensityof`, which bypassed MeasureBase's checks and its batched path, to MeasureBase's extension points `logdensityof_impl` and `batched_logdensityof_impl`, so posteriors evaluate the prior for the whole batch and the likelihood only where the prior is nonzero. Proposals are built as `VectorOfSimilarVectors` and evaluated from the chain state's buffer, so that MeasureBase sees flat storage. Fused batched kernels can differ from per-variate results in the last ulp, so MCMC runs with a fixed seed may take a different path. Assisted-By: generative AI
`DensitySampleVector` stores its variates in the layout of `rand(μ^n)`, with `Fill` columns for constants, instead of `ShapedAsNTArray`s over flat storage, and everything that produces or consumes samples follows. The new `DensitySampleVector(X, μ)` takes a batch or a flat matrix of variates of a measure and checks constant components against their point masses, `varshape(samples)` is derived from the batch, and statistics, effective sample sizes, credible intervals and sample reports have the structure of the variates. `transform_samples` transports whole batches via `transports_with_ladj`, with log-densities corrected by batched Jacobians, and the importance samplers, initial values and the HDF5, MGVI, NestedSamplers and UltraNest extensions follow the new layout. Broadcasting a value shape over samples and `bat_transform` with a value shape on samples are deprecated. Assisted-By: generative AI
`reshape_variate`, `reshape_variates` and their shape-compatibility checks have no callers anymore. The samplers produce the structured layout directly, and initial values are reshaped to the variate shape of the target via the flat layout. Assisted-By: generative AI
Each object written by `bat_write` carries a `bat_layout` attribute (samples, namedtuple, scalar, array, const or group), plus the field order of named tuples, the length of constant columns, the element type of sample provenance batches and the format version of sample groups. The reader rebuilds the exact structured batch. Files without layout attributes, written by BAT 5.x, go through the value-based heuristic, which now also reads matrix-valued parameters. Assisted-By: generative AI
The recipes and `bat_marginalize` resolve parameter selections against the variate shape derived from the batch, as positions of the free parameters in the standard-variate layout, and read values via `flatbatch`, with one resolver for names, positions, ranges and mixed selections. `bat_marginalize` keeps the sample weights, `MarginalDist` stores the variate shape of the selection, marginals of a `MarginalDist` sum bin probabilities, custom bins apply to the selected parameters and KDE panels of MCMC diagnostics use the sample weights. The unused plotting leftovers `Marginalization`, `chainresults` and `partition_tree` go, and the recipes are tested via RecipesBase. Assisted-By: generative AI
Describe parameter selection by name, path and position (labeled `v_1`, `v_2`, ... for samples without names) and the MCMC diagnostics plots, and drop references to the removed `bat_stats`. Remove the dev-internal plotting examples, which were not part of the docs and used the BAT 1.x result of `bat_sample`. Assisted-By: generative AI
In the tutorial, statistics of samples have the structure of the parameters, the covariance is computed on the samples directly and HDF5 files keep the structure of the variates. The internal API docs list the batch layer and the batched density checks, since the docs build stops on documented internals missing from the manual. Assisted-By: generative AI
BAT 6 builds on MeasureBase 0.15 and stores samples in the layout of their measures. See the pull request for the release notes. Assisted-By: generative AI
MeasureBase renamed pullback to pullbck, the old name is deprecated and fails under --depwarn=error. Both call sites are reached for generic adaptive transforms. Assisted-By: generative AI
`transport_target(f)` and `MeasureBase.atomof(μ)` replace the reads of the fields of `TransportFunction` and `Dirac`. Assisted-By: generative AI
MeasureBase masks the densities of distributions by `Distributions.insupport`, which BATTestCases doesn't define for `FunnelDistribution`. Its support is that of an MvNormal. Assisted-By: generative AI
Chains, steps, purposes and walkers draw from child streams of a keyed RNG, derived with `subrng`. Each sampling run starts from a fresh root drawn from the context RNG, so repeated runs differ while a context built from the same seed reproduces them. A context RNG without a stream key serves as the entropy of such a root and is used directly otherwise. The Random123 partition scheme (`RNGPartition`, counter resets per cycle and step) and the Random123 dependency go away. Two statistical tests that only passed for their particular random streams now use enough draws to be robust. Assisted-By: generative AI
Keep importance samples and coordinate transforms on the selected device. Batch Gaussian scores and bound concurrent likelihood reverse tapes.
DensityInterface v1.0 only moves test_density_interface into a Test extension, which BAT doesn't use. Assisted-By: generative AI
The root of a sampling run was created with `rootrng(rng)`, which always uses the default generator of the compute unit. It is now a child of a keyed context RNG, under a nonce drawn from it, so a run draws with the kind of generator that was chosen for the context. An RNG without a stream key still only provides the entropy of a root. The docs of `BATContext` now state that the algorithm of the default generator is not part of the API. Assisted-By: generative AI
The step-size search of HamiltonianMC drew the momenta of all walkers from one RNG in storage order, so the step size it found depended on the storage order of the walkers. Each walker now probes with a child stream of its own, at chain initialization and after transform changes. The probe streams derive from a tuning stream root that each proposal creates from a child RNG of its own. The HMC components of a multi-proposal shared one tuning stream after a transform change, they now probe independently. Assisted-By: generative AI
Each step derives a single child RNG from cycle and step number, instead of streams per purpose and per walker. The selection of the active proposal draws from it first. Proposals that declare batched draws (RandomWalk, MALAProposal, MCMCGlobalProposal) then draw the variates of all walkers in one call and the acceptance uniforms in a second one, both in the order of the walker ids, so results stay independent of the storage order of the walkers. All other proposals, like HamiltonianMC, get one child stream per walker. Walker rerolls of MCMCRetryInit derive their streams from walker ids instead of storage positions. MCMC results for a fixed seed change. Assisted-By: generative AI
The statistical test of MCMCMultiProposal doesn't show whether every component proposal gets selected, or whether a tuning update reaches the state of the proposal that made the step. Assisted-By: generative AI
Assisted-By: generative AI
Brings in v5.2.2 with the faster HDF5 sample reads (#588). The version stays at 6.0.0. The by-column copy from main replaces the per-sample conversion in the reader for BAT 5.x files, the one place where new-mb still built a wide named tuple for every sample. The test for many parameters also reads a file in the BAT 5.x layout now. Assisted-By: generative AI
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #581 +/- ##
==========================================
+ Coverage 69.76% 70.80% +1.04%
==========================================
Files 123 116 -7
Lines 7253 6549 -704
==========================================
- Hits 5060 4637 -423
+ Misses 2193 1912 -281 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Seed native Philox once per accelerator call from the BAT context RNG. Keep CPU sampling on the existing mutable RNG.
The affine space transformation of an MCMC chain was applied walker by walker, with one matrix-vector product or triangular solve each. It now acts on the flat batch of walker positions in one matrix operation, also when the walker positions are mapped back after a transform update. For 256 walkers in 32 dimensions that is about 20 times faster, and it is a step towards walker positions on GPUs. `transform_samples` applies affine maps to flat batches the same way. Assisted-By: generative AI
For zero-centered normal and t innovations the Hastings terms of the random-walk proposal cancel exactly. The proposal no longer evaluates the innovation measure for them, which it did twice per step. Innovation measures given by the user may be asymmetric and keep their Hastings terms. Assisted-By: generative AI
The update takes the innovations and the proposed positions of the walkers as matrix columns. The factor update becomes a matrix product and the new offset a single matrix-vector product. The rule is unchanged, except for walkers that didn't move: they get zero weight now. A single such walker used to make the tuner skip the update for all walkers. Assisted-By: generative AI
A chain with several walkers summed up the RAM updates of its walkers. The adaptation steps grew with the number of walkers, and the summed update could be indefinite, which a modified Cholesky decomposition had to catch. The update is now the mean over the walkers. It is always positive definite, so a plain Cholesky decomposition suffices, which can also run on GPUs. If it fails due to rounding, the factor stays as it is. Chains with a single walker update as before. Where their rank-one update is not applicable, they use the plain decomposition as well. Assisted-By: generative AI
The step selects the samples that go to the output and the new current samples with the acceptance mask, column by column, instead of collecting index lists and assigning samples through them. Flat walker positions are selected as whole columns of their matrix, which also works for device arrays. The ids of the proposed samples are set in one broadcast. Results don't change. Assisted-By: generative AI
The HMC iteration test required a sample with weight zero in its output. A NUTS step that ends where it started repeats the current sample, so only the sample at the start of the iteration can keep a weight of zero, and only if the first step moves away from it. About one in twelve random streams doesn't do that, and with the new MCMC random streams the stream of this test is one of them on CI. The test now checks that at most one sample has weight zero. Assisted-By: generative AI
Nothing uses `proposaldist_logpdf`, `proposal_rand!` and `mv_proposaldist` any more, and their test file was an empty test set. Assisted-By: generative AI
The device check switched scalar indexing of device arrays back on after it ran, for all tests that follow, with a warning from GPUArraysCore. The test file now disallows it once and leaves it that way, like test runs outside of interactive sessions do by default. Assisted-By: generative AI
The random walk drew its innovations through multivariate normal and t measures, and the t-distribution can only be sampled on the host. Normal and t innovations are now drawn directly as arrays from the generation context of the chain: standard normal variates, for t innovations divided by the root of a chi-square variate over its degrees of freedom. They stay on the compute unit of the context and have its precision. Whether a random walk has Hastings terms now follows from the kind of its innovations: none for normal and t innovations, which are symmetric, the computed ones for innovation measures given by the user. This replaces the flag in the proposal state. MCMC results of random walks for a fixed seed change. Assisted-By: generative AI
For a context with a compute unit other than the host, the chain state keeps the walker positions, their density values and weights, the acceptance mask and the affine space transformation on that unit, positions and transformation in the precision of the context. The transformation and the initial positions are still set up on the host and moved once, and the target is adapted to the compute unit. Such chains support the random walk with normal or t innovations, a triangular affine or no space transformation with RAM or no transform tuning, and repetition weights. Other components throw an ArgumentError. MCMC steps on these chain states follow in later commits. Assisted-By: generative AI
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description written by Claude Code (models Claude Fable 5.1 and 5.2, standard to high effort) on behalf of the maintainer; the commits carry the trailer
Assisted-By: generative AI.BAT 6 builds on the rebuilt MeasureBase (JuliaMath/MeasureBase.jl#182): BAT's own measure layer, distribution wrappers and transforms are gone, hierarchical priors are
mbinds, walker densities are evaluated in batches, and sample storage is structured (StructArrays laid out like MeasureBase'srand, constants asFillcolumns).redesign.md(first commit of the branch, to be dropped before merging) records the design and open points.The release notes below will go into the registrator comment of v6.0.0.
BAT.jl v6.0.0
Breaking changes
BAT's measures are now MeasureBase measures. BAT used to carry its own
measure layer (
BATMeasureand friends), which duplicated what MeasureBasenow provides, and its own distribution-transform implementation. Both are
gone, BAT builds on MeasureBase 0.15 instead.
BAT.BATMeasureis gone, BAT's measure types subtypeMeasureBase.AbstractMeasuredirectly.The wrapper types
BAT.BATDistMeasure,BAT.BATPwrMeasure,BAT.BATPushFwdMeasure,BAT.BATWeightedMeasureandBAT.BATSuperpositionMeasureare gone. UseMeasureBase.asmeasure,powermeasure(^),pushfwd,weightedmeasureandsuperpose.BAT.StandardUvUniform,BAT.StandardMvUniform,BAT.StandardUvNormaland
BAT.StandardMvNormalare gone. UseMeasureBase.StdUniform(),MeasureBase.StdNormal()and their powers, likeStdNormal()^n.BAT.DistributionTransformis gone. UseMeasureBase.transport_to.bat_transformand the transform intents (NormalBased,UniformBased,ToRealVector,DoNotTransform) are unchanged, but the transformationfunctions they return are
MeasureBase.TransportFunctions and transformedmeasures are
MeasureBase.PushforwardMeasures.HierarchicalDistributionis gone. UseMeasureBase.mbind.distprodnow returns aMeasureBase.ProductMeasureinstead of aValueShapes.NamedTupleDistresp. aDistributions.Product. Marginalsare specified as before (distributions, intervals, arrays of those, or
constant values), constant marginals become
MeasureBase.Dirac.lbqintegralanddistbindare deprecated in favour ofMeasureBase.mintegrate/mintegrate_expandMeasureBase.mbind.DensitySampleVectorstores the variates in the layout thatrand(μ^n)produces for their measure
μ, instead of aShapedAsNTArrayover flatstorage: numbers in vectors, arrays as nested arrays (ArraysOfArrays),
named tuples as
StructArrays of their components and constantcomponents as
Fillcolumns.varshape(samples)is derived from thislayout. Vectors of named tuples passed to the constructor are converted
to it. The new constructor
DensitySampleVector(X, μ; logd, weight, info, aux)takes a batch ofvariates of
μor a matrix of their flat free components. Constantcomponents must match the point masses of
μ, and all variates mustshare one layout.
Statistics of samples have the structure of the variates:
mean,var,std,quantile,median,minimum,maximumandmodereturn namedtuples resp. arrays instead of
ShapedAsNTvalues, and so do effectivesample sizes and credible intervals.
minimumandmaximumareelementwise, they used to return the lexicographically smallest resp.
largest variate.
covandcorwork on the flat free components of anysamples.
unshaped.(samples)andbat_transform(Vector, samples)return sampleswith the flat free components of the variates.
vcatandmergeofsamples allocate and promote the types of the weights and log-densities,
so samples with integer and real weights can be combined (
mergeused tokeep the types of its first argument).
push!,append!andmerge!require growable storage, like the flat storage of MCMC output, and throw
an
ArgumentErrorotherwise.Broadcasting a value shape over samples,
vs.(samples), andbat_transformwith a value shape on samples are deprecated. UseDensitySampleVector(X, μ)to get samples in the layout of a measure.BAT no longer defines
mean,var,stdandcovof measures.MeasureBase defines them for measures with exactly known moments and
throws an
ArgumentErrorfor others. BAT keeps itsmedianandmodemethods.
MeasureBase renamed
insupportandparamstoinsupportofandparamsof. BAT implementsinsupportoffor its measures andlikelihoods, custom measures and likelihoods used with BAT must add
methods to the new names as well.
batmeasureis BAT's canonicalization: it turns distributions intomeasures, named tuples of distributions and
NamedTupleDists into productmeasures, and MeasureBase density measures into
PosteriorMeasures.Densities of measures are
-Infoutside their support instead ofthrowing, and transports are
NaNthere; the boundary of the support cantransport to infinite variates. Accordingly,
BAT.checked_logdensityofno longer throws for aNaNdensity at anon-finite variate, it returns a zero density there.
GridSamplerplaces its grid points at the cell midpoints of the unithypercube instead of including its boundary.
DistributionsAD,ForwardDiffPullbacksandRandom123are no longerdependencies.
New features
ImportanceSamplersSampling(experimental) samples with the algorithmsof ImportanceSamplers.jl, on the CPU or on the GPU that the compute unit
of the
BATContextselects. ImportanceSamplers.jl must be loadedexplicitly.
Further behavior changes
MeasureBase.basemeasureof a posterior measure is its prior andMeasureBase.logdensity_defis the log-density of its likelihood, so aposterior behaves like the
mintegrate_exp(likelihood, prior)itrepresents.
MeasureBase.weightedmeasureno longer special-cases posterior measures,it yields a
WeightedMeasurefor them like for any other measure. BAT'sown renormalization (e.g.
bat_sampleon an auto-renormalized target)still rescales the likelihood and keeps the posterior structure.
ValueShapes.varshapeandValueShapes.unshapedare defined forMeasureBase measures and derived from the measure structure.
truncate_batmeasuresupports measures with scalar variates, viaMeasureBase's
truncate. Products of univariate distributions andposterior measures (via their prior) are truncated as before. It still
doesn't renormalize.
bat_writemarks the layout of each object in HDF5 files with anattribute
bat_layout(format version 6, seeBATHDF5IO), sobat_readrestores nested named tuples, array-valued and constant parameters
exactly. Files of BAT 5.x are read as before, with the layout inferred
from the values, now including matrix-valued parameters.
Samples without parameter names, like flat samples, can be plotted.
They are selected by position (among the free parameters, in the order
of
unshaped.(samples)) and labeledv_1,v_2and so on.BAT.bat_marginalizekeeps the sample weights.Sample reports work for samples without free parameters, they list the
constant parameters only.
EllipticalSliceMCMCSamplingrequires a posterior measure whose priortransforms to a standard multivariate normal, and says so instead of
failing later.
Algorithms that record the degrees of freedom of a measure (e.g. the
importance samplers) record
nothingfor measures that declare none,instead of throwing.
The MCMC samplers evaluate the target density for all walkers of a step in
one batched call (
MeasureBase.logdensities) instead of walker by walker.Measures with fused batched kernels may return results that differ from the
per-walker ones in the last ulp, so MCMC runs with a fixed RNG seed can take
a different path than before.
MCMC derives the random streams of its chains and steps from RNGs with
stream keys (
rootrngandsubrngin HeterogeneousComputing) instead ofpartitions of a counter-based RNG. Each sampling run starts from a fresh
keyed root: repeated runs on one context differ, and a context built from
the same seed reproduces them. The root is a child of the RNG of the
context if that has a stream key, and then uses its kind of generator.
Any other RNG provides the entropy of the root. Each step draws from one
stream. Random-walk, MALA and global proposals draw the variates of all
walkers in one call and the acceptance uniforms in a second one, other
proposals like HMC use one child stream per walker. The default RNG of
BATContextis a keyed RNG seeded fromRandom.default_rng(), thealgorithm of the default generator is not part of the API. MCMC results
for a fixed seed differ from BAT 5.
Support for GPUs is provisional. Sample storage, its flat conversion and
concatenation work with device arrays. Batches of variates are drawn in
one call,
rand(rng, μ^n), so on a GPU they come from a single deviceRNG. Reports,
plots and HDF5 output require samples on the host.