Support passing previous image manifest(s) to influence packing

[jlebon]

This is something that rpm-ostree does and makes sense to do here too. In setups like FCOS where the update cadence is slower, this should make a huge difference in layer efficiency.

[jlebon]

A preparatory step for this is to enhance component names to fully identify the layer contents. Otherwise it's impossible to reliably tell which layers from the previous image can actually be reused.

[jlebon]

I think also this has a wider scope than just having good deltas against a previous image, but also when doing derived builds. E.g. if you're adding content on top of a base image with content-based layers, you may want to do content-based splitting. But without the RFE here, any resplitting may trivially "contaminate" the lower base layers, reducing the good delta you'd normally inherit from base updates.

Could be interesting also to support a mode where you rechunk just those "top" layers and leave the base layers alone.

[cgwalters]

In #97 you say

The reason that hasn't been a priority for me is that most users I think don't have clear update/build cadances like FCOS/OCP does so the focus is on optimizing for good deltas in general.

I don't think that's true; a weekly/monthly patching cadence is a pretty common baseline I hear.

[jlebon]

I don't think that's true; a weekly/monthly patching cadence is a pretty common baseline I hear.

Worth teasing apart "update cadence" from "build cadence" here. The trend is definitely more towards rebuilding and pushing everyday or even faster (all of Fedora minus FCOS), bootc images, UBI, Hummingbird, etc... There's been talk in the past to have slower update cadence for the Atomic desktops for better deltas but I haven't heard much on that topic in a while either. Perhaps the appetite is no longer there either.

Edit: but anyway, yes your point is taken that a lot of people/orgs do truly have well-defined and spaced release cadences. I don't have a measure of what's more common than what so I shouldn't have said "most users" there. Regardless we do want this RFE eventually!

[HastD]

I have an idea for how to implement this in combination with #107: With the new statistical partitioning algorithm, as I understand it, the only potential source of layer instability between successive builds of the same image would be if a package's stability score puts it close to the boundary between the high/mid/low stability tiers.

So, to reduce the chance of packages moving between stability tiers, the previous image manifest could be taken into account by essentially introducing "padding" around the tier the package was previously in. For example, something like this:

• If a package was high-stability in the previous image, the package would only be moved to a different tier if its stability is less than mean + 0.8*stddev in the new image.
• If a package was mid-stability, the package would only be moved if its stability is outside the range (mean - 1.2*stddev, mean + 1.2*stddev).
• If a package was low-stability, the package would only be moved if its stability is greater than mean - 0.8*stddev.
• If a package wasn't present in the previous image, the stability tier thresholds would be computed as usual.

This would essentially make the stability tiers somewhat "sticky", so small changes in the stability score couldn't move a package to a different tier, but a package genuinely having a significant change in stability over time could still move tiers.

Thoughts?

[jlebon]

@HastD That makes sense to me offhand, yes. Thanks for chiming in!

When initially building chunkah, I was originally thinking of this case (where we have the previous manifest), we could treat the packing problem very differently because it no longer needs to be probabilistic (we know the answer!). But first, we can't assume that that all users will be using the same "previous" manifest (if e.g. they didn't update in a while), and second, we still can't predict what the next update after this build will be, so we can't either keep skewing from the natural statistical approach because then new consumers would have a degraded experience.

So yes, it makes sense to me to instead in this mode think of it more as the same algorithm being "influenced" (not to mention also, this helps keep code complexity low).

[HastD]

we can't assume that that all users will be using the same "previous" manifest (if e.g. they didn't update in a while)

I don't understand, how would users' update frequency be relevant? This is a build-time consideration, not runtime, and presumably the previous image manifest would be specified at build-time (for example by providing an image reference).

[jlebon]

we can't assume that that all users will be using the same "previous" manifest (if e.g. they didn't update in a while)

I don't understand, how would users' update frequency be relevant? This is a build-time consideration, not runtime, and presumably the previous image manifest would be specified at build-time (for example by providing an image reference).

What matters in the end is what clients already have locally and we can dedupe against. A clearer way to phrase that sentence is:

we can't assume that all users have already downloaded the same previous build described by the "previous" manifest (if e.g. they didn't update in a while)

And this relates to my earlier comment. If you're building daily, but you know the update cadence on the consuming side is not that fast, feeding a previous manifest is not going to buy you much. Perhaps, if we supported multiple previous manifests so we cover a more reasonable lookback window, but I don't think it's worth the complexity for the (likely) micro-gains.

[TheMDev]

Just to chime in here and add some additional use case context, @jlebon comment would probably be the most useful for me.

Could be interesting also to support a mode where you rechunk just those "top" layers and leave the base layers alone.

Usually my images are fairly additive so de-duplicating against the base image has almost no benefit. Where as rechunking the added top layers would have a huge impact on pulls between minor changes.