diff --git a/book/src/drive/cost-tracking.md b/book/src/drive/cost-tracking.md index b24e8e67737..50872290464 100644 --- a/book/src/drive/cost-tracking.md +++ b/book/src/drive/cost-tracking.md @@ -22,10 +22,14 @@ pub enum LowLevelDriveOperation { FunctionOperation(FunctionOp), CalculatedCostOperation(OperationCost), PreCalculatedFeeResult(FeeResult), + CalculatedCostOperationWithRefundOwners { + cost: OperationCost, + refund_owners: RefundOwnersByIdentifier, + }, } ``` -Four variants, each representing a different kind of cost: +Five variants, each representing a different kind of cost: ### GroveOperation @@ -108,6 +112,10 @@ impl FunctionOp { A fee result that was already computed elsewhere and just needs to be included in the total. This is a pass-through -- no further calculation needed. +### CalculatedCostOperationWithRefundOwners + +A `CalculatedCostOperation` whose sectioned storage removal comes with the typed `RefundOwner` recorded for every carrier key when the removed bytes were split. It is pushed by the batch apply generations that split removed bytes with typed storage flags (`push_drive_operation_result_with_refund_owners`). `operation_cost()` rejects it on purpose: a fee decoder that predates typed owners reaches `operation_cost()` through its catch-all arm and therefore fails closed instead of pricing a refund whose owner it cannot route. `combine_cost_operations` leaves it out for the same reason (folding it into a plain `OperationCost` would erase the owners); `combine_cost_operations_with_refund_owners` sums plain and typed costs together and returns the owner of every sectioned removal key: a plain cost's keys are identities, a typed cost must have recorded an owner for each of its own keys, the system key is never an owner, and one key attributed to two owners is refused. See [Refunds](../fees/overview.md#refunds) for how the owners are recorded. + ## BaseOp: Arithmetic Operation Costs For simple computational operations (not storage-related), the `BaseOp` enum provides fixed costs: @@ -320,7 +328,7 @@ These provide a cleaner API than constructing `QualifiedGroveDbOp` directly, and - Let operations accumulate in the `drive_operations` vector throughout the call chain. **Do not:** -- Call `operation_cost()` on a `GroveOperation` -- it will return an error. Grove operations must be executed first; only `CalculatedCostOperation` carries a usable cost. +- Call `operation_cost()` on a `GroveOperation` -- it will return an error. Grove operations must be executed first; only `CalculatedCostOperation` carries a usable cost. `CalculatedCostOperationWithRefundOwners` also returns an error there, by design: its owners must be read by a decoder that routes typed owners. - Forget that storage fees and processing fees are calculated differently. Storage fees are proportional to bytes. Processing fees are a complex function of seeks, loads, hashes, and byte movements. - Assume fee rates are constant. They are versioned through `FeeVersion` and can change between protocol versions. - Ignore `removed_bytes_from_system`. This tracks bytes removed that belong to the system rather than a specific identity, affecting the refund calculation. diff --git a/book/src/fees/overview.md b/book/src/fees/overview.md index c26820e77ea..74d1b699e6f 100644 --- a/book/src/fees/overview.md +++ b/book/src/fees/overview.md @@ -203,13 +203,33 @@ operations and measures the actual cost of each insert, delete, and query. When data is removed from GroveDB (a document is deleted, a key is removed), the system calculates a refund of the original storage fee. Refunds are tracked -per identity per epoch: +per owner per epoch: ```rust -pub struct FeeRefunds(pub CreditsPerEpochByIdentifier); -// BTreeMap> +pub struct FeeRefunds(pub CreditsPerEpochByIdentifier, pub RefundOwnersByIdentifier); +// BTreeMap<[u8; 32], BTreeMap> plus BTreeMap<[u8; 32], RefundOwner> ``` +The first field is the carrier GroveDB hands back: removed bytes keyed by a +32-byte identifier. The second field records the typed owner of every carrier +key. `RefundOwner` (`rs-dpp/src/fee/refund_owner`) names the owner +explicitly: `Identity(id)` for bytes an identity paid for, or +`ContractBucket { contract_id, position }` for bytes a contract credit bucket +paid for. An identity's carrier key is its id, so every historical record +keeps its key; a bucket's carrier key is a domain separated double SHA-256 of +the contract id and the bucket position. The kind is always read from the +record and never inferred from the shape of the key. + +The owner is bound to the bytes when they are stored, in the element flags +(`rs-drive/src/util/storage_flags`). Flag type bytes 0 to 3 are the +historical unowned and identity-owned encodings, produced and parsed by the +pinned `grovedb-epoch-based-storage-flags` crate so they cannot drift. Type +bytes 4 and 5 carry a contract bucket owner. Only the batch apply generation +that knows typed owners splits removed bytes for type bytes 4 and 5 and it +records the owner of every sectioned removal on the cost operation it pushes +(`CalculatedCostOperationWithRefundOwners`); the earlier generation hands the +flags to the crate and fails closed on a type byte it does not know. + Refunds are not 1:1 with the original fee because storage fees are distributed across future epochs (see below). The refund amount depends on how many epochs have elapsed since the data was stored — the longer the data has been stored, the diff --git a/packages/rs-dpp/src/fee/fee_result/mod.rs b/packages/rs-dpp/src/fee/fee_result/mod.rs index 010cc04bbdb..61ddc476816 100644 --- a/packages/rs-dpp/src/fee/fee_result/mod.rs +++ b/packages/rs-dpp/src/fee/fee_result/mod.rs @@ -40,6 +40,7 @@ use crate::consensus::fee::fee_error::FeeError; use crate::fee::fee_result::refunds::FeeRefunds; use crate::fee::fee_result::BalanceChange::{AddToBalance, NoBalanceChange, RemoveFromBalance}; +use crate::fee::refund_owner::RefundOwner; use crate::fee::Credits; use crate::prelude::UserFeeIncrease; use crate::ProtocolError; @@ -121,13 +122,29 @@ impl BalanceChangeForIdentity { &self.change } - /// Returns refund amount of credits for other identities + /// Returns refund amount of credits for other identities. + /// + /// Identity keyed view kept for the shipped balance consumer, which runs + /// only under generations that predate typed owners. A path that may + /// hold bucket owned refunds uses [`Self::other_typed_refunds`], or + /// checks `FeeRefunds::ensure_identity_owners_only` first. pub fn other_refunds(&self) -> BTreeMap { self.fee_result .fee_refunds .calculate_all_refunds_except_identity(self.identity_id) } + /// Returns the refund amount of credits for every recorded owner other + /// than the paying identity, keyed by the typed owner. + /// + /// A carrier key without a recorded owner is an error: the refund cannot + /// be routed and must not be guessed at. + pub fn other_typed_refunds(&self) -> Result, ProtocolError> { + self.fee_result + .fee_refunds + .calculate_all_refunds_except_owner(&RefundOwner::Identity(self.identity_id)) + } + /// Convert into a fee result pub fn into_fee_result(self) -> FeeResult { self.fee_result @@ -286,7 +303,9 @@ mod tests { use super::*; use crate::consensus::fee::fee_error::FeeError; use crate::fee::epoch::CreditsPerEpoch; - use crate::fee::fee_result::refunds::{CreditsPerEpochByIdentifier, FeeRefunds}; + use crate::fee::fee_result::refunds::{ + CreditsPerEpochByIdentifier, FeeRefunds, RefundOwnersByIdentifier, + }; fn make_id(byte: u8) -> Identifier { Identifier::from([byte; 32]) @@ -298,7 +317,11 @@ mod tests { credits_per_epoch.insert(0, credits); let mut map = CreditsPerEpochByIdentifier::new(); map.insert(*identity_id.as_bytes(), credits_per_epoch); - FeeRefunds(map) + let owners = RefundOwnersByIdentifier::from([( + *identity_id.as_bytes(), + RefundOwner::Identity(identity_id), + )]); + FeeRefunds(map, owners) } // --- BalanceChangeForIdentity::change() --- @@ -344,7 +367,11 @@ mod tests { let mut map = CreditsPerEpochByIdentifier::new(); map.insert(*id.as_bytes(), credits_per_epoch_self); map.insert(*other_id.as_bytes(), credits_per_epoch_other); - let refunds = FeeRefunds(map); + let owners = RefundOwnersByIdentifier::from([ + (*id.as_bytes(), RefundOwner::Identity(id)), + (*other_id.as_bytes(), RefundOwner::Identity(other_id)), + ]); + let refunds = FeeRefunds(map, owners); let fee_result = FeeResult { storage_fee: 100, diff --git a/packages/rs-dpp/src/fee/fee_result/refunds.rs b/packages/rs-dpp/src/fee/fee_result/refunds.rs index 65c453dab37..ae50a44040d 100644 --- a/packages/rs-dpp/src/fee/fee_result/refunds.rs +++ b/packages/rs-dpp/src/fee/fee_result/refunds.rs @@ -2,12 +2,18 @@ //! //! Fee refunds are calculated based on removed bytes per epoch. //! +//! The carrier that GroveDB hands back keys removed bytes by a 32-byte +//! identifier. `FeeRefunds` keeps that carrier as its first field and records +//! the typed [`RefundOwner`] of every carrier key in its second field, so a +//! consumer that routes a refund reads the owner's kind from the record and +//! never infers it from the key. use crate::block::epoch::{Epoch, EpochIndex}; use crate::fee::default_costs::KnownCostItem::StorageDiskUsageCreditPerByte; use crate::fee::default_costs::{CachedEpochIndexFeeVersions, EpochCosts}; use crate::fee::epoch::distribution::calculate_storage_fee_refund_amount_and_leftovers; use crate::fee::epoch::{BytesPerEpoch, CreditsPerEpoch}; +use crate::fee::refund_owner::RefundOwner; use crate::fee::Credits; use crate::ProtocolError; use bincode::{Decode, Encode}; @@ -28,12 +34,27 @@ pub type CreditsPerEpochByIdentifier = BTreeMap<[u8; 32], CreditsPerEpoch>; /// Bytes per Epoch by Identifier pub type BytesPerEpochByIdentifier = BTreeMap<[u8; 32], BytesPerEpoch>; -/// Fee refunds to identities based on removed data from specific epochs +pub use crate::fee::refund_owner::RefundOwnersByIdentifier; + +/// Fee refunds to owners based on removed data from specific epochs. +/// +/// The first field is the per-owner, per-epoch credit carrier keyed by +/// [`RefundOwner::removal_key`]. The second field records the typed owner of +/// each carrier key. Every constructor that prices a storage removal fills +/// both; a key without a recorded owner cannot be routed and is treated as an +/// invariant failure by the typed accessors. #[derive(Debug, Clone, Eq, PartialEq, Default, Serialize, Deserialize, Encode, Decode)] -pub struct FeeRefunds(pub CreditsPerEpochByIdentifier); +pub struct FeeRefunds( + pub CreditsPerEpochByIdentifier, + pub RefundOwnersByIdentifier, +); impl FeeRefunds { /// Create fee refunds from GroveDB's StorageRemovalPerEpochByIdentifier + /// + /// This is the untyped path: every carrier key is an identity id, as + /// written by the identity-owned storage flags, and is recorded as + /// [`RefundOwner::Identity`] explicitly. pub fn from_storage_removal( storage_removal: I, current_epoch_index: EpochIndex, @@ -45,7 +66,82 @@ impl FeeRefunds { C: IntoIterator, E: TryInto, { - let refunds_per_epoch_by_identifier = storage_removal + let refunds_per_epoch_by_identifier = Self::price_storage_removal( + storage_removal, + current_epoch_index, + epochs_per_era, + previous_fee_versions, + )?; + + let owners = refunds_per_epoch_by_identifier + .keys() + .map(|identifier| { + ( + *identifier, + RefundOwner::Identity(Identifier::from(*identifier)), + ) + }) + .collect(); + + Ok(Self(refunds_per_epoch_by_identifier, owners)) + } + + /// Create fee refunds from GroveDB's StorageRemovalPerEpochByIdentifier + /// with the owners recorded when the bytes were split. + /// + /// Every carrier key must have an entry in `owners`; a key without one is + /// a removal whose owner was never recorded, which is an invariant + /// failure rather than something to guess at. The system carrier key is + /// never an owner and must be removed by the caller before pricing. + pub fn from_typed_storage_removal( + storage_removal: I, + owners: &RefundOwnersByIdentifier, + current_epoch_index: EpochIndex, + epochs_per_era: u16, + previous_fee_versions: &CachedEpochIndexFeeVersions, + ) -> Result + where + I: IntoIterator, + C: IntoIterator, + E: TryInto, + { + let refunds_per_epoch_by_identifier = Self::price_storage_removal( + storage_removal, + current_epoch_index, + epochs_per_era, + previous_fee_versions, + )?; + + let recorded_owners = refunds_per_epoch_by_identifier + .keys() + .map(|identifier| { + owners + .get(identifier) + .map(|owner| (*identifier, *owner)) + .ok_or_else(|| { + ProtocolError::CorruptedCodeExecution(format!( + "storage removal carrier key {} has no recorded refund owner", + hex::encode(identifier) + )) + }) + }) + .collect::>()?; + + Ok(Self(refunds_per_epoch_by_identifier, recorded_owners)) + } + + fn price_storage_removal( + storage_removal: I, + current_epoch_index: EpochIndex, + epochs_per_era: u16, + previous_fee_versions: &CachedEpochIndexFeeVersions, + ) -> Result + where + I: IntoIterator, + C: IntoIterator, + E: TryInto, + { + storage_removal .into_iter() .map(|(identifier, bytes_per_epochs)| { bytes_per_epochs @@ -72,14 +168,67 @@ impl FeeRefunds { .collect::>() .map(|credits_per_epochs| (identifier, credits_per_epochs)) }) - .collect::>()?; - - Ok(Self(refunds_per_epoch_by_identifier)) + .collect::>() } /// Adds and self assigns result between two Fee Results + /// + /// Credits merge per carrier key and epoch. Recorded owners merge by key; + /// one carrier key naming two different owners is a collision that is + /// reported, never resolved by picking one. pub fn checked_add_assign(&mut self, rhs: Self) -> Result<(), ProtocolError> { - for (identifier, mut int_map_b) in rhs.0.into_iter() { + let Self(rhs_credits, rhs_owners) = rhs; + // owners are checked before either map changes, so a rejected merge + // leaves the accumulator exactly as it was + for identifier in rhs_credits.keys() { + let Some(owner) = rhs_owners.get(identifier) else { + return Err(ProtocolError::CorruptedCodeExecution(format!( + "storage refund carrier key {} has no recorded refund owner", + hex::encode(identifier) + ))); + }; + match self.1.get(identifier) { + Some(existing) if existing != owner => { + return Err(ProtocolError::CorruptedCodeExecution(format!( + "storage removal carrier key {} is recorded for two different refund owners: {:?} and {:?}", + hex::encode(identifier), + existing, + owner + ))); + } + // credits already held without an owner must not acquire + // one from the other side + None if self.0.contains_key(identifier) => { + return Err(ProtocolError::CorruptedCodeExecution(format!( + "storage refund carrier key {} has no recorded refund owner", + hex::encode(identifier) + ))); + } + _ => {} + } + } + for (identifier, owner) in &rhs_owners { + match self.1.get(identifier) { + Some(existing) if existing != owner => { + return Err(ProtocolError::CorruptedCodeExecution(format!( + "storage removal carrier key {} is recorded for two different refund owners: {:?} and {:?}", + hex::encode(identifier), + existing, + owner + ))); + } + // an owner record arriving without credits must not lend an + // owner to credits already held without one + None if self.0.contains_key(identifier) => { + return Err(ProtocolError::CorruptedCodeExecution(format!( + "storage refund carrier key {} has no recorded refund owner", + hex::encode(identifier) + ))); + } + _ => {} + } + } + for (identifier, mut int_map_b) in rhs_credits.into_iter() { let to_insert_int_map = if let Some(sint_map_a) = self.0.remove(&identifier) { // other has an int_map with the same identifier let intersection = sint_map_a @@ -101,6 +250,7 @@ impl FeeRefunds { // reinsert the now combined IntMap self.0.insert(identifier, to_insert_int_map); } + self.1.extend(rhs_owners); Ok(()) } @@ -114,6 +264,32 @@ impl FeeRefunds { self.0.iter() } + /// The recorded owner of a carrier key + pub fn owner_of(&self, key: &[u8; 32]) -> Option { + self.1.get(key).copied() + } + + /// Iterates the refunds with their recorded owners. + /// + /// A carrier key without a recorded owner yields an error: a refund that + /// cannot name its owner cannot be routed and must halt rather than be + /// burned, minted or guessed. + pub fn iter_typed( + &self, + ) -> impl Iterator> + { + self.0.iter().map(|(identifier, credits_per_epoch)| { + self.owner_of(identifier) + .map(|owner| (owner, identifier, credits_per_epoch)) + .ok_or_else(|| { + ProtocolError::CorruptedCodeExecution(format!( + "storage refund carrier key {} has no recorded refund owner", + hex::encode(identifier) + )) + }) + }) + } + /// Sums the fee result among all identities pub fn sum_per_epoch(self) -> CreditsPerEpoch { let mut summed_credits = CreditsPerEpoch::default(); @@ -131,7 +307,36 @@ impl FeeRefunds { summed_credits } - /// Calculates a refund amount of credits per identity excluding specified identity id + /// Checks that every recorded owner is an identity, so that the identity + /// keyed accessors below can be used without reading a bucket's carrier + /// key as an identity id. A carrier key without a recorded owner fails + /// the same way. + /// + /// The identity keyed accessors keep their historical signatures because + /// the shipped balance consumer calls them; that consumer runs only under + /// generations that predate typed owners, and Drive fails closed there on + /// a carrier key that has no identity balance. New callers check this + /// guard first or use the typed accessors. + pub fn ensure_identity_owners_only(&self) -> Result<(), ProtocolError> { + for entry in self.iter_typed() { + let (owner, identifier, _) = entry?; + if owner.as_identity().is_none() { + return Err(ProtocolError::CorruptedCodeExecution(format!( + "storage refund carrier key {} belongs to {:?}, not to an identity", + hex::encode(identifier), + owner + ))); + } + } + Ok(()) + } + + /// Calculates a refund amount of credits per identity excluding specified identity id. + /// + /// Identity keyed view: every carrier key is returned as an identity id. + /// Call [`Self::ensure_identity_owners_only`] first on a path that may + /// hold bucket owned refunds, or use + /// [`Self::calculate_all_refunds_except_owner`]. pub fn calculate_all_refunds_except_identity( &self, identity_id: Identifier, @@ -151,7 +356,39 @@ impl FeeRefunds { .collect() } - /// Calculates a refund amount of credits for specified identity id + /// Calculates the refund amount of credits per recorded owner, excluding + /// the given owner. + /// + /// Owners are compared as typed values, so an identity and a contract + /// bucket never match each other. A carrier key without a recorded owner + /// is an error, as is a sum that overflows. + pub fn calculate_all_refunds_except_owner( + &self, + skip_owner: &RefundOwner, + ) -> Result, ProtocolError> { + let mut refunds_by_owner = BTreeMap::new(); + for entry in self.iter_typed() { + let (owner, _, credits_per_epoch) = entry?; + if owner == *skip_owner { + continue; + } + let credits = credits_per_epoch.values().try_fold(0u64, |sum, credits| { + sum.checked_add(*credits) + .ok_or(ProtocolError::Overflow("storage refund sum overflow")) + })?; + let total: &mut Credits = refunds_by_owner.entry(owner).or_insert(0); + *total = total + .checked_add(credits) + .ok_or(ProtocolError::Overflow("storage refund sum overflow"))?; + } + Ok(refunds_by_owner) + } + + /// Calculates a refund amount of credits for specified identity id. + /// + /// Identity keyed view: looks the identity id up as a carrier key. An + /// identity id can never equal a bucket's carrier key except by a hash + /// preimage, so this stays exact for identity payers. pub fn calculate_refunds_amount_for_identity( &self, identity_id: Identifier, @@ -176,16 +413,23 @@ impl IntoIterator for FeeRefunds { #[cfg(test)] mod tests { use super::*; + use nohash_hasher::IntMap; use once_cell::sync::Lazy; use platform_version::version::fee::FeeVersion; + use std::iter::FromIterator; static EPOCH_CHANGE_FEE_VERSION_TEST: Lazy = Lazy::new(|| BTreeMap::from([(0, FeeVersion::first())])); + fn bucket_owner(contract_byte: u8, position: u16) -> RefundOwner { + RefundOwner::ContractBucket { + contract_id: Identifier::from([contract_byte; 32]), + position, + } + } + mod from_storage_removal { use super::*; - use nohash_hasher::IntMap; - use std::iter::FromIterator; #[test] fn should_filter_out_refunds_under_the_limit() { @@ -208,5 +452,354 @@ mod tests { assert!(credits_per_epoch.get(&0).is_none()); assert!(credits_per_epoch.get(&1).is_some()); } + + #[test] + fn should_record_an_identity_owner_for_every_carrier_key() { + let first = [1u8; 32]; + let second = [2u8; 32]; + let storage_removal = BytesPerEpochByIdentifier::from_iter([ + (first, IntMap::from_iter([(0, 100)])), + (second, IntMap::from_iter([(1, 100)])), + ]); + + let fee_refunds = FeeRefunds::from_storage_removal( + storage_removal, + 3, + 20, + &EPOCH_CHANGE_FEE_VERSION_TEST, + ) + .expect("should create fee refunds"); + + assert_eq!( + fee_refunds.owner_of(&first), + Some(RefundOwner::Identity(Identifier::from(first))) + ); + assert_eq!( + fee_refunds.owner_of(&second), + Some(RefundOwner::Identity(Identifier::from(second))) + ); + assert_eq!(fee_refunds.1.len(), 2); + } + } + + mod from_typed_storage_removal { + use super::*; + + #[test] + fn should_record_the_owner_from_the_map_for_identity_and_bucket_keys() { + let identity = RefundOwner::Identity(Identifier::from([5u8; 32])); + let bucket = bucket_owner(6, 2); + let owners = RefundOwnersByIdentifier::from_iter([ + (identity.removal_key(), identity), + (bucket.removal_key(), bucket), + ]); + let storage_removal = BytesPerEpochByIdentifier::from_iter([ + (identity.removal_key(), IntMap::from_iter([(0, 100)])), + (bucket.removal_key(), IntMap::from_iter([(1, 200)])), + ]); + + let fee_refunds = FeeRefunds::from_typed_storage_removal( + storage_removal.clone(), + &owners, + 3, + 20, + &EPOCH_CHANGE_FEE_VERSION_TEST, + ) + .expect("should create fee refunds"); + + assert_eq!( + fee_refunds.owner_of(&identity.removal_key()), + Some(identity) + ); + assert_eq!(fee_refunds.owner_of(&bucket.removal_key()), Some(bucket)); + + // the credits are priced exactly as the untyped path prices them + let untyped = FeeRefunds::from_storage_removal( + storage_removal, + 3, + 20, + &EPOCH_CHANGE_FEE_VERSION_TEST, + ) + .expect("should create fee refunds"); + assert_eq!(fee_refunds.0, untyped.0); + } + + #[test] + fn should_reject_a_carrier_key_without_a_recorded_owner() { + let bucket = bucket_owner(6, 2); + let owners = RefundOwnersByIdentifier::new(); + let storage_removal = BytesPerEpochByIdentifier::from_iter([( + bucket.removal_key(), + IntMap::from_iter([(1, 200)]), + )]); + + let result = FeeRefunds::from_typed_storage_removal( + storage_removal, + &owners, + 3, + 20, + &EPOCH_CHANGE_FEE_VERSION_TEST, + ); + + assert!(matches!( + result, + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + } + } + + mod checked_add_assign { + use super::*; + + fn refunds_for(owner: RefundOwner, epoch: u16, credits: Credits) -> FeeRefunds { + FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([( + owner.removal_key(), + CreditsPerEpoch::from_iter([(epoch, credits)]), + )]), + RefundOwnersByIdentifier::from_iter([(owner.removal_key(), owner)]), + ) + } + + #[test] + fn should_merge_credits_and_owners_from_both_sides() { + let identity = RefundOwner::Identity(Identifier::from([1u8; 32])); + let bucket = bucket_owner(2, 0); + + let mut refunds = refunds_for(identity, 0, 10); + refunds + .checked_add_assign(refunds_for(identity, 1, 5)) + .expect("should merge the same owner"); + refunds + .checked_add_assign(refunds_for(bucket, 0, 7)) + .expect("should merge a second owner"); + + assert_eq!( + refunds.get(&identity.removal_key()), + Some(&CreditsPerEpoch::from_iter([(0, 10), (1, 5)])) + ); + assert_eq!( + refunds.get(&bucket.removal_key()), + Some(&CreditsPerEpoch::from_iter([(0, 7)])) + ); + assert_eq!(refunds.owner_of(&identity.removal_key()), Some(identity)); + assert_eq!(refunds.owner_of(&bucket.removal_key()), Some(bucket)); + } + + #[test] + fn should_reject_one_carrier_key_recorded_for_two_owners() { + let key = [9u8; 32]; + let identity = RefundOwner::Identity(Identifier::from(key)); + let bucket = bucket_owner(2, 0); + + let mut refunds = FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([( + key, + CreditsPerEpoch::from_iter([(0, 10)]), + )]), + RefundOwnersByIdentifier::from_iter([(key, identity)]), + ); + let colliding = FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([( + key, + CreditsPerEpoch::from_iter([(0, 1)]), + )]), + RefundOwnersByIdentifier::from_iter([(key, bucket)]), + ); + + let before = refunds.clone(); + let result = refunds.checked_add_assign(colliding); + + assert!(matches!( + result, + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + assert_eq!(refunds, before, "a rejected merge changes nothing"); + } + + #[test] + fn should_reject_a_merge_that_would_give_unowned_credits_an_owner() { + let bucket = bucket_owner(2, 0); + let key = bucket.removal_key(); + let unowned = FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([( + key, + CreditsPerEpoch::from_iter([(0, 10)]), + )]), + RefundOwnersByIdentifier::new(), + ); + let owned = refunds_for(bucket, 0, 1); + + let mut left = unowned.clone(); + let before = left.clone(); + assert!(matches!( + left.checked_add_assign(owned.clone()), + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + assert_eq!(left, before, "a rejected merge changes nothing"); + + let mut right = owned; + let before = right.clone(); + assert!(matches!( + right.checked_add_assign(unowned.clone()), + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + assert_eq!(right, before, "a rejected merge changes nothing"); + + // an owner record with no credits of its own must not attach + // itself to credits held without an owner + let owner_only = FeeRefunds( + CreditsPerEpochByIdentifier::new(), + RefundOwnersByIdentifier::from_iter([(key, bucket)]), + ); + let mut left = unowned; + let before = left.clone(); + assert!(matches!( + left.checked_add_assign(owner_only), + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + assert_eq!(left, before, "a rejected merge changes nothing"); + } + } + + mod typed_accessors { + use super::*; + + #[test] + fn should_sum_refunds_per_owner_and_skip_the_given_owner() { + let payer = RefundOwner::Identity(Identifier::from([1u8; 32])); + let other = RefundOwner::Identity(Identifier::from([2u8; 32])); + let bucket = bucket_owner(3, 4); + + let refunds = FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([ + (payer.removal_key(), CreditsPerEpoch::from_iter([(0, 100)])), + ( + other.removal_key(), + CreditsPerEpoch::from_iter([(0, 20), (1, 30)]), + ), + (bucket.removal_key(), CreditsPerEpoch::from_iter([(2, 7)])), + ]), + RefundOwnersByIdentifier::from_iter([ + (payer.removal_key(), payer), + (other.removal_key(), other), + (bucket.removal_key(), bucket), + ]), + ); + + let others = refunds + .calculate_all_refunds_except_owner(&payer) + .expect("should sum"); + + assert_eq!(others, BTreeMap::from_iter([(other, 50), (bucket, 7)])); + + // the identity view keeps its historical shape for identity keys + assert_eq!( + refunds.calculate_refunds_amount_for_identity(Identifier::from([2u8; 32])), + Some(50) + ); + let by_identity = + refunds.calculate_all_refunds_except_identity(Identifier::from([1u8; 32])); + assert_eq!(by_identity.get(&Identifier::from([2u8; 32])), Some(&50)); + } + + #[test] + fn should_fail_closed_on_a_carrier_key_without_a_recorded_owner() { + let refunds = FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([( + [4u8; 32], + CreditsPerEpoch::from_iter([(0, 100)]), + )]), + RefundOwnersByIdentifier::new(), + ); + + let entries: Vec<_> = refunds.iter_typed().collect(); + assert_eq!(entries.len(), 1); + assert!(matches!( + entries[0], + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + assert!(matches!( + refunds.calculate_all_refunds_except_owner(&bucket_owner(1, 1)), + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + } + + #[test] + fn should_guard_the_identity_keyed_view_against_bucket_owned_refunds() { + let identity = RefundOwner::Identity(Identifier::from([1u8; 32])); + let bucket = bucket_owner(3, 4); + let owners = RefundOwnersByIdentifier::from_iter([ + (identity.removal_key(), identity), + (bucket.removal_key(), bucket), + ]); + let removal = BytesPerEpochByIdentifier::from_iter([ + (identity.removal_key(), IntMap::from_iter([(0, 100)])), + (bucket.removal_key(), IntMap::from_iter([(0, 200)])), + ]); + let typed = FeeRefunds::from_typed_storage_removal( + removal, + &owners, + 3, + 20, + &EPOCH_CHANGE_FEE_VERSION_TEST, + ) + .expect("should create fee refunds"); + + assert!(matches!( + typed.ensure_identity_owners_only(), + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + + let identities_only = FeeRefunds::from_storage_removal( + BytesPerEpochByIdentifier::from_iter([( + identity.removal_key(), + IntMap::from_iter([(0, 100)]), + )]), + 3, + 20, + &EPOCH_CHANGE_FEE_VERSION_TEST, + ) + .expect("should create fee refunds"); + identities_only + .ensure_identity_owners_only() + .expect("identity owners pass"); + + let unowned = FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([( + [4u8; 32], + CreditsPerEpoch::from_iter([(0, 100)]), + )]), + RefundOwnersByIdentifier::new(), + ); + assert!(matches!( + unowned.ensure_identity_owners_only(), + Err(ProtocolError::CorruptedCodeExecution(_)) + )); + } + + #[test] + fn should_sum_per_epoch_across_owners_of_both_kinds() { + let identity = RefundOwner::Identity(Identifier::from([1u8; 32])); + let bucket = bucket_owner(3, 4); + let refunds = FeeRefunds( + CreditsPerEpochByIdentifier::from_iter([ + ( + identity.removal_key(), + CreditsPerEpoch::from_iter([(0, 100), (1, 1)]), + ), + (bucket.removal_key(), CreditsPerEpoch::from_iter([(0, 7)])), + ]), + RefundOwnersByIdentifier::from_iter([ + (identity.removal_key(), identity), + (bucket.removal_key(), bucket), + ]), + ); + + assert_eq!( + refunds.sum_per_epoch(), + CreditsPerEpoch::from_iter([(0, 107), (1, 1)]) + ); + } } } diff --git a/packages/rs-dpp/src/fee/mod.rs b/packages/rs-dpp/src/fee/mod.rs index f89ba6be42a..32d60d2a08c 100644 --- a/packages/rs-dpp/src/fee/mod.rs +++ b/packages/rs-dpp/src/fee/mod.rs @@ -2,5 +2,6 @@ pub mod default_costs; pub mod epoch; #[cfg(feature = "fee-distribution")] pub mod fee_result; +pub mod refund_owner; pub use crate::balances::credits::{Credits, SignedCredits}; diff --git a/packages/rs-dpp/src/fee/refund_owner/mod.rs b/packages/rs-dpp/src/fee/refund_owner/mod.rs new file mode 100644 index 00000000000..ca5f0d1ee69 --- /dev/null +++ b/packages/rs-dpp/src/fee/refund_owner/mod.rs @@ -0,0 +1,208 @@ +//! Refund owners +//! +//! Storage refunds are paid back to whoever paid for the bytes that were +//! freed. Historically that was always an identity and the owner was carried +//! as a bare 32-byte identifier. Contract credit buckets can also pay for +//! storage, so the owner of stored bytes is now a typed value whose kind is +//! recorded when the bytes are stored and never inferred from an identifier. +//! +//! The in-memory carrier that GroveDB hands back to Drive keys removed bytes +//! by a 32-byte identifier. [`RefundOwner::removal_key`] gives every owner +//! exactly one such carrier key: an identity's key is its identifier, a +//! contract bucket's key is a domain separated double SHA-256 of the contract +//! id and the bucket position. The recorded owner travels alongside the +//! carrier so that the kind is always read from the record, not from the key. + +use crate::util::hash::hash_double; +use bincode::{Decode, Encode}; +use platform_value::Identifier; +use serde::{Deserialize, Serialize}; +use std::collections::BTreeMap; + +/// Position of a credit bucket inside a contract's credit tree. +/// +/// Stored as a two-byte big-endian key under the contract's credit tree. +/// The contract credit tree lands with the contract credits work on this +/// release branch; this alias keeps the refund owner independent of its +/// arrival order and is the same width either way. +pub type ContractCreditBucketPosition = u16; + +/// Domain separator for the carrier key of a contract bucket refund owner. +/// +/// 35 ASCII bytes without a terminator. The trailing `/0` is the derivation +/// generation: a future change to the derivation is a new storage flag type +/// byte, so the derivation carries no separate version field. +/// +/// Provisional: proposed as the refund owner encoding allocation and pending +/// the owner's confirmation in the fees workstream register (issue 4689). +pub const REFUND_OWNER_CONTRACT_BUCKET_DOMAIN: &[u8; 35] = b"dash-platform/refund-owner/bucket/0"; + +/// Size of a contract bucket carrier key preimage: domain, contract id and +/// big-endian position, no length prefixes +const CONTRACT_BUCKET_PREIMAGE_SIZE: usize = 35 + 32 + 2; + +/// The system carrier key. Bytes that no owner paid for are sectioned under +/// this key and are never refunded, so it is never a recorded owner. +pub const SYSTEM_REFUND_CARRIER_KEY: [u8; 32] = [0; 32]; + +/// The recorded refund owner of every carrier key of a storage removal +pub type RefundOwnersByIdentifier = BTreeMap<[u8; 32], RefundOwner>; + +/// The owner of stored bytes, recorded when the bytes are stored so that a +/// later removal knows where the refund goes. +/// +/// The kind is explicit. It is never derived from the shape or value of an +/// identifier. +#[derive( + Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize, Encode, Decode, +)] +pub enum RefundOwner { + /// An identity paid for the bytes. Refunds are credited to its balance + /// whether or not it may currently spend. + Identity(Identifier), + /// A contract credit bucket paid for the bytes. Refunds are credited to + /// the bucket on the contract credit path. + ContractBucket { + /// The contract whose credit tree holds the bucket. + contract_id: Identifier, + /// The bucket's position inside the contract credit tree. + position: ContractCreditBucketPosition, + }, +} + +impl RefundOwner { + /// The 32-byte key under which this owner's removed bytes are sectioned + /// in GroveDB's storage removal carrier. + /// + /// An identity's key is its identifier verbatim, so every historical + /// record keeps its key. A contract bucket's key is + /// `hash_double(domain || contract_id || position_be)` with the domain + /// from [`REFUND_OWNER_CONTRACT_BUCKET_DOMAIN`], 69 bytes of preimage + /// with no length prefixes. + pub fn removal_key(&self) -> [u8; 32] { + match self { + RefundOwner::Identity(identity_id) => identity_id.to_buffer(), + RefundOwner::ContractBucket { + contract_id, + position, + } => { + let mut preimage = [0u8; CONTRACT_BUCKET_PREIMAGE_SIZE]; + preimage[..35].copy_from_slice(REFUND_OWNER_CONTRACT_BUCKET_DOMAIN); + preimage[35..67].copy_from_slice(contract_id.as_bytes()); + preimage[67..].copy_from_slice(&position.to_be_bytes()); + hash_double(preimage) + } + } + } + + /// The identity when this owner is an identity. + pub fn as_identity(&self) -> Option { + match self { + RefundOwner::Identity(identity_id) => Some(*identity_id), + RefundOwner::ContractBucket { .. } => None, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use bincode::config; + + #[test] + fn should_use_the_identity_id_verbatim_as_the_carrier_key() { + let identity_id = Identifier::from([7u8; 32]); + + assert_eq!(RefundOwner::Identity(identity_id).removal_key(), [7u8; 32]); + } + + #[test] + fn should_derive_the_pinned_carrier_key_for_a_contract_bucket() { + let owner = RefundOwner::ContractBucket { + contract_id: Identifier::from([0x11u8; 32]), + position: 7, + }; + + assert_eq!( + hex::encode(owner.removal_key()), + "e0cb6d5af10d1ece2cf9dccc39dac999ce9cd59e8c1314bc7017f19cfb9e5a1d" + ); + } + + #[test] + fn should_derive_the_bucket_key_from_the_documented_preimage() { + let contract_id = Identifier::from([0x11u8; 32]); + let owner = RefundOwner::ContractBucket { + contract_id, + position: 7, + }; + + let mut preimage = REFUND_OWNER_CONTRACT_BUCKET_DOMAIN.to_vec(); + preimage.extend_from_slice(&[0x11u8; 32]); + preimage.extend_from_slice(&7u16.to_be_bytes()); + assert_eq!(preimage.len(), 69); + + assert_eq!(owner.removal_key(), hash_double(preimage)); + } + + #[test] + fn should_give_distinct_carrier_keys_to_distinct_buckets() { + let contract_a = Identifier::from([1u8; 32]); + let contract_b = Identifier::from([2u8; 32]); + + let a0 = RefundOwner::ContractBucket { + contract_id: contract_a, + position: 0, + } + .removal_key(); + let a1 = RefundOwner::ContractBucket { + contract_id: contract_a, + position: 1, + } + .removal_key(); + let b0 = RefundOwner::ContractBucket { + contract_id: contract_b, + position: 0, + } + .removal_key(); + + assert_ne!(a0, a1); + assert_ne!(a0, b0); + assert_ne!(a1, b0); + assert_ne!(a0, SYSTEM_REFUND_CARRIER_KEY); + } + + #[test] + fn should_never_read_a_bucket_key_as_the_contract_id() { + let contract_id = Identifier::from([9u8; 32]); + let owner = RefundOwner::ContractBucket { + contract_id, + position: 0, + }; + + assert_ne!(owner.removal_key(), contract_id.to_buffer()); + assert_eq!(owner.as_identity(), None); + assert_eq!( + RefundOwner::Identity(contract_id).as_identity(), + Some(contract_id) + ); + } + + #[test] + fn should_round_trip_both_owner_kinds_through_bincode() { + let owners = [ + RefundOwner::Identity(Identifier::from([3u8; 32])), + RefundOwner::ContractBucket { + contract_id: Identifier::from([4u8; 32]), + position: u16::MAX, + }, + ]; + + for owner in owners { + let bytes = bincode::encode_to_vec(owner, config::standard()).expect("should encode"); + let (decoded, _): (RefundOwner, usize) = + bincode::decode_from_slice(&bytes, config::standard()).expect("should decode"); + assert_eq!(decoded, owner); + } + } +} diff --git a/packages/rs-drive/src/drive/contract/get_fetch/fetch_contract_ids/v0/mod.rs b/packages/rs-drive/src/drive/contract/get_fetch/fetch_contract_ids/v0/mod.rs index dd510dc1d15..8971c88f513 100644 --- a/packages/rs-drive/src/drive/contract/get_fetch/fetch_contract_ids/v0/mod.rs +++ b/packages/rs-drive/src/drive/contract/get_fetch/fetch_contract_ids/v0/mod.rs @@ -60,12 +60,12 @@ impl Drive { #[cfg(test)] mod tests { + use crate::util::storage_flags::StorageFlags; use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; use dpp::block::block_info::BlockInfo; use dpp::data_contract::accessors::v0::DataContractV0Setters; use dpp::tests::json_document::json_document_to_contract; use dpp::version::PlatformVersion; - use grovedb_epoch_based_storage_flags::StorageFlags; fn setup_contracts(count: usize) -> (crate::drive::Drive, Vec<[u8; 32]>) { let drive = setup_drive_with_initial_state_structure(None); diff --git a/packages/rs-drive/src/drive/contract/get_fetch/fetch_contracts/v0/mod.rs b/packages/rs-drive/src/drive/contract/get_fetch/fetch_contracts/v0/mod.rs index a266ae2747e..a32a010c9f8 100644 --- a/packages/rs-drive/src/drive/contract/get_fetch/fetch_contracts/v0/mod.rs +++ b/packages/rs-drive/src/drive/contract/get_fetch/fetch_contracts/v0/mod.rs @@ -108,13 +108,13 @@ impl Drive { #[cfg(test)] mod tests { + use crate::util::storage_flags::StorageFlags; use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; use dpp::block::block_info::BlockInfo; use dpp::data_contract::accessors::v0::{DataContractV0Getters, DataContractV0Setters}; use dpp::data_contract::config::v0::DataContractConfigSettersV0; use dpp::tests::json_document::json_document_to_contract; use dpp::version::PlatformVersion; - use grovedb_epoch_based_storage_flags::StorageFlags; fn setup_contracts(count: usize) -> (crate::drive::Drive, Vec<[u8; 32]>) { let drive = setup_drive_with_initial_state_structure(None); diff --git a/packages/rs-drive/src/drive/group/insert/add_group_action/v0/mod.rs b/packages/rs-drive/src/drive/group/insert/add_group_action/v0/mod.rs index e1561556e4a..da02ffc52b6 100644 --- a/packages/rs-drive/src/drive/group/insert/add_group_action/v0/mod.rs +++ b/packages/rs-drive/src/drive/group/insert/add_group_action/v0/mod.rs @@ -13,6 +13,7 @@ use crate::util::grove_operations::{ }; use crate::util::object_size_info::PathKeyInfo::PathFixedSizeKeyRef; use crate::util::object_size_info::{DriveKeyInfo, PathKeyElementInfo}; +use crate::util::storage_flags::StorageFlags; use dpp::block::block_info::BlockInfo; use dpp::data_contract::group::GroupMemberPower; use dpp::data_contract::GroupContractPosition; @@ -26,7 +27,6 @@ use grovedb::batch::KeyInfoPath; use grovedb::element::SumValue; use grovedb::MaybeTree::NotTree; use grovedb::{Element, EstimatedLayerInformation, MaybeTree, TransactionArg, TreeType}; -use grovedb_epoch_based_storage_flags::StorageFlags; use std::collections::HashMap; impl Drive { diff --git a/packages/rs-drive/src/drive/identity/balance/update.rs b/packages/rs-drive/src/drive/identity/balance/update.rs index 83d8278467b..836440c80de 100644 --- a/packages/rs-drive/src/drive/identity/balance/update.rs +++ b/packages/rs-drive/src/drive/identity/balance/update.rs @@ -522,6 +522,7 @@ mod tests { use dpp::fee::epoch::{CreditsPerEpoch, GENESIS_EPOCH_INDEX}; use dpp::fee::fee_result::refunds::{CreditsPerEpochByIdentifier, FeeRefunds}; use dpp::fee::fee_result::FeeResult; + use dpp::fee::refund_owner::RefundOwner; use dpp::fee::{Credits, SignedCredits}; use dpp::version::PlatformVersion; use grovedb::batch::GroveOp; @@ -529,6 +530,62 @@ mod tests { use nohash_hasher::IntMap; use std::collections::BTreeMap; + /// The shipped consumer reads refund carrier keys as identity ids. + /// A bucket owned refund can only reach it under a generation that + /// predates typed owners; it must halt there, never credit anyone. + #[test] + fn should_fail_closed_on_a_bucket_owned_refund() { + let drive = setup_drive_with_initial_state_structure(None); + + let platform_version = PlatformVersion::latest(); + + let identity = create_test_identity(&drive, [0; 32], Some(15), None, platform_version) + .expect("expected to create an identity"); + let bucket = RefundOwner::ContractBucket { + contract_id: [7; 32].into(), + position: 1, + }; + + // the payer's own refund keeps its balance change non-zero, so + // the consumer goes on to the other refunds instead of + // returning early + let refunds_per_epoch_by_identifier: CreditsPerEpochByIdentifier = + BTreeMap::from_iter([ + ( + identity.id().to_buffer(), + IntMap::from_iter([(GENESIS_EPOCH_INDEX, 100000)]), + ), + ( + bucket.removal_key(), + IntMap::from_iter([(GENESIS_EPOCH_INDEX, 200000)]), + ), + ]); + let refund_owners = BTreeMap::from_iter([ + ( + identity.id().to_buffer(), + RefundOwner::Identity(identity.id()), + ), + (bucket.removal_key(), bucket), + ]); + + let fee_result = FeeResult { + fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier, refund_owners), + ..Default::default() + }; + let fee_change = fee_result.into_balance_change(identity.id()); + + let result = drive.apply_balance_change_from_fee_to_identity_operations( + fee_change, + None, + platform_version, + ); + + assert!(matches!( + result, + Err(Error::Drive(DriveError::CorruptedCodeExecution(_))) + )); + } + #[test] fn should_do_nothing_if_there_is_no_balance_change() { let drive = setup_drive_with_initial_state_structure(None); @@ -579,9 +636,19 @@ mod tests { (identity.id().to_buffer(), credits_per_epoch), (other_identity.id().to_buffer(), other_credits_per_epoch), ]); + let refund_owners = BTreeMap::from_iter([ + ( + identity.id().to_buffer(), + RefundOwner::Identity(identity.id()), + ), + ( + other_identity.id().to_buffer(), + RefundOwner::Identity(other_identity.id()), + ), + ]); let fee_result = FeeResult { - fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier), + fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier, refund_owners), ..Default::default() }; let fee_change = fee_result.clone().into_balance_change(identity.id()); @@ -675,9 +742,13 @@ mod tests { let refunds_per_epoch_by_identifier: CreditsPerEpochByIdentifier = BTreeMap::from_iter([(identity.id().to_buffer(), credits_per_epoch)]); + let refund_owners = BTreeMap::from_iter([( + identity.id().to_buffer(), + RefundOwner::Identity(identity.id()), + )]); let fee_result = FeeResult { - fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier), + fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier, refund_owners), ..Default::default() }; let fee_change = fee_result.clone().into_balance_change(identity.id()); @@ -750,9 +821,13 @@ mod tests { let refunds_per_epoch_by_identifier: CreditsPerEpochByIdentifier = BTreeMap::from_iter([(identity.id().to_buffer(), credits_per_epoch)]); + let refund_owners = BTreeMap::from_iter([( + identity.id().to_buffer(), + RefundOwner::Identity(identity.id()), + )]); let fee_result = FeeResult { - fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier), + fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier, refund_owners), ..Default::default() }; let fee_change = fee_result.clone().into_balance_change(identity.id()); diff --git a/packages/rs-drive/src/drive/tokens/balance/update.rs b/packages/rs-drive/src/drive/tokens/balance/update.rs index df479eb6200..19b2db818dd 100644 --- a/packages/rs-drive/src/drive/tokens/balance/update.rs +++ b/packages/rs-drive/src/drive/tokens/balance/update.rs @@ -541,6 +541,7 @@ mod tests { use dpp::fee::epoch::{CreditsPerEpoch, GENESIS_EPOCH_INDEX}; use dpp::fee::fee_result::refunds::{CreditsPerEpochByIdentifier, FeeRefunds}; use dpp::fee::fee_result::FeeResult; + use dpp::fee::refund_owner::RefundOwner; use dpp::fee::{Credits, SignedCredits}; use dpp::version::PlatformVersion; use grovedb::batch::GroveOp; @@ -598,9 +599,19 @@ mod tests { (identity.id().to_buffer(), credits_per_epoch), (other_identity.id().to_buffer(), other_credits_per_epoch), ]); + let refund_owners = BTreeMap::from_iter([ + ( + identity.id().to_buffer(), + RefundOwner::Identity(identity.id()), + ), + ( + other_identity.id().to_buffer(), + RefundOwner::Identity(other_identity.id()), + ), + ]); let fee_result = FeeResult { - fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier), + fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier, refund_owners), ..Default::default() }; let fee_change = fee_result.clone().into_balance_change(identity.id()); @@ -694,9 +705,13 @@ mod tests { let refunds_per_epoch_by_identifier: CreditsPerEpochByIdentifier = BTreeMap::from_iter([(identity.id().to_buffer(), credits_per_epoch)]); + let refund_owners = BTreeMap::from_iter([( + identity.id().to_buffer(), + RefundOwner::Identity(identity.id()), + )]); let fee_result = FeeResult { - fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier), + fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier, refund_owners), ..Default::default() }; let fee_change = fee_result.clone().into_balance_change(identity.id()); @@ -769,9 +784,13 @@ mod tests { let refunds_per_epoch_by_identifier: CreditsPerEpochByIdentifier = BTreeMap::from_iter([(identity.id().to_buffer(), credits_per_epoch)]); + let refund_owners = BTreeMap::from_iter([( + identity.id().to_buffer(), + RefundOwner::Identity(identity.id()), + )]); let fee_result = FeeResult { - fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier), + fee_refunds: FeeRefunds(refunds_per_epoch_by_identifier, refund_owners), ..Default::default() }; let fee_change = fee_result.clone().into_balance_change(identity.id()); diff --git a/packages/rs-drive/src/fees/op.rs b/packages/rs-drive/src/fees/op.rs index 7e55a751bd4..503240168da 100644 --- a/packages/rs-drive/src/fees/op.rs +++ b/packages/rs-drive/src/fees/op.rs @@ -19,15 +19,18 @@ use crate::error::drive::DriveError; use crate::error::Error; use crate::fees::get_overflow_error; use crate::fees::op::LowLevelDriveOperation::{ - CalculatedCostOperation, FunctionOperation, GroveOperation, PreCalculatedFeeResult, + CalculatedCostOperation, CalculatedCostOperationWithRefundOwners, FunctionOperation, + GroveOperation, PreCalculatedFeeResult, }; use crate::util::batch::grovedb_op_batch::GroveDbOpBatchV0Methods; use crate::util::storage_flags::StorageFlags; use dpp::block::epoch::Epoch; use dpp::fee::default_costs::CachedEpochIndexFeeVersions; -use dpp::fee::fee_result::refunds::FeeRefunds; +use dpp::fee::fee_result::refunds::{FeeRefunds, RefundOwnersByIdentifier}; use dpp::fee::fee_result::FeeResult; +use dpp::fee::refund_owner::{RefundOwner, SYSTEM_REFUND_CARRIER_KEY}; use dpp::fee::Credits; +use dpp::identifier::Identifier as PlatformIdentifier; use platform_version::version::fee::FeeVersion; /// Base ops @@ -211,6 +214,22 @@ pub enum LowLevelDriveOperation { CalculatedCostOperation(OperationCost), /// Pre Calculated Fee Result PreCalculatedFeeResult(FeeResult), + /// A calculated cost whose sectioned storage removal carries the typed + /// owner recorded for every carrier key when the bytes were split. + /// + /// Pushed by the batch apply generations that split removed bytes with + /// typed storage flags. Only a fee decoder that knows how to route typed + /// owners may consume it: `operation_cost` rejects it, so a decoder that + /// predates typed owners fails closed instead of pricing a removal whose + /// owner it cannot name. + CalculatedCostOperationWithRefundOwners { + /// The measured cost, whose removed bytes are sectioned under the + /// owners' carrier keys + cost: OperationCost, + /// The recorded owner of every carrier key in the sectioned removal, + /// the system key excepted + refund_owners: RefundOwnersByIdentifier, + }, } /// Shared rejection message for the three `Element` wrappers @@ -325,10 +344,24 @@ impl LowLevelDriveOperation { FunctionOperation(_) => Err(Error::Drive(DriveError::CorruptedCodeExecution( "function operations should not be requested by operation costs", ))), + CalculatedCostOperationWithRefundOwners { .. } => { + Err(Error::Drive(DriveError::CorruptedCodeExecution( + "a cost operation carrying typed refund owners reached a fee decoder that \ + cannot route typed owners", + ))) + } } } - /// Filters the groveDB ops from a list of operations and puts them in a `GroveDbOpBatch`. + /// Sums the plain calculated costs of a list of operations. + /// + /// Only `CalculatedCostOperation` is folded. A typed cost operation is + /// left out on purpose, as function operations and pre-calculated fee + /// results are: folding it into a plain `OperationCost` would erase the + /// recorded refund owners that `operation_cost` refuses to drop, and the + /// operation itself stays in the list for the fee decoder. Use + /// [`Self::combine_cost_operations_with_refund_owners`] to aggregate + /// typed costs without losing their owners. pub fn combine_cost_operations(operations: &[LowLevelDriveOperation]) -> OperationCost { let mut cost = OperationCost::default(); operations.iter().for_each(|op| { @@ -339,6 +372,95 @@ impl LowLevelDriveOperation { cost } + /// Sums the plain and typed calculated costs of a list of operations, + /// establishing the refund owner of every sectioned removal key. + /// + /// Each input attributes its own removal keys before anything merges: a + /// plain cost's sectioned keys are identities, as the identity-only flags + /// that produced them say; a typed cost must carry a recorded owner for + /// every sectioned key of its own, and an unrecorded key is an invariant + /// failure rather than something to guess at. The system key is never an + /// owner. Owners then merge by carrier key, and one key attributed to two + /// different owners, within an input or across inputs, is reported and + /// never resolved by picking one. + pub fn combine_cost_operations_with_refund_owners( + operations: &[LowLevelDriveOperation], + ) -> Result<(OperationCost, RefundOwnersByIdentifier), Error> { + let mut cost = OperationCost::default(); + let mut owners = RefundOwnersByIdentifier::new(); + for op in operations { + match op { + CalculatedCostOperation(operation_cost) => { + if let SectionedStorageRemoval(removal) = + &operation_cost.storage_cost.removed_bytes + { + for key in removal.keys() { + if *key == SYSTEM_REFUND_CARRIER_KEY { + continue; + } + Self::record_refund_owner( + &mut owners, + *key, + RefundOwner::Identity(PlatformIdentifier::from(*key)), + )?; + } + } + cost += operation_cost.clone(); + } + CalculatedCostOperationWithRefundOwners { + cost: operation_cost, + refund_owners, + } => { + if let SectionedStorageRemoval(removal) = + &operation_cost.storage_cost.removed_bytes + { + for key in removal.keys() { + if *key == SYSTEM_REFUND_CARRIER_KEY { + continue; + } + let owner = refund_owners.get(key).ok_or(Error::Drive( + DriveError::CorruptedCodeExecution( + "a typed cost operation sections removed bytes under a \ + carrier key it recorded no refund owner for", + ), + ))?; + Self::record_refund_owner(&mut owners, *key, *owner)?; + } + } + for (key, owner) in refund_owners { + if *key == SYSTEM_REFUND_CARRIER_KEY { + continue; + } + Self::record_refund_owner(&mut owners, *key, *owner)?; + } + cost += operation_cost.clone(); + } + _ => {} + } + } + Ok((cost, owners)) + } + + fn record_refund_owner( + owners: &mut RefundOwnersByIdentifier, + key: [u8; 32], + owner: RefundOwner, + ) -> Result<(), Error> { + match owners.get(&key) { + Some(existing) if *existing != owner => { + Err(Error::Drive(DriveError::CorruptedCodeExecution( + "two different refund owners share one storage removal carrier key across \ + combined cost operations", + ))) + } + Some(_) => Ok(()), + None => { + owners.insert(key, owner); + Ok(()) + } + } + } + /// Filters the groveDB ops from a list of operations and puts them in a `GroveDbOpBatch`. pub fn grovedb_operations_batch( insert_operations: &[LowLevelDriveOperation], @@ -1643,8 +1765,12 @@ impl DriveCost for OperationCost { #[allow(clippy::identity_op)] mod tests { use super::*; - use grovedb_costs::storage_cost::removal::StorageRemovedBytes; + use dpp::identifier::Identifier; + use grovedb_costs::storage_cost::removal::{ + StorageRemovalPerEpochByIdentifier, StorageRemovedBytes, + }; use grovedb_costs::storage_cost::StorageCost; + use intmap::IntMap; use platform_version::version::fee::storage::FeeStorageVersion; use platform_version::version::fee::FeeVersion; @@ -2018,6 +2144,258 @@ mod tests { ); } + #[test] + fn should_reject_a_typed_cost_operation_from_operation_cost() { + let op = CalculatedCostOperationWithRefundOwners { + cost: OperationCost::default(), + refund_owners: Default::default(), + }; + let result = op.operation_cost(); + let err_msg = format!("{:?}", result.expect_err("typed costs are not plain costs")); + assert!( + err_msg.contains("cannot route typed owners"), + "unexpected error: {}", + err_msg + ); + } + + /// The shipped decoder reaches `operation_cost` through its catch-all + /// arm, so a typed cost operation makes it fail closed instead of pricing + /// a removal whose owner it cannot route. + #[test] + fn should_fail_closed_when_consume_to_fees_v0_meets_a_typed_cost_operation() { + let owner = RefundOwner::Identity(Identifier::from([5u8; 32])); + let mut removal = StorageRemovalPerEpochByIdentifier::new(); + removal.insert(owner.removal_key(), IntMap::from_iter([(0u16, 100u32)])); + let op = CalculatedCostOperationWithRefundOwners { + cost: OperationCost { + seek_count: 1, + storage_cost: StorageCost { + added_bytes: 0, + replaced_bytes: 0, + removed_bytes: SectionedStorageRemoval(removal), + }, + storage_loaded_bytes: 0, + hash_node_calls: 0, + sinsemilla_hash_calls: 0, + }, + refund_owners: BTreeMap::from([(owner.removal_key(), owner)]), + }; + + let result = LowLevelDriveOperation::consume_to_fees_v0( + vec![op], + &Epoch::new(1).expect("epoch"), + 20, + fee_version(), + None, + ); + + assert!(matches!( + result, + Err(Error::Drive(DriveError::CorruptedCodeExecution(_))) + )); + } + + /// The plain combiner cannot carry owners, so it leaves typed costs out + /// rather than laundering them into a plain cost that the shipped + /// decoder would price by reading a bucket's carrier key as an identity. + #[test] + fn should_leave_typed_cost_operations_out_of_the_plain_combiner() { + let cost = OperationCost { + seek_count: 4, + storage_cost: StorageCost { + added_bytes: 7, + replaced_bytes: 3, + removed_bytes: StorageRemovedBytes::NoStorageRemoval, + }, + storage_loaded_bytes: 11, + hash_node_calls: 2, + sinsemilla_hash_calls: 0, + }; + let owner = RefundOwner::Identity(Identifier::from([5u8; 32])); + let operations = vec![ + CalculatedCostOperation(cost.clone()), + CalculatedCostOperationWithRefundOwners { + cost: cost.clone(), + refund_owners: BTreeMap::from([(owner.removal_key(), owner)]), + }, + ]; + + let combined = LowLevelDriveOperation::combine_cost_operations(&operations); + + assert_eq!(combined, cost, "only the plain cost is folded"); + } + + fn sectioned_cost(entries: &[([u8; 32], u32)]) -> OperationCost { + let mut removal = StorageRemovalPerEpochByIdentifier::new(); + for (key, bytes) in entries { + removal.insert(*key, IntMap::from_iter([(0u16, *bytes)])); + } + OperationCost { + seek_count: 1, + storage_cost: StorageCost { + added_bytes: 0, + replaced_bytes: 0, + removed_bytes: SectionedStorageRemoval(removal), + }, + storage_loaded_bytes: 0, + hash_node_calls: 0, + sinsemilla_hash_calls: 0, + } + } + + #[test] + fn should_attribute_every_sectioned_key_when_combining_plain_and_typed_costs() { + let identity = RefundOwner::Identity(Identifier::from([5u8; 32])); + let bucket = RefundOwner::ContractBucket { + contract_id: Identifier::from([6u8; 32]), + position: 1, + }; + let operations = vec![ + // a plain cost from identity-only flags: its keys are identities + CalculatedCostOperation(sectioned_cost(&[ + (identity.removal_key(), 100), + (SYSTEM_REFUND_CARRIER_KEY, 7), + ])), + CalculatedCostOperationWithRefundOwners { + cost: sectioned_cost(&[(bucket.removal_key(), 40), (SYSTEM_REFUND_CARRIER_KEY, 3)]), + refund_owners: BTreeMap::from([(bucket.removal_key(), bucket)]), + }, + // the same identity again through a typed cost merges cleanly + CalculatedCostOperationWithRefundOwners { + cost: sectioned_cost(&[(identity.removal_key(), 20)]), + refund_owners: BTreeMap::from([(identity.removal_key(), identity)]), + }, + FunctionOperation(FunctionOp::new_with_round_count(HashFunction::Sha256, 1)), + ]; + + let (combined, owners) = + LowLevelDriveOperation::combine_cost_operations_with_refund_owners(&operations) + .expect("should combine"); + + assert_eq!(combined.seek_count, 3); + let SectionedStorageRemoval(removal) = &combined.storage_cost.removed_bytes else { + panic!("sectioned removals stay sectioned"); + }; + assert_eq!(removal[&identity.removal_key()].get(0u16), Some(&120)); + assert_eq!(removal[&bucket.removal_key()].get(0u16), Some(&40)); + assert_eq!(removal[&SYSTEM_REFUND_CARRIER_KEY].get(0u16), Some(&10)); + assert_eq!( + owners, + BTreeMap::from([ + (identity.removal_key(), identity), + (bucket.removal_key(), bucket), + ]) + ); + assert!( + !owners.contains_key(&SYSTEM_REFUND_CARRIER_KEY), + "the system key is never an owner" + ); + + // the aggregate prices as a whole through the typed constructor + let mut priced_removal = removal.clone(); + priced_removal.remove(&SYSTEM_REFUND_CARRIER_KEY); + FeeRefunds::from_typed_storage_removal( + priced_removal, + &owners, + 3, + 20, + &BTreeMap::from([(0, FeeVersion::first())]), + ) + .expect("every key has an owner"); + } + + /// The typed split closure never records the system key (the all-zero + /// identity records nothing, a bucket deriving it fails the batch), so a + /// system key in an owner map can only come from a hand-built operation. + /// The aggregate drops it rather than hand a consumer an owner that is + /// not routable. + #[test] + fn should_never_record_the_system_key_as_an_owner_from_a_typed_owner_map() { + let operations = vec![CalculatedCostOperationWithRefundOwners { + cost: sectioned_cost(&[(SYSTEM_REFUND_CARRIER_KEY, 7)]), + refund_owners: BTreeMap::from([( + SYSTEM_REFUND_CARRIER_KEY, + RefundOwner::Identity(Identifier::from(SYSTEM_REFUND_CARRIER_KEY)), + )]), + }]; + + let (_, owners) = + LowLevelDriveOperation::combine_cost_operations_with_refund_owners(&operations) + .expect("should combine"); + + assert!(owners.is_empty()); + } + + #[test] + fn should_reject_a_typed_cost_that_sections_bytes_under_an_unrecorded_key() { + let bucket = RefundOwner::ContractBucket { + contract_id: Identifier::from([6u8; 32]), + position: 1, + }; + // 100 unattributed bytes plus 40 attributed bytes under one key must + // not become 140 bucket owned bytes + let operations = vec![ + CalculatedCostOperationWithRefundOwners { + cost: sectioned_cost(&[(bucket.removal_key(), 100)]), + refund_owners: BTreeMap::new(), + }, + CalculatedCostOperationWithRefundOwners { + cost: sectioned_cost(&[(bucket.removal_key(), 40)]), + refund_owners: BTreeMap::from([(bucket.removal_key(), bucket)]), + }, + ]; + + let result = + LowLevelDriveOperation::combine_cost_operations_with_refund_owners(&operations); + + assert!(matches!( + result, + Err(Error::Drive(DriveError::CorruptedCodeExecution(_))) + )); + } + + #[test] + fn should_reject_one_carrier_key_with_two_owners_across_combined_cost_operations() { + let bucket = RefundOwner::ContractBucket { + contract_id: Identifier::from([1u8; 32]), + position: 0, + }; + let key = bucket.removal_key(); + + // a plain identity removal whose key equals the bucket carrier key + let plain_then_typed = vec![ + CalculatedCostOperation(sectioned_cost(&[(key, 10)])), + CalculatedCostOperationWithRefundOwners { + cost: sectioned_cost(&[(key, 1)]), + refund_owners: BTreeMap::from([(key, bucket)]), + }, + ]; + assert!(matches!( + LowLevelDriveOperation::combine_cost_operations_with_refund_owners(&plain_then_typed), + Err(Error::Drive(DriveError::CorruptedCodeExecution(_))) + )); + + // two typed records naming different owners for one key + let typed_conflict = vec![ + CalculatedCostOperationWithRefundOwners { + cost: OperationCost::default(), + refund_owners: BTreeMap::from([( + key, + RefundOwner::Identity(Identifier::from(key)), + )]), + }, + CalculatedCostOperationWithRefundOwners { + cost: OperationCost::default(), + refund_owners: BTreeMap::from([(key, bucket)]), + }, + ]; + assert!(matches!( + LowLevelDriveOperation::combine_cost_operations_with_refund_owners(&typed_conflict), + Err(Error::Drive(DriveError::CorruptedCodeExecution(_))) + )); + } + // --------------------------------------------------------------- // 6. combine_cost_operations — filter and sum // --------------------------------------------------------------- diff --git a/packages/rs-drive/src/util/grove_operations/batch_move/mod.rs b/packages/rs-drive/src/util/grove_operations/batch_move/mod.rs index eb8b64d8214..e6d234cf090 100644 --- a/packages/rs-drive/src/util/grove_operations/batch_move/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/batch_move/mod.rs @@ -5,9 +5,9 @@ use crate::error::drive::DriveError; use crate::error::Error; use crate::fees::op::LowLevelDriveOperation; use crate::util::grove_operations::BatchMoveApplyType; +use crate::util::storage_flags::StorageFlags; use dpp::version::drive_versions::DriveVersion; use grovedb::TransactionArg; -use grovedb_epoch_based_storage_flags::StorageFlags; use grovedb_path::SubtreePath; impl Drive { diff --git a/packages/rs-drive/src/util/grove_operations/batch_move/v0/mod.rs b/packages/rs-drive/src/util/grove_operations/batch_move/v0/mod.rs index 3b4c7025471..98ed93be1e2 100644 --- a/packages/rs-drive/src/util/grove_operations/batch_move/v0/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/batch_move/v0/mod.rs @@ -4,11 +4,11 @@ use crate::error::Error; use crate::fees::op::LowLevelDriveOperation; use crate::fees::op::LowLevelDriveOperation::GroveOperation; use crate::util::grove_operations::{push_drive_operation_result, BatchMoveApplyType, QueryType}; +use crate::util::storage_flags::StorageFlags; use grovedb::batch::key_info::KeyInfo; use grovedb::batch::{KeyInfoPath, QualifiedGroveDbOp}; use grovedb::operations::delete::DeleteOptions; use grovedb::{Element, GroveDb, TransactionArg}; -use grovedb_epoch_based_storage_flags::StorageFlags; use grovedb_path::SubtreePath; use grovedb_storage::rocksdb_storage::RocksDbStorage; use platform_version::version::drive_versions::DriveVersion; diff --git a/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/mod.rs b/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/mod.rs index 7c31d0b7a66..f65f9ad6a8f 100644 --- a/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/mod.rs @@ -8,8 +8,8 @@ use crate::util::grove_operations::BatchMoveApplyType; use dpp::version::drive_versions::DriveVersion; +use crate::util::storage_flags::StorageFlags; use grovedb::{PathQuery, TransactionArg}; -use grovedb_epoch_based_storage_flags::StorageFlags; impl Drive { /// Pushes multiple "delete element" and "insert element operations for items in a given path based on a `PathQuery` to `drive_operations`. diff --git a/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/v0/mod.rs b/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/v0/mod.rs index b65b3ccb69e..66dfc91f48c 100644 --- a/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/v0/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/batch_move_items_in_path_query/v0/mod.rs @@ -4,12 +4,12 @@ use crate::error::Error; use crate::fees::op::LowLevelDriveOperation; use crate::fees::op::LowLevelDriveOperation::GroveOperation; use crate::util::grove_operations::{push_drive_operation_result, BatchMoveApplyType}; +use crate::util::storage_flags::StorageFlags; use grovedb::batch::key_info::KeyInfo; use grovedb::batch::{KeyInfoPath, QualifiedGroveDbOp}; use grovedb::operations::delete::DeleteOptions; use grovedb::query_result_type::QueryResultType; use grovedb::{GroveDb, PathQuery, TransactionArg}; -use grovedb_epoch_based_storage_flags::StorageFlags; use grovedb_storage::rocksdb_storage::RocksDbStorage; use platform_version::version::drive_versions::DriveVersion; diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/mod.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/mod.rs index ccea6a9242a..31c316d7af5 100644 --- a/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/mod.rs @@ -1,4 +1,8 @@ mod v0; +mod v1; + +#[cfg(test)] +mod tests; use crate::util::batch::GroveDbOpBatch; @@ -14,6 +18,11 @@ use grovedb::TransactionArg; impl Drive { /// Applies the given groveDB operations batch and gets and passes the costs to `push_drive_operation_result`. /// + /// Version 0 splits removed bytes with the identity-only storage flags + /// and pushes a plain cost operation. Version 1 splits with the typed + /// storage flags and pushes a cost operation that carries the recorded + /// refund owner of every sectioned removal. + /// /// # Parameters /// * `ops`: The groveDB operations batch. /// * `validate`: Specifies whether to validate that insertions do not override existing entries. @@ -40,9 +49,16 @@ impl Drive { drive_operations, drive_version, ), + 1 => self.grove_apply_batch_with_add_costs_v1( + ops, + validate, + transaction, + drive_operations, + drive_version, + ), version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "grove_apply_batch_with_add_costs".to_string(), - known_versions: vec![0], + known_versions: vec![0, 1], received: version, })), } diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/tests.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/tests.rs new file mode 100644 index 00000000000..cbe93765a13 --- /dev/null +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/tests.rs @@ -0,0 +1,507 @@ +//! Tests that run both batch apply generations through the dispatcher. +//! +//! Version 1 is selected only by a test-built drive version: no protocol +//! version references it until the fee decoder that consumes the typed cost +//! operation lands. + +use crate::drive::system::misc_path_vec; +use crate::drive::Drive; +use crate::error::Error; +use crate::fees::op::LowLevelDriveOperation; +use crate::util::batch::grovedb_op_batch::GroveDbOpBatchV0Methods; +use crate::util::batch::GroveDbOpBatch; +use crate::util::storage_flags::StorageFlags; +use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; +use dpp::fee::refund_owner::{RefundOwner, RefundOwnersByIdentifier}; +use dpp::identifier::Identifier; +use dpp::version::PlatformVersion; +use grovedb::batch::QualifiedGroveDbOp; +use grovedb::Element; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes::{ + NoStorageRemoval, SectionedStorageRemoval, +}; +use grovedb_costs::OperationCost; +use grovedb_path::SubtreePath; +use platform_version::version::drive_versions::DriveVersion; + +const OWNER_ID: [u8; 32] = [0x22; 32]; +const CONTRACT_ID: [u8; 32] = [0x11; 32]; +const KEY: &[u8] = b"fix-08-flagged-item"; + +/// The latest drive version with both batch apply slots moved to version 1. +fn typed_drive_version() -> DriveVersion { + let mut drive_version = PlatformVersion::latest().drive.clone(); + drive_version.grove_methods.apply.grove_apply_batch = 1; + drive_version.grove_methods.apply.grove_apply_partial_batch = 1; + drive_version +} + +/// Inserts a flagged 200 byte item under the misc tree with the shipped +/// generation, which accepts the historical flag types. +fn insert_flagged_item(drive: &Drive, flags: &StorageFlags, drive_version: &DriveVersion) { + let mut batch = GroveDbOpBatch::new(); + batch.add_insert( + misc_path_vec(), + KEY.to_vec(), + Element::new_item_with_flags(vec![7u8; 200], flags.to_some_element_flags()), + ); + drive + .grove_apply_batch_with_add_costs(batch, false, None, &mut vec![], drive_version) + .expect("should insert the flagged item"); +} + +fn delete_item( + drive: &Drive, + drive_version: &DriveVersion, +) -> Result, Error> { + let mut batch = GroveDbOpBatch::new(); + batch.add_delete(misc_path_vec(), KEY.to_vec()); + let mut drive_operations = vec![]; + drive.grove_apply_batch_with_add_costs( + batch, + false, + None, + &mut drive_operations, + drive_version, + )?; + Ok(drive_operations) +} + +fn root_hash(drive: &Drive, drive_version: &DriveVersion) -> [u8; 32] { + drive + .grove + .root_hash(None, &drive_version.grove_version) + .unwrap() + .expect("should get root hash") +} + +fn sectioned_removal(cost: &OperationCost) -> &StorageRemovedBytes { + &cost.storage_cost.removed_bytes +} + +#[test] +fn should_delete_an_identity_owned_item_identically_under_both_generations() { + let shipped = PlatformVersion::latest().drive.clone(); + let typed = typed_drive_version(); + let flags = StorageFlags::SingleEpochOwned(3, OWNER_ID); + + let drive_v0 = setup_drive_with_initial_state_structure(None); + insert_flagged_item(&drive_v0, &flags, &shipped); + let drive_v1 = setup_drive_with_initial_state_structure(None); + insert_flagged_item(&drive_v1, &flags, &shipped); + assert_eq!( + root_hash(&drive_v0, &shipped), + root_hash(&drive_v1, &shipped) + ); + + let ops_v0 = delete_item(&drive_v0, &shipped).expect("v0 should delete"); + let ops_v1 = delete_item(&drive_v1, &typed).expect("v1 should delete"); + + assert_eq!(root_hash(&drive_v0, &shipped), root_hash(&drive_v1, &typed)); + + let [LowLevelDriveOperation::CalculatedCostOperation(cost_v0)] = ops_v0.as_slice() else { + panic!("v0 should push one plain cost operation, got {:?}", ops_v0); + }; + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { + cost: cost_v1, + refund_owners, + }] = ops_v1.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops_v1); + }; + + assert_eq!(cost_v0, cost_v1, "only the operation shape differs"); + let SectionedStorageRemoval(removal) = sectioned_removal(cost_v1) else { + panic!("an owned delete sections its removed bytes"); + }; + assert_eq!(removal.keys().copied().collect::>(), vec![OWNER_ID]); + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([( + OWNER_ID, + RefundOwner::Identity(Identifier::from(OWNER_ID)) + )]) + ); +} + +#[test] +fn should_record_the_bucket_owner_when_a_bucket_owned_item_is_deleted_under_v1() { + let typed = typed_drive_version(); + let owner = RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position: 7, + }; + let flags = StorageFlags::new_single_epoch_for_owner(3, Some(owner)); + + let drive = setup_drive_with_initial_state_structure(None); + // the insert carries the flags through unchanged: no closure parses + // flags on a plain insert + insert_flagged_item(&drive, &flags, &typed); + + let ops = delete_item(&drive, &typed).expect("v1 should delete a bucket owned item"); + + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { + cost, + refund_owners, + }] = ops.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops); + }; + let SectionedStorageRemoval(removal) = sectioned_removal(cost) else { + panic!("an owned delete sections its removed bytes"); + }; + assert_eq!( + removal.keys().copied().collect::>(), + vec![owner.removal_key()] + ); + let bytes_removed: u32 = removal[&owner.removal_key()].values().sum(); + assert!(bytes_removed >= 200, "the item value is at least 200 bytes"); + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([(owner.removal_key(), owner)]) + ); +} + +/// A replace whose new flags name a different owner transfers the bytes. +/// GroveDB prices a replace with the old flags still attached, so a smaller +/// payload in a later epoch takes the shrinking path, where the crate returns +/// the old single epoch flags untouched. The typed generation must still +/// transfer: the bytes freed by the shrink refund the bucket that paid for +/// them, and the delete afterwards refunds the identity, not the bucket. +#[test] +fn should_transfer_a_bucket_owned_item_to_an_identity_on_a_later_epoch_shrinking_replace() { + let typed = typed_drive_version(); + let bucket = RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position: 7, + }; + let identity = RefundOwner::Identity(Identifier::from(OWNER_ID)); + + let drive = setup_drive_with_initial_state_structure(None); + insert_flagged_item( + &drive, + &StorageFlags::new_single_epoch_for_owner(1, Some(bucket)), + &typed, + ); + + let mut batch = GroveDbOpBatch::new(); + batch.push(QualifiedGroveDbOp::replace_op( + misc_path_vec(), + KEY.to_vec(), + Element::new_item_with_flags( + vec![7u8; 100], + StorageFlags::new_single_epoch_for_owner(2, Some(identity)).to_some_element_flags(), + ), + )); + let mut replace_operations = vec![]; + drive + .grove_apply_batch_with_add_costs(batch, false, None, &mut replace_operations, &typed) + .expect("v1 should replace across kinds"); + + // the bytes freed by the shrink were paid for by the bucket + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { + cost: shrink_cost, + refund_owners: shrink_owners, + }] = replace_operations.as_slice() + else { + panic!( + "v1 should push one typed cost operation, got {:?}", + replace_operations + ); + }; + let SectionedStorageRemoval(shrink_removal) = sectioned_removal(shrink_cost) else { + panic!("a shrinking replace sections its removed bytes"); + }; + assert_eq!( + shrink_removal.keys().copied().collect::>(), + vec![bucket.removal_key()] + ); + assert_eq!( + *shrink_owners, + RefundOwnersByIdentifier::from([(bucket.removal_key(), bucket)]) + ); + + let stored = drive + .grove + .get( + SubtreePath::from(misc_path_vec().as_slice()), + KEY, + None, + &typed.grove_version, + ) + .unwrap() + .expect("item should exist"); + let flags = StorageFlags::map_some_element_flags_ref(stored.get_flags()) + .expect("flags should decode") + .expect("flags should be present"); + assert_eq!(flags, StorageFlags::SingleEpochOwned(1, OWNER_ID)); + + let ops = delete_item(&drive, &typed).expect("v1 should delete"); + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { + cost, + refund_owners, + }] = ops.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops); + }; + let SectionedStorageRemoval(removal) = sectioned_removal(cost) else { + panic!("an owned delete sections its removed bytes"); + }; + assert_eq!(removal.keys().copied().collect::>(), vec![OWNER_ID]); + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([(OWNER_ID, identity)]) + ); +} + +/// Replaces `KEY` under the typed generation with `payload_len` bytes and +/// the given flags, returning the pushed operations. +fn replace_item( + drive: &Drive, + payload_len: usize, + flags: &StorageFlags, + drive_version: &DriveVersion, +) -> Result, Error> { + let mut batch = GroveDbOpBatch::new(); + batch.push(QualifiedGroveDbOp::replace_op( + misc_path_vec(), + KEY.to_vec(), + Element::new_item_with_flags(vec![7u8; payload_len], flags.to_some_element_flags()), + )); + let mut drive_operations = vec![]; + drive.grove_apply_batch_with_add_costs( + batch, + false, + None, + &mut drive_operations, + drive_version, + )?; + Ok(drive_operations) +} + +fn stored_flags(drive: &Drive, drive_version: &DriveVersion) -> StorageFlags { + let stored = drive + .grove + .get( + SubtreePath::from(misc_path_vec().as_slice()), + KEY, + None, + &drive_version.grove_version, + ) + .unwrap() + .expect("item should exist"); + StorageFlags::map_some_element_flags_ref(stored.get_flags()) + .expect("flags should decode") + .expect("flags should be present") +} + +/// GroveDB prices a replace with the old flags attached and asks the update +/// closure whether the flags changed. A same-epoch replace across kinds +/// leaves the proposed flags as they are but changes the header width (35 +/// bytes for an identity, 37 for a bucket), so the closure must report a +/// change or GroveDB keeps a stale price and rejects the write. +#[test] +fn should_replace_across_kinds_in_the_same_epoch_when_the_header_width_changes() { + let typed = typed_drive_version(); + let bucket = RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position: 7, + }; + let identity = RefundOwner::Identity(Identifier::from(OWNER_ID)); + + // identity to bucket, payload one byte bigger + let drive = setup_drive_with_initial_state_structure(None); + insert_flagged_item( + &drive, + &StorageFlags::new_single_epoch_for_owner(1, Some(identity)), + &typed, + ); + let ops = replace_item( + &drive, + 201, + &StorageFlags::new_single_epoch_for_owner(1, Some(bucket)), + &typed, + ) + .expect("identity to bucket replace should apply"); + assert!(matches!( + ops.as_slice(), + [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { .. }] + )); + assert_eq!( + stored_flags(&drive, &typed), + StorageFlags::SingleEpochContractBucket(1, CONTRACT_ID, 7) + ); + let ops = delete_item(&drive, &typed).expect("v1 should delete"); + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { refund_owners, .. }] = + ops.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops); + }; + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([(bucket.removal_key(), bucket)]) + ); + + // bucket to identity, payload one byte bigger: priced with the old + // header it looks like growth, with the new header it is a one byte + // shrink, and the freed byte belongs to the bucket that paid for it + let drive = setup_drive_with_initial_state_structure(None); + insert_flagged_item( + &drive, + &StorageFlags::new_single_epoch_for_owner(1, Some(bucket)), + &typed, + ); + let ops = replace_item( + &drive, + 201, + &StorageFlags::new_single_epoch_for_owner(1, Some(identity)), + &typed, + ) + .expect("bucket to identity replace should apply"); + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { + cost, + refund_owners, + }] = ops.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops); + }; + let SectionedStorageRemoval(removal) = sectioned_removal(cost) else { + panic!("the one byte shrink is sectioned, got {:?}", cost); + }; + assert_eq!( + removal.keys().copied().collect::>(), + vec![bucket.removal_key()] + ); + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([(bucket.removal_key(), bucket)]) + ); + assert_eq!( + stored_flags(&drive, &typed), + StorageFlags::SingleEpochOwned(1, OWNER_ID) + ); +} + +/// A later-epoch replace whose payload shrinks by exactly the header growth +/// of the new owner kind nets to zero bytes. The first pricing pass sees a +/// shrink and transfers ownership; the recalculated pass sees the same size +/// and must resolve ownership the same way, or GroveDB's update loop +/// alternates between the two answers and never converges. +#[test] +fn should_converge_when_a_later_epoch_replace_nets_to_the_same_size_across_kinds() { + let typed = typed_drive_version(); + let bucket = RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position: 7, + }; + let identity = RefundOwner::Identity(Identifier::from(OWNER_ID)); + + let drive = setup_drive_with_initial_state_structure(None); + insert_flagged_item( + &drive, + &StorageFlags::new_single_epoch_for_owner(1, Some(identity)), + &typed, + ); + let ops = replace_item( + &drive, + 198, + &StorageFlags::new_single_epoch_for_owner(2, Some(bucket)), + &typed, + ) + .expect("a net same size replace across kinds should apply"); + + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { cost, .. }] = + ops.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops); + }; + assert_eq!(cost.storage_cost.added_bytes, 0, "nothing was added on net"); + assert_eq!( + *sectioned_removal(cost), + NoStorageRemoval, + "nothing was freed on net" + ); + // the bytes keep their original epoch and move to the new owner + assert_eq!( + stored_flags(&drive, &typed), + StorageFlags::SingleEpochContractBucket(1, CONTRACT_ID, 7) + ); + + let ops = delete_item(&drive, &typed).expect("v1 should delete"); + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { refund_owners, .. }] = + ops.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops); + }; + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([(bucket.removal_key(), bucket)]) + ); +} + +#[test] +fn should_fail_closed_when_a_bucket_owned_item_is_deleted_under_v0() { + let shipped = PlatformVersion::latest().drive.clone(); + let typed = typed_drive_version(); + let owner = RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position: 7, + }; + let flags = StorageFlags::new_single_epoch_for_owner(3, Some(owner)); + + let drive = setup_drive_with_initial_state_structure(None); + insert_flagged_item(&drive, &flags, &typed); + let before = root_hash(&drive, &shipped); + + let mut batch = GroveDbOpBatch::new(); + batch.add_delete(misc_path_vec(), KEY.to_vec()); + let mut drive_operations = vec![]; + let error = drive + .grove_apply_batch_with_add_costs(batch, false, None, &mut drive_operations, &shipped) + .expect_err("v0 cannot attribute bucket owned bytes"); + + assert!( + error + .to_string() + .contains("unknown storage flags serialization"), + "unexpected error: {}", + error + ); + assert_eq!(root_hash(&drive, &shipped), before, "nothing was applied"); + // the read cost incurred before the failure is still pushed, as for any + // failed grove operation, but no removed bytes and no owners reach the + // fee path + for op in &drive_operations { + let LowLevelDriveOperation::CalculatedCostOperation(cost) = op else { + panic!("only plain read costs may be pushed, got {:?}", op); + }; + assert_eq!(*sectioned_removal(cost), NoStorageRemoval); + } + + // the item is still there and version 1 removes it with the recorded owner + let ops = delete_item(&drive, &typed).expect("v1 should delete"); + assert!(matches!( + ops.as_slice(), + [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { .. }] + )); +} + +#[test] +fn should_reject_an_unknown_generation() { + let drive = setup_drive_with_initial_state_structure(None); + let mut drive_version = PlatformVersion::latest().drive.clone(); + drive_version.grove_methods.apply.grove_apply_batch = 2; + + let mut batch = GroveDbOpBatch::new(); + batch.add_delete(misc_path_vec(), KEY.to_vec()); + + let error = drive + .grove_apply_batch_with_add_costs(batch, false, None, &mut vec![], &drive_version) + .expect_err("version 2 does not exist"); + + assert!(matches!( + error, + Error::Drive(crate::error::drive::DriveError::UnknownVersionMismatch { .. }) + )); +} diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/v0/mod.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/v0/mod.rs index e44000512d7..d41c3c9b289 100644 --- a/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/v0/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/v0/mod.rs @@ -6,9 +6,13 @@ use crate::query::GroveError; use crate::util::batch::grovedb_op_batch::GroveDbOpBatchV0Methods; use crate::util::batch::GroveDbOpBatch; use crate::util::grove_operations::push_drive_operation_result; -use crate::util::storage_flags::StorageFlags; +// The shipped generation is bound to the crate's flags type on purpose: it +// splits and combines exactly the four historical flag types and rejects any +// other type byte, so bytes owned by a contract credit bucket can never be +// sectioned by this generation. The typed flags live in version 1. use grovedb::batch::{BatchApplyOptions, QualifiedGroveDbOp}; use grovedb::TransactionArg; +use grovedb_epoch_based_storage_flags::StorageFlags; use platform_version::version::drive_versions::DriveVersion; impl Drive { diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/v1/mod.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/v1/mod.rs new file mode 100644 index 00000000000..0c7918b579a --- /dev/null +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_batch_with_add_costs/v1/mod.rs @@ -0,0 +1,123 @@ +use crate::drive::Drive; +use crate::error::drive::DriveError; +use crate::error::Error; +use crate::fees::op::LowLevelDriveOperation; +use crate::query::GroveError; +use crate::util::batch::grovedb_op_batch::GroveDbOpBatchV0Methods; +use crate::util::batch::GroveDbOpBatch; +use crate::util::grove_operations::push_drive_operation_result_with_refund_owners; +use crate::util::storage_flags::StorageFlags; +use dpp::fee::refund_owner::RefundOwnersByIdentifier; +use grovedb::batch::{BatchApplyOptions, QualifiedGroveDbOp}; +use grovedb::TransactionArg; +use platform_version::version::drive_versions::DriveVersion; + +impl Drive { + /// Applies the given groveDB operations batch and passes the costs to + /// `push_drive_operation_result_with_refund_owners`. + /// + /// This generation splits removed bytes with the typed storage flags: + /// every owned removal is sectioned under the owner's carrier key and + /// the owner is recorded next to it, so the cost operation it pushes + /// carries the recorded owners for the fee decoder to route. It accepts + /// contract bucket owned flags, which the previous generation rejects. + pub(super) fn grove_apply_batch_with_add_costs_v1( + &self, + ops: GroveDbOpBatch, + validate: bool, + transaction: TransactionArg, + drive_operations: &mut Vec, + drive_version: &DriveVersion, + ) -> Result<(), Error> { + if ops.is_empty() { + return Err(Error::Drive(DriveError::BatchIsEmpty( + "batch is empty when trying to apply batch with add costs".to_string(), + ))); + } + + if self.config.batching_consistency_verification { + let consistency_results = + QualifiedGroveDbOp::verify_consistency_of_operations(&ops.operations); + if !consistency_results.is_empty() { + tracing::error!( + ?consistency_results, + "grovedb consistency verification failed" + ); + return Err(Error::Drive(DriveError::GroveDBInsertion( + "insertion order error", + ))); + } + } + + // Clone ops only if we log them + #[cfg(feature = "grovedb_operations_logging")] + let maybe_params_for_logs = if tracing::event_enabled!(target: "drive_grovedb_operations", tracing::Level::TRACE) + { + let root_hash = self + .grove + .root_hash(transaction, &drive_version.grove_version) + .unwrap() + .map_err(Error::from)?; + + Some((ops.clone(), root_hash)) + } else { + None + }; + + let mut refund_owners = RefundOwnersByIdentifier::new(); + + let cost_context = self.grove.apply_batch_with_element_flags_update( + ops.operations, + Some(BatchApplyOptions { + validate_insertion_does_not_override: validate, + validate_insertion_does_not_override_tree: validate, + disable_operation_consistency_check: !self.config.batching_consistency_verification, + base_root_storage_is_free: true, + batch_pause_height: None, + }), + |cost, old_flags, new_flags| { + StorageFlags::update_element_flags_typed(cost, old_flags, new_flags) + .map_err(|e| GroveError::JustInTimeElementFlagsClientError(e.to_string())) + }, + |flags, removed_key_bytes, removed_value_bytes| { + StorageFlags::split_removal_bytes_typed( + flags, + removed_key_bytes, + removed_value_bytes, + &mut refund_owners, + ) + .map_err(|e| GroveError::SplitRemovalBytesClientError(e.to_string())) + }, + transaction, + &drive_version.grove_version, + ); + + #[cfg(feature = "grovedb_operations_logging")] + if tracing::event_enabled!(target: "drive_grovedb_operations", tracing::Level::TRACE) + && cost_context.value.is_ok() + { + if let Some((ops, previous_root_hash)) = maybe_params_for_logs { + let root_hash = self + .grove + .root_hash(transaction, &drive_version.grove_version) + .unwrap() + .map_err(Error::from)?; + + tracing::trace!( + target: "drive_grovedb_operations", + ?ops, + ?root_hash, + ?previous_root_hash, + is_transactional = transaction.is_some(), + "grovedb batch applied", + ); + } + } + + push_drive_operation_result_with_refund_owners( + cost_context, + refund_owners, + drive_operations, + ) + } +} diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/mod.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/mod.rs index 7ba94976f5b..5c6ebb1a430 100644 --- a/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/mod.rs @@ -1,4 +1,8 @@ mod v0; +mod v1; + +#[cfg(test)] +mod tests; use crate::util::batch::GroveDbOpBatch; @@ -16,6 +20,11 @@ use grovedb_costs::OperationCost; impl Drive { /// Applies the given groveDB operations batch, gets and passes the costs to `push_drive_operation_result`. /// + /// Version 0 splits removed bytes with the identity-only storage flags + /// and pushes a plain cost operation. Version 1 splits with the typed + /// storage flags and pushes a cost operation that carries the recorded + /// refund owner of every sectioned removal. + /// /// # Parameters /// * `ops`: The batch of groveDB operations to retrieve costs for. /// * `validate`: Specifies whether to validate that insertions do not override existing entries. @@ -49,9 +58,17 @@ impl Drive { drive_operations, drive_version, ), + 1 => self.grove_apply_partial_batch_with_add_costs_v1( + ops, + validate, + transaction, + add_on_operations, + drive_operations, + drive_version, + ), version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "grove_apply_partial_batch_with_add_costs".to_string(), - known_versions: vec![0], + known_versions: vec![0, 1], received: version, })), } diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/tests.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/tests.rs new file mode 100644 index 00000000000..2fba1ce3ba6 --- /dev/null +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/tests.rs @@ -0,0 +1,195 @@ +//! Tests that run both partial batch apply generations through the +//! dispatcher. Version 1 is selected only by a test-built drive version. + +use crate::drive::system::misc_path_vec; +use crate::drive::Drive; +use crate::error::Error; +use crate::fees::op::LowLevelDriveOperation; +use crate::util::batch::grovedb_op_batch::GroveDbOpBatchV0Methods; +use crate::util::batch::GroveDbOpBatch; +use crate::util::storage_flags::StorageFlags; +use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; +use dpp::fee::refund_owner::{RefundOwner, RefundOwnersByIdentifier}; +use dpp::identifier::Identifier; +use dpp::version::PlatformVersion; +use grovedb::Element; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes::{ + NoStorageRemoval, SectionedStorageRemoval, +}; +use platform_version::version::drive_versions::DriveVersion; + +const OWNER_ID: [u8; 32] = [0x33; 32]; +const CONTRACT_ID: [u8; 32] = [0x44; 32]; +const KEY: &[u8] = b"fix-08-partial-flagged-item"; + +fn typed_drive_version() -> DriveVersion { + let mut drive_version = PlatformVersion::latest().drive.clone(); + drive_version.grove_methods.apply.grove_apply_batch = 1; + drive_version.grove_methods.apply.grove_apply_partial_batch = 1; + drive_version +} + +fn insert_flagged_item(drive: &Drive, flags: &StorageFlags, drive_version: &DriveVersion) { + let mut batch = GroveDbOpBatch::new(); + batch.add_insert( + misc_path_vec(), + KEY.to_vec(), + Element::new_item_with_flags(vec![9u8; 150], flags.to_some_element_flags()), + ); + drive + .grove_apply_batch_with_add_costs(batch, false, None, &mut vec![], drive_version) + .expect("should insert the flagged item"); +} + +fn delete_item_partially( + drive: &Drive, + drive_version: &DriveVersion, +) -> Result, Error> { + let mut batch = GroveDbOpBatch::new(); + batch.add_delete(misc_path_vec(), KEY.to_vec()); + let mut drive_operations = vec![]; + drive.grove_apply_partial_batch_with_add_costs( + batch, + false, + None, + |_cost, _ops_by_level| Ok(vec![]), + &mut drive_operations, + drive_version, + )?; + Ok(drive_operations) +} + +#[test] +fn should_push_the_same_cost_for_an_identity_owned_delete_under_both_generations() { + let shipped = PlatformVersion::latest().drive.clone(); + let typed = typed_drive_version(); + let flags = StorageFlags::SingleEpochOwned(2, OWNER_ID); + + let drive_v0 = setup_drive_with_initial_state_structure(None); + insert_flagged_item(&drive_v0, &flags, &shipped); + let drive_v1 = setup_drive_with_initial_state_structure(None); + insert_flagged_item(&drive_v1, &flags, &shipped); + + let ops_v0 = delete_item_partially(&drive_v0, &shipped).expect("v0 should delete"); + let ops_v1 = delete_item_partially(&drive_v1, &typed).expect("v1 should delete"); + + let [LowLevelDriveOperation::CalculatedCostOperation(cost_v0)] = ops_v0.as_slice() else { + panic!("v0 should push one plain cost operation, got {:?}", ops_v0); + }; + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { + cost: cost_v1, + refund_owners, + }] = ops_v1.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops_v1); + }; + assert_eq!(cost_v0, cost_v1); + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([( + OWNER_ID, + RefundOwner::Identity(Identifier::from(OWNER_ID)) + )]) + ); +} + +#[test] +fn should_record_the_bucket_owner_when_a_bucket_owned_item_is_deleted_under_v1() { + let typed = typed_drive_version(); + let owner = RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position: 1, + }; + let flags = StorageFlags::new_single_epoch_for_owner(2, Some(owner)); + + let drive = setup_drive_with_initial_state_structure(None); + insert_flagged_item(&drive, &flags, &typed); + + let ops = delete_item_partially(&drive, &typed).expect("v1 should delete"); + let [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { + cost, + refund_owners, + }] = ops.as_slice() + else { + panic!("v1 should push one typed cost operation, got {:?}", ops); + }; + let SectionedStorageRemoval(removal) = &cost.storage_cost.removed_bytes else { + panic!("an owned delete sections its removed bytes"); + }; + assert_eq!( + removal.keys().copied().collect::>(), + vec![owner.removal_key()] + ); + assert_eq!( + *refund_owners, + RefundOwnersByIdentifier::from([(owner.removal_key(), owner)]) + ); +} + +/// The shipped partial batch generation is bound to the crate's flags type, +/// which does not know the bucket type bytes, so a bucket-owned removal fails +/// the batch: nothing is applied and no cost operation is pushed. Without +/// this, the bucket's carrier key would reach the shipped fee decoder as a +/// plain sectioned removal and be read as an identity. +#[test] +fn should_fail_closed_when_a_bucket_owned_item_is_deleted_under_v0() { + let shipped = PlatformVersion::latest().drive.clone(); + let typed = typed_drive_version(); + let owner = RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position: 1, + }; + let flags = StorageFlags::new_single_epoch_for_owner(2, Some(owner)); + + let drive = setup_drive_with_initial_state_structure(None); + insert_flagged_item(&drive, &flags, &typed); + let before = drive + .grove + .root_hash(None, &shipped.grove_version) + .unwrap() + .expect("should get root hash"); + + let mut batch = GroveDbOpBatch::new(); + batch.add_delete(misc_path_vec(), KEY.to_vec()); + let mut drive_operations = vec![]; + let error = drive + .grove_apply_partial_batch_with_add_costs( + batch, + false, + None, + |_cost, _ops_by_level| Ok(vec![]), + &mut drive_operations, + &shipped, + ) + .expect_err("v0 cannot attribute bucket owned bytes"); + + assert!( + error + .to_string() + .contains("unknown storage flags serialization"), + "unexpected error: {}", + error + ); + // the read cost incurred before the failure is still pushed, as for any + // failed grove operation, but no removed bytes and no owners reach the + // fee path + for op in &drive_operations { + let LowLevelDriveOperation::CalculatedCostOperation(cost) = op else { + panic!("only plain read costs may be pushed, got {:?}", op); + }; + assert_eq!(cost.storage_cost.removed_bytes, NoStorageRemoval); + } + let after = drive + .grove + .root_hash(None, &shipped.grove_version) + .unwrap() + .expect("should get root hash"); + assert_eq!(after, before, "nothing was applied"); + + // the item is still there and version 1 can still remove it + let ops = delete_item_partially(&drive, &typed).expect("v1 should delete"); + assert!(matches!( + ops.as_slice(), + [LowLevelDriveOperation::CalculatedCostOperationWithRefundOwners { .. }] + )); +} diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/v0/mod.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/v0/mod.rs index a736817ab8d..5127e7ee012 100644 --- a/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/v0/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/v0/mod.rs @@ -6,12 +6,16 @@ use crate::query::GroveError; use crate::util::batch::grovedb_op_batch::GroveDbOpBatchV0Methods; use crate::util::batch::GroveDbOpBatch; use crate::util::grove_operations::push_drive_operation_result; -use crate::util::storage_flags::{MergingOwnersStrategy, StorageFlags}; +// The shipped generation is bound to the crate's flags type on purpose: it +// splits and combines exactly the four historical flag types and rejects any +// other type byte, so bytes owned by a contract credit bucket can never be +// sectioned by this generation. The typed flags live in version 1. use grovedb::batch::{BatchApplyOptions, OpsByLevelPath, QualifiedGroveDbOp}; use grovedb::TransactionArg; use grovedb_costs::storage_cost::removal::StorageRemovedBytes::BasicStorageRemoval; use grovedb_costs::storage_cost::transition::OperationStorageTransitionType; use grovedb_costs::OperationCost; +use grovedb_epoch_based_storage_flags::{MergingOwnersStrategy, StorageFlags}; use platform_version::version::drive_versions::DriveVersion; impl Drive { diff --git a/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/v1/mod.rs b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/v1/mod.rs new file mode 100644 index 00000000000..a181e93fc6d --- /dev/null +++ b/packages/rs-drive/src/util/grove_operations/grove_apply_partial_batch_with_add_costs/v1/mod.rs @@ -0,0 +1,85 @@ +use crate::drive::Drive; +use crate::error::drive::DriveError; +use crate::error::Error; +use crate::fees::op::LowLevelDriveOperation; +use crate::query::GroveError; +use crate::util::batch::grovedb_op_batch::GroveDbOpBatchV0Methods; +use crate::util::batch::GroveDbOpBatch; +use crate::util::grove_operations::push_drive_operation_result_with_refund_owners; +use crate::util::storage_flags::StorageFlags; +use dpp::fee::refund_owner::RefundOwnersByIdentifier; +use grovedb::batch::{BatchApplyOptions, OpsByLevelPath, QualifiedGroveDbOp}; +use grovedb::TransactionArg; +use grovedb_costs::OperationCost; +use platform_version::version::drive_versions::DriveVersion; + +impl Drive { + /// Applies the given groveDB operations batch with add-on operations and + /// passes the costs to `push_drive_operation_result_with_refund_owners`. + /// + /// This generation splits removed bytes with the typed storage flags and + /// records the owner of every owned removal on the cost operation it + /// pushes. The flag update closure is the typed one as well, which also + /// removes the previous generation's inline copy of that logic. + pub(super) fn grove_apply_partial_batch_with_add_costs_v1( + &self, + ops: GroveDbOpBatch, + validate: bool, + transaction: TransactionArg, + add_on_operations: impl FnMut( + &OperationCost, + &Option, + ) -> Result, GroveError>, + drive_operations: &mut Vec, + drive_version: &DriveVersion, + ) -> Result<(), Error> { + if ops.is_empty() { + return Err(Error::Drive(DriveError::BatchIsEmpty( + "batch is empty when trying to apply partial batch with add costs".to_string(), + ))); + } + if self.config.batching_consistency_verification { + let consistency_results = + QualifiedGroveDbOp::verify_consistency_of_operations(&ops.operations); + if !consistency_results.is_empty() { + return Err(Error::Drive(DriveError::GroveDBInsertion( + "insertion order error", + ))); + } + } + + let mut refund_owners = RefundOwnersByIdentifier::new(); + + let cost_context = self.grove.apply_partial_batch_with_element_flags_update( + ops.operations, + Some(BatchApplyOptions { + validate_insertion_does_not_override: validate, + validate_insertion_does_not_override_tree: validate, + disable_operation_consistency_check: false, + base_root_storage_is_free: true, + batch_pause_height: None, + }), + |cost, old_flags, new_flags| { + StorageFlags::update_element_flags_typed(cost, old_flags, new_flags) + .map_err(|e| GroveError::JustInTimeElementFlagsClientError(e.to_string())) + }, + |flags, removed_key_bytes, removed_value_bytes| { + StorageFlags::split_removal_bytes_typed( + flags, + removed_key_bytes, + removed_value_bytes, + &mut refund_owners, + ) + .map_err(|e| GroveError::SplitRemovalBytesClientError(e.to_string())) + }, + add_on_operations, + transaction, + &drive_version.grove_version, + ); + push_drive_operation_result_with_refund_owners( + cost_context, + refund_owners, + drive_operations, + ) + } +} diff --git a/packages/rs-drive/src/util/grove_operations/mod.rs b/packages/rs-drive/src/util/grove_operations/mod.rs index ef3b826c99e..5b8f899c2d8 100644 --- a/packages/rs-drive/src/util/grove_operations/mod.rs +++ b/packages/rs-drive/src/util/grove_operations/mod.rs @@ -218,7 +218,10 @@ use grovedb::{EstimatedLayerInformation, MaybeTree, TreeType}; use crate::error::Error; use crate::fees::op::LowLevelDriveOperation; -use crate::fees::op::LowLevelDriveOperation::CalculatedCostOperation; +use crate::fees::op::LowLevelDriveOperation::{ + CalculatedCostOperation, CalculatedCostOperationWithRefundOwners, +}; +use dpp::fee::fee_result::refunds::RefundOwnersByIdentifier; use grovedb::Error as GroveError; @@ -237,6 +240,28 @@ fn push_drive_operation_result( value.map_err(Error::from) } +/// Pushes an operation's `OperationCost` together with the refund owners +/// recorded while its removed bytes were split, and returns the operation's +/// return value. +/// +/// The owners ride with the cost as `CalculatedCostOperationWithRefundOwners` +/// so that the fee decoder reads each carrier key's owner from the record +/// rather than inferring it from the key. +fn push_drive_operation_result_with_refund_owners( + cost_context: CostContext>, + refund_owners: RefundOwnersByIdentifier, + drive_operations: &mut Vec, +) -> Result { + let CostContext { value, cost } = cost_context; + if !cost.is_nothing() { + drive_operations.push(CalculatedCostOperationWithRefundOwners { + cost, + refund_owners, + }); + } + value.map_err(Error::from) +} + /// Pushes an operation's `OperationCost` to `drive_operations` given its `CostContext` /// if `drive_operations` is given. Returns the operation's return value. fn push_drive_operation_result_optional( diff --git a/packages/rs-drive/src/util/storage_flags.rs b/packages/rs-drive/src/util/storage_flags.rs deleted file mode 100644 index d933d14dac3..00000000000 --- a/packages/rs-drive/src/util/storage_flags.rs +++ /dev/null @@ -1,3 +0,0 @@ -pub use grovedb_epoch_based_storage_flags::{ - MergingOwnersStrategy, StorageFlags, SINGLE_EPOCH_FLAGS_SIZE, -}; diff --git a/packages/rs-drive/src/util/storage_flags/codec.rs b/packages/rs-drive/src/util/storage_flags/codec.rs new file mode 100644 index 00000000000..3846fc50534 --- /dev/null +++ b/packages/rs-drive/src/util/storage_flags/codec.rs @@ -0,0 +1,321 @@ +//! Serialization of storage flags. +//! +//! Types 0 to 3 are produced and parsed by the pinned +//! `grovedb_epoch_based_storage_flags` crate so that their bytes cannot +//! drift from what is on chain. Types 4 and 5 are the contract bucket +//! variants defined here. + +use super::{ + BaseEpoch, BytesAddedInEpoch, ContractId, CrateStorageFlags, EpochIndex, StorageFlags, + CONTRACT_BUCKET_OWNER_SIZE, OWNER_ID_SIZE, SINGLE_EPOCH_FLAGS_SIZE, +}; +use dpp::fee::refund_owner::ContractCreditBucketPosition; +use grovedb::ElementFlags; +use grovedb_epoch_based_storage_flags::error::StorageFlagsError; +use integer_encoding::VarInt; +use std::borrow::Cow; +use std::collections::BTreeMap; + +/// Type byte of the single epoch contract bucket variant +const SINGLE_EPOCH_CONTRACT_BUCKET_TYPE: u8 = 4; + +/// Type byte of the multi epoch contract bucket variant +const MULTI_EPOCH_CONTRACT_BUCKET_TYPE: u8 = 5; + +/// Size of the fixed header of both contract bucket variants: type byte, +/// contract id, position and base epoch +const CONTRACT_BUCKET_HEADER_SIZE: usize = + (SINGLE_EPOCH_FLAGS_SIZE + CONTRACT_BUCKET_OWNER_SIZE) as usize; + +/// Smallest possible epoch map entry: two epoch index bytes and one varint byte +const MIN_EPOCH_MAP_ENTRY_SIZE: usize = 3; + +impl StorageFlags { + /// Serialize storage flags + pub fn serialize(&self) -> Vec { + match self { + StorageFlags::SingleEpochContractBucket(base_epoch, contract_id, position) => { + let mut buffer = Vec::with_capacity(CONTRACT_BUCKET_HEADER_SIZE); + Self::append_contract_bucket_header( + &mut buffer, + SINGLE_EPOCH_CONTRACT_BUCKET_TYPE, + contract_id, + *position, + *base_epoch, + ); + buffer + } + StorageFlags::MultiEpochContractBucket(base_epoch, epochs, contract_id, position) => { + let mut buffer = Vec::with_capacity( + CONTRACT_BUCKET_HEADER_SIZE + epochs.len() * MIN_EPOCH_MAP_ENTRY_SIZE, + ); + Self::append_contract_bucket_header( + &mut buffer, + MULTI_EPOCH_CONTRACT_BUCKET_TYPE, + contract_id, + *position, + *base_epoch, + ); + Self::append_epoch_map(&mut buffer, epochs); + buffer + } + // the two historical multi epoch variants write the crate's + // layout without cloning their epoch map; equality with the + // crate's bytes is pinned by test + StorageFlags::MultiEpoch(base_epoch, epochs) => { + let mut buffer = Vec::with_capacity( + SINGLE_EPOCH_FLAGS_SIZE as usize + epochs.len() * MIN_EPOCH_MAP_ENTRY_SIZE, + ); + buffer.push(self.type_byte()); + buffer.extend_from_slice(&base_epoch.to_be_bytes()); + Self::append_epoch_map(&mut buffer, epochs); + buffer + } + StorageFlags::MultiEpochOwned(base_epoch, epochs, owner_id) => { + let mut buffer = Vec::with_capacity( + (SINGLE_EPOCH_FLAGS_SIZE + OWNER_ID_SIZE) as usize + + epochs.len() * MIN_EPOCH_MAP_ENTRY_SIZE, + ); + buffer.push(self.type_byte()); + buffer.extend_from_slice(owner_id); + buffer.extend_from_slice(&base_epoch.to_be_bytes()); + Self::append_epoch_map(&mut buffer, epochs); + buffer + } + // the single epoch variants carry no map, so the crate value is + // a plain copy + StorageFlags::SingleEpoch(_) | StorageFlags::SingleEpochOwned(..) => { + self.to_crate_flags_keyed_by_removal_key().serialize() + } + } + } + + /// Serialized size of storage flags, equal to `serialize().len()` + pub fn serialized_size(&self) -> u32 { + match self { + StorageFlags::SingleEpochContractBucket(..) => CONTRACT_BUCKET_HEADER_SIZE as u32, + StorageFlags::MultiEpochContractBucket(_, epochs, ..) => { + CONTRACT_BUCKET_HEADER_SIZE as u32 + Self::epoch_map_size(epochs) + } + StorageFlags::MultiEpoch(_, epochs) => { + SINGLE_EPOCH_FLAGS_SIZE + Self::epoch_map_size(epochs) + } + StorageFlags::MultiEpochOwned(_, epochs, _) => { + SINGLE_EPOCH_FLAGS_SIZE + OWNER_ID_SIZE + Self::epoch_map_size(epochs) + } + StorageFlags::SingleEpoch(_) => SINGLE_EPOCH_FLAGS_SIZE, + StorageFlags::SingleEpochOwned(..) => SINGLE_EPOCH_FLAGS_SIZE + OWNER_ID_SIZE, + } + } + + fn append_contract_bucket_header( + buffer: &mut Vec, + type_byte: u8, + contract_id: &ContractId, + position: ContractCreditBucketPosition, + base_epoch: BaseEpoch, + ) { + buffer.push(type_byte); + buffer.extend_from_slice(contract_id); + buffer.extend_from_slice(&position.to_be_bytes()); + buffer.extend_from_slice(&base_epoch.to_be_bytes()); + } + + fn append_epoch_map(buffer: &mut Vec, epochs: &BTreeMap) { + epochs.iter().for_each(|(epoch_index, bytes_added)| { + buffer.extend_from_slice(&epoch_index.to_be_bytes()); + buffer.extend(bytes_added.encode_var_vec()); + }) + } + + fn epoch_map_size(epochs: &BTreeMap) -> u32 { + epochs + .values() + .map(|bytes_added| 2 + bytes_added.encode_var_vec().len() as u32) + .sum() + } + + fn deserialize_contract_bucket_header( + data: &[u8], + ) -> Result<(BaseEpoch, ContractId, ContractCreditBucketPosition), StorageFlagsError> { + let contract_id: ContractId = data + .get(1..33) + .and_then(|bytes| bytes.try_into().ok()) + .ok_or_else(|| { + StorageFlagsError::StorageFlagsWrongSize( + "contract bucket flags must have 32 bytes of contract id".to_string(), + ) + })?; + let position = data + .get(33..35) + .and_then(|bytes| bytes.try_into().ok()) + .map(u16::from_be_bytes) + .ok_or_else(|| { + StorageFlagsError::StorageFlagsWrongSize( + "contract bucket flags must have 2 bytes of bucket position".to_string(), + ) + })?; + let base_epoch = data + .get(35..37) + .and_then(|bytes| bytes.try_into().ok()) + .map(u16::from_be_bytes) + .ok_or_else(|| { + StorageFlagsError::StorageFlagsWrongSize( + "contract bucket flags must have 2 bytes of base epoch".to_string(), + ) + })?; + Ok((base_epoch, contract_id, position)) + } + + /// Deserialize single epoch contract bucket storage flags from bytes + fn deserialize_single_epoch_contract_bucket(data: &[u8]) -> Result { + if data.len() != CONTRACT_BUCKET_HEADER_SIZE { + return Err(StorageFlagsError::StorageFlagsWrongSize( + "single epoch contract bucket must be 37 bytes total".to_string(), + )); + } + let (base_epoch, contract_id, position) = Self::deserialize_contract_bucket_header(data)?; + Ok(StorageFlags::SingleEpochContractBucket( + base_epoch, + contract_id, + position, + )) + } + + /// Deserialize multi epoch contract bucket storage flags from bytes. + /// + /// Unlike the crate's multi epoch decoders this one is strict: the epoch + /// map must hold at least one entry and the bytes must end exactly at + /// the end of the last entry. + fn deserialize_multi_epoch_contract_bucket(data: &[u8]) -> Result { + let len = data.len(); + if len < CONTRACT_BUCKET_HEADER_SIZE + MIN_EPOCH_MAP_ENTRY_SIZE { + return Err(StorageFlagsError::StorageFlagsWrongSize( + "multi epoch contract bucket must be at least 40 bytes total".to_string(), + )); + } + let (base_epoch, contract_id, position) = Self::deserialize_contract_bucket_header(data)?; + let mut offset = CONTRACT_BUCKET_HEADER_SIZE; + let mut bytes_per_epoch: BTreeMap = BTreeMap::default(); + while offset < len { + let epoch_index = data + .get(offset..offset + 2) + .and_then(|bytes| bytes.try_into().ok()) + .map(u16::from_be_bytes) + .ok_or_else(|| { + StorageFlagsError::StorageFlagsWrongSize( + "multi epoch contract bucket must have enough bytes for epoch indexes" + .to_string(), + ) + })?; + offset += 2; + let (bytes_at_epoch, bytes_used) = data + .get(offset..) + .and_then(u32::decode_var) + .ok_or_else(|| { + StorageFlagsError::StorageFlagsWrongSize( + "multi epoch contract bucket must have enough bytes for the amount of bytes used" + .to_string(), + ) + })?; + offset += bytes_used; + bytes_per_epoch.insert(epoch_index, bytes_at_epoch); + } + if bytes_per_epoch.is_empty() { + return Err(StorageFlagsError::StorageFlagsWrongSize( + "multi epoch contract bucket must carry at least one epoch entry".to_string(), + )); + } + Ok(StorageFlags::MultiEpochContractBucket( + base_epoch, + bytes_per_epoch, + contract_id, + position, + )) + } + + /// Deserialize storage flags from bytes. + /// + /// Decodes all six variants: the four historical ones through the crate, + /// the two contract bucket ones here. An unknown type byte is an error. + pub fn deserialize(data: &[u8]) -> Result, StorageFlagsError> { + match data.first() { + None => Ok(None), + Some(&SINGLE_EPOCH_CONTRACT_BUCKET_TYPE) => { + Ok(Some(Self::deserialize_single_epoch_contract_bucket(data)?)) + } + Some(&MULTI_EPOCH_CONTRACT_BUCKET_TYPE) => { + Ok(Some(Self::deserialize_multi_epoch_contract_bucket(data)?)) + } + Some(_) => Ok(CrateStorageFlags::deserialize(data)?.map(Self::from)), + } + } + + /// Creates storage flags from a slice. + pub fn from_slice(data: &[u8]) -> Result, StorageFlagsError> { + Self::deserialize(data) + } + + /// Creates storage flags from element flags. + pub fn from_element_flags_ref(data: &ElementFlags) -> Result, StorageFlagsError> { + Self::from_slice(data.as_slice()) + } + + /// Create Storage flags from optional element flags ref + pub fn map_some_element_flags_ref( + data: &Option, + ) -> Result, StorageFlagsError> { + match data { + None => Ok(None), + Some(data) => Self::from_slice(data.as_slice()), + } + } + + /// Create Storage flags from optional element flags ref + pub fn map_cow_some_element_flags_ref( + data: &Option, + ) -> Result>, StorageFlagsError> { + match data { + None => Ok(None), + Some(data) => Self::from_slice(data.as_slice()).map(|option| option.map(Cow::Owned)), + } + } + + /// Map to owned optional element flags + pub fn map_owned_to_element_flags(maybe_storage_flags: Option) -> ElementFlags { + maybe_storage_flags + .map(|storage_flags| storage_flags.serialize()) + .unwrap_or_default() + } + + /// Map to optional element flags + pub fn map_to_some_element_flags(maybe_storage_flags: Option<&Self>) -> Option { + maybe_storage_flags.map(|storage_flags| storage_flags.serialize()) + } + + /// Map to optional element flags + pub fn map_cow_to_some_element_flags( + maybe_storage_flags: Option>, + ) -> Option { + maybe_storage_flags.map(|storage_flags| storage_flags.serialize()) + } + + /// Map to optional element flags + pub fn map_borrowed_cow_to_some_element_flags( + maybe_storage_flags: &Option>, + ) -> Option { + maybe_storage_flags + .as_ref() + .map(|storage_flags| storage_flags.serialize()) + } + + /// Creates optional element flags + pub fn to_some_element_flags(&self) -> Option { + Some(self.serialize()) + } + + /// Creates element flags. + pub fn to_element_flags(&self) -> ElementFlags { + self.serialize() + } +} diff --git a/packages/rs-drive/src/util/storage_flags/combine.rs b/packages/rs-drive/src/util/storage_flags/combine.rs new file mode 100644 index 00000000000..7a81e2b9a62 --- /dev/null +++ b/packages/rs-drive/src/util/storage_flags/combine.rs @@ -0,0 +1,184 @@ +//! Combining storage flags when an element is replaced. +//! +//! The epoch arithmetic is the crate's. The typed owner of each side is +//! replaced by its carrier key for the duration of the crate call and mapped +//! back through the inputs afterwards, so the crate's owner comparison +//! (equality of 32-byte ids) keeps two different owners apart and the kind +//! of the winning owner is read from the input that supplied it, never from +//! the key. + +use super::{CrateStorageFlags, MergingOwnersStrategy, StorageFlags}; +use dpp::fee::refund_owner::RefundOwner; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes; +use grovedb_epoch_based_storage_flags::error::StorageFlagsError; + +impl StorageFlags { + /// Optional combine added bytes + pub fn optional_combine_added_bytes( + ours: Option, + theirs: Self, + added_bytes: u32, + merging_owners_strategy: MergingOwnersStrategy, + ) -> Result { + match ours { + None => Ok(theirs), + Some(ours) => ours.combine_added_bytes(theirs, added_bytes, merging_owners_strategy), + } + } + + /// Optional combine removed bytes + pub fn optional_combine_removed_bytes( + ours: Option, + theirs: Self, + removed_bytes: &StorageRemovedBytes, + merging_owners_strategy: MergingOwnersStrategy, + ) -> Result { + match ours { + None => Ok(theirs), + Some(ours) => { + ours.combine_removed_bytes(theirs, removed_bytes, merging_owners_strategy) + } + } + } + + /// Combine added bytes + pub fn combine_added_bytes( + self, + rhs: Self, + added_bytes: u32, + merging_owners_strategy: MergingOwnersStrategy, + ) -> Result { + let ours_owner = self.refund_owner(); + let theirs_owner = rhs.refund_owner(); + Self::reject_colliding_owners(ours_owner, theirs_owner)?; + let combined = self + .into_crate_flags_keyed_by_removal_key() + .combine_added_bytes( + rhs.into_crate_flags_keyed_by_removal_key(), + added_bytes, + merging_owners_strategy, + )?; + Self::from_crate_combined(combined, ours_owner, theirs_owner) + } + + /// Combine removed bytes + /// + /// When a single epoch element shrinks in a later epoch there is no epoch + /// map to subtract from and the crate returns the old flags untouched, + /// before it looks at the merging strategy. The typed path resolves the + /// owner itself in that case so that `UseTheirs` transfers ownership on a + /// shrinking replace exactly as it does on a growing one. + pub fn combine_removed_bytes( + self, + rhs: Self, + removed_bytes: &StorageRemovedBytes, + merging_owners_strategy: MergingOwnersStrategy, + ) -> Result { + let ours_owner = self.refund_owner(); + let theirs_owner = rhs.refund_owner(); + Self::reject_colliding_owners(ours_owner, theirs_owner)?; + if self.epoch_index_map().is_none() && self.base_epoch() < rhs.base_epoch() { + let owner = Self::resolve_owner(ours_owner, theirs_owner, merging_owners_strategy)?; + return Ok(Self::new_single_epoch_for_owner(*self.base_epoch(), owner)); + } + let combined = self + .into_crate_flags_keyed_by_removal_key() + .combine_removed_bytes( + rhs.into_crate_flags_keyed_by_removal_key(), + removed_bytes, + merging_owners_strategy, + )?; + Self::from_crate_combined(combined, ours_owner, theirs_owner) + } + + /// Combine for a replace that moved no bytes: the epochs stay ours and + /// the owner follows the merging strategy. + /// + /// This is the branch GroveDB reaches when a replace nets to the same + /// size. It must resolve ownership exactly as the added and removed + /// branches do, because GroveDB may price one replace as a shrink first + /// and as same size after the flags changed width; two different answers + /// would never converge. + pub fn combine_same_size( + self, + rhs: Self, + merging_owners_strategy: MergingOwnersStrategy, + ) -> Result { + let ours_owner = self.refund_owner(); + let theirs_owner = rhs.refund_owner(); + Self::reject_colliding_owners(ours_owner, theirs_owner)?; + let owner = Self::resolve_owner(ours_owner, theirs_owner, merging_owners_strategy)?; + let (base_epoch, epochs, _) = self.into_parts(); + Ok(Self::from_parts(base_epoch, epochs, owner)) + } + + /// The crate's owner rule over typed owners: a side without an owner + /// yields to the other, equal owners keep, different owners follow the + /// strategy. + fn resolve_owner( + ours: Option, + theirs: Option, + merging_owners_strategy: MergingOwnersStrategy, + ) -> Result, StorageFlagsError> { + match (ours, theirs) { + (None, theirs) => Ok(theirs), + (ours, None) => Ok(ours), + (Some(ours), Some(theirs)) if ours == theirs => Ok(Some(ours)), + (Some(ours), Some(theirs)) => match merging_owners_strategy { + MergingOwnersStrategy::RaiseIssue => { + Err(StorageFlagsError::MergingStorageFlagsFromDifferentOwners( + "can not merge from different owners".to_string(), + )) + } + MergingOwnersStrategy::UseOurs => Ok(Some(ours)), + MergingOwnersStrategy::UseTheirs => Ok(Some(theirs)), + }, + } + } + + /// Two different typed owners with one carrier key can never be told + /// apart by the crate, so the merge is refused before it starts. + fn reject_colliding_owners( + ours: Option, + theirs: Option, + ) -> Result<(), StorageFlagsError> { + if let (Some(ours), Some(theirs)) = (ours, theirs) { + if ours != theirs && ours.removal_key() == theirs.removal_key() { + return Err(StorageFlagsError::MergingStorageFlagsFromDifferentOwners( + "two different refund owners share one storage removal carrier key".to_string(), + )); + } + } + Ok(()) + } + + /// Maps the crate's combined flags back to the typed owner that supplied + /// the winning carrier key. + fn from_crate_combined( + combined: CrateStorageFlags, + ours: Option, + theirs: Option, + ) -> Result { + let Some(key) = combined.owner_id().copied() else { + return Ok(Self::from(combined)); + }; + let owner = [ours, theirs] + .into_iter() + .flatten() + .find(|owner| owner.removal_key() == key) + .ok_or_else(|| { + StorageFlagsError::MergingStorageFlagsFromDifferentOwners( + "combined storage flags name an owner that neither side supplied".to_string(), + ) + })?; + let (base_epoch, epochs) = match combined { + CrateStorageFlags::SingleEpoch(base_epoch) + | CrateStorageFlags::SingleEpochOwned(base_epoch, _) => (base_epoch, None), + CrateStorageFlags::MultiEpoch(base_epoch, epochs) + | CrateStorageFlags::MultiEpochOwned(base_epoch, epochs, _) => { + (base_epoch, Some(epochs)) + } + }; + Ok(Self::from_parts(base_epoch, epochs, Some(owner))) + } +} diff --git a/packages/rs-drive/src/util/storage_flags/mod.rs b/packages/rs-drive/src/util/storage_flags/mod.rs new file mode 100644 index 00000000000..b0a11f0e951 --- /dev/null +++ b/packages/rs-drive/src/util/storage_flags/mod.rs @@ -0,0 +1,451 @@ +//! Storage flags +//! +//! Element flags are the opaque bytes GroveDB stores next to every element. +//! Drive uses them to remember which epoch paid for the bytes and who paid, +//! so that a later removal can refund the right owner at the right rate. +//! +//! GroveDB never interprets these bytes; only the closures Drive passes to +//! the batch apply and the readers in this crate do. That is why the codec +//! lives here. The four historical variants keep the exact encoding of the +//! `grovedb_epoch_based_storage_flags` crate: their bytes are produced and +//! parsed by that crate, so they cannot drift. Two variants are added for +//! bytes paid for by a contract credit bucket. The type byte declares the +//! owner's kind; it is never inferred from the shape of an identifier. +//! +//! The parse helpers (`deserialize`, `from_element_flags_ref`, the `map_*` +//! helpers) decode all six variants so that readers carry bucket-owned flags +//! through unchanged. The batch apply generations that predate typed owners +//! do not use this type at all: their closures are bound to the crate's +//! flags type, which rejects the two bucket variants, so they fail closed on +//! flags they were never written to price. The typed closure entry points +//! here (`update_element_flags_typed`, `split_removal_bytes_typed`) accept +//! all six variants and record the owner of every sectioned removal. + +mod codec; +mod combine; +mod split; +mod update; + +use dpp::fee::refund_owner::{ContractCreditBucketPosition, RefundOwner}; +use dpp::identifier::Identifier; +use grovedb_epoch_based_storage_flags::StorageFlags as CrateStorageFlags; +pub use grovedb_epoch_based_storage_flags::{ + MergingOwnersStrategy, MINIMUM_NON_BASE_FLAGS_SIZE, SINGLE_EPOCH_FLAGS_SIZE, +}; +use std::borrow::Cow; +use std::collections::BTreeMap; +use std::fmt; + +use self::StorageFlags::{ + MultiEpoch, MultiEpochContractBucket, MultiEpochOwned, SingleEpoch, SingleEpochContractBucket, + SingleEpochOwned, +}; + +/// An epoch index as stored in element flags +pub type EpochIndex = u16; + +/// The epoch in which an element was first stored +pub type BaseEpoch = EpochIndex; + +/// Bytes added to an element in a later epoch +pub type BytesAddedInEpoch = u32; + +/// The identity that owns stored bytes, as 32 raw bytes +pub type OwnerId = [u8; 32]; + +/// The contract whose credit bucket owns stored bytes, as 32 raw bytes +pub type ContractId = [u8; 32]; + +/// Size of the owner id carried by the identity-owned variants +const OWNER_ID_SIZE: u32 = 32; + +/// Size of the contract bucket owner: contract id plus two-byte position +const CONTRACT_BUCKET_OWNER_SIZE: u32 = 34; + +/// Storage flags: the epoch that paid for an element's bytes and, when +/// someone other than the system paid, the recorded owner. +/// +/// Variants 0 to 3 are encoded and decoded by the pinned +/// `grovedb_epoch_based_storage_flags` crate and are byte for byte what is on +/// chain today. Variants 4 and 5 carry a contract credit bucket owner. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum StorageFlags { + /// Single epoch, unowned + /// represented as byte 0 + SingleEpoch(BaseEpoch), + + /// Multi epoch, unowned + /// represented as byte 1 + MultiEpoch(BaseEpoch, BTreeMap), + + /// Single epoch owned by an identity + /// represented as byte 2 + SingleEpochOwned(BaseEpoch, OwnerId), + + /// Multi epoch owned by an identity + /// represented as byte 3 + MultiEpochOwned(BaseEpoch, BTreeMap, OwnerId), + + /// Single epoch owned by a contract credit bucket + /// represented as byte 4 + /// + /// Layout: type byte, contract id (32 bytes), bucket position (2 bytes, + /// big-endian), base epoch (2 bytes, big-endian); 37 bytes in total. + /// + /// Provisional: the type byte allocation is proposed in the fees + /// workstream register (issue 4689) and pending the owner's confirmation. + SingleEpochContractBucket(BaseEpoch, ContractId, ContractCreditBucketPosition), + + /// Multi epoch owned by a contract credit bucket + /// represented as byte 5 + /// + /// Layout: the 37-byte header of the single epoch bucket variant followed + /// by the crate's epoch map encoding (2-byte epoch index, varint bytes). + MultiEpochContractBucket( + BaseEpoch, + BTreeMap, + ContractId, + ContractCreditBucketPosition, + ), +} + +impl fmt::Display for StorageFlags { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + SingleEpoch(base_epoch) => { + write!(f, "SingleEpoch(BaseEpoch: {})", base_epoch) + } + MultiEpoch(base_epoch, epochs) => { + write!(f, "MultiEpoch(BaseEpoch: {}, Epochs: ", base_epoch)?; + for (index, bytes) in epochs { + write!(f, "[EpochIndex: {}, BytesAdded: {}] ", index, bytes)?; + } + write!(f, ")") + } + SingleEpochOwned(base_epoch, owner_id) => { + write!( + f, + "SingleEpochOwned(BaseEpoch: {}, OwnerId: {})", + base_epoch, + hex::encode(owner_id) + ) + } + MultiEpochOwned(base_epoch, epochs, owner_id) => { + write!(f, "MultiEpochOwned(BaseEpoch: {}, Epochs: ", base_epoch)?; + for (index, bytes) in epochs { + write!(f, "[EpochIndex: {}, BytesAdded: {}] ", index, bytes)?; + } + write!(f, ", OwnerId: {})", hex::encode(owner_id)) + } + SingleEpochContractBucket(base_epoch, contract_id, position) => { + write!( + f, + "SingleEpochContractBucket(BaseEpoch: {}, ContractId: {}, Position: {})", + base_epoch, + hex::encode(contract_id), + position + ) + } + MultiEpochContractBucket(base_epoch, epochs, contract_id, position) => { + write!( + f, + "MultiEpochContractBucket(BaseEpoch: {}, Epochs: ", + base_epoch + )?; + for (index, bytes) in epochs { + write!(f, "[EpochIndex: {}, BytesAdded: {}] ", index, bytes)?; + } + write!( + f, + ", ContractId: {}, Position: {})", + hex::encode(contract_id), + position + ) + } + } + } +} + +impl From for StorageFlags { + fn from(value: CrateStorageFlags) -> Self { + match value { + CrateStorageFlags::SingleEpoch(base_epoch) => SingleEpoch(base_epoch), + CrateStorageFlags::MultiEpoch(base_epoch, epochs) => MultiEpoch(base_epoch, epochs), + CrateStorageFlags::SingleEpochOwned(base_epoch, owner_id) => { + SingleEpochOwned(base_epoch, owner_id) + } + CrateStorageFlags::MultiEpochOwned(base_epoch, epochs, owner_id) => { + MultiEpochOwned(base_epoch, epochs, owner_id) + } + } + } +} + +impl StorageFlags { + /// Create new single epoch storage flags, owned by an identity when an + /// owner id is given + pub fn new_single_epoch(epoch: BaseEpoch, maybe_owner_id: Option) -> Self { + match maybe_owner_id { + None => SingleEpoch(epoch), + Some(owner_id) => SingleEpochOwned(epoch, owner_id), + } + } + + /// Create new single epoch storage flags for a typed owner, or unowned + /// flags when no owner is given + pub fn new_single_epoch_for_owner(epoch: BaseEpoch, owner: Option) -> Self { + match owner { + None => SingleEpoch(epoch), + Some(RefundOwner::Identity(identity_id)) => { + SingleEpochOwned(epoch, identity_id.to_buffer()) + } + Some(RefundOwner::ContractBucket { + contract_id, + position, + }) => SingleEpochContractBucket(epoch, contract_id.to_buffer(), position), + } + } + + /// Sets the owner id if we have identity-owned storage flags + pub fn set_owner_id(&mut self, owner_id: OwnerId) { + match self { + SingleEpochOwned(_, previous_owner_id) | MultiEpochOwned(_, _, previous_owner_id) => { + *previous_owner_id = owner_id; + } + _ => {} + } + } + + /// Returns base epoch + pub fn base_epoch(&self) -> &BaseEpoch { + match self { + SingleEpoch(base_epoch) + | MultiEpoch(base_epoch, _) + | SingleEpochOwned(base_epoch, _) + | MultiEpochOwned(base_epoch, ..) + | SingleEpochContractBucket(base_epoch, ..) + | MultiEpochContractBucket(base_epoch, ..) => base_epoch, + } + } + + /// Returns the owner id when an identity owns the bytes. + /// + /// Contract bucket owners are not identities and return `None` here; use + /// [`Self::refund_owner`] for the typed owner of any variant. + pub fn owner_id(&self) -> Option<&OwnerId> { + match self { + SingleEpochOwned(_, owner_id) | MultiEpochOwned(_, _, owner_id) => Some(owner_id), + _ => None, + } + } + + /// Returns the typed owner of the bytes, if anyone other than the system + /// paid for them + pub fn refund_owner(&self) -> Option { + match self { + SingleEpoch(_) | MultiEpoch(..) => None, + SingleEpochOwned(_, owner_id) | MultiEpochOwned(_, _, owner_id) => { + Some(RefundOwner::Identity(Identifier::from(*owner_id))) + } + SingleEpochContractBucket(_, contract_id, position) + | MultiEpochContractBucket(_, _, contract_id, position) => { + Some(RefundOwner::ContractBucket { + contract_id: Identifier::from(*contract_id), + position: *position, + }) + } + } + } + + /// Returns epoch index map + pub fn epoch_index_map(&self) -> Option<&BTreeMap> { + match self { + MultiEpoch(_, epoch_int_map) + | MultiEpochOwned(_, epoch_int_map, _) + | MultiEpochContractBucket(_, epoch_int_map, ..) => Some(epoch_int_map), + _ => None, + } + } + + /// Returns optional default storage flags + pub fn optional_default() -> Option { + None + } + + /// Returns default optional storage flag as ref + pub fn optional_default_as_ref() -> Option<&'static Self> { + None + } + + /// Returns default optional storage flag as ref + pub fn optional_default_as_cow() -> Option> { + None + } + + /// Returns type byte + pub fn type_byte(&self) -> u8 { + match self { + SingleEpoch(_) => 0, + MultiEpoch(..) => 1, + SingleEpochOwned(..) => 2, + MultiEpochOwned(..) => 3, + SingleEpochContractBucket(..) => 4, + MultiEpochContractBucket(..) => 5, + } + } + + /// Approximate serialized size of flags with or without an identity owner + pub fn approximate_size( + has_owner_id: bool, + approximate_changes_and_bytes_count: Option<(u16, u8)>, + ) -> u32 { + CrateStorageFlags::approximate_size(has_owner_id, approximate_changes_and_bytes_count) + } + + /// Approximate serialized size of flags for a typed owner: 3 bytes for + /// the type byte and base epoch, plus 32 for an identity owner or 34 for + /// a contract bucket owner, plus the approximate epoch map size + pub fn approximate_size_for_owner( + owner: Option<&RefundOwner>, + approximate_changes_and_bytes_count: Option<(u16, u8)>, + ) -> u32 { + let mut size = SINGLE_EPOCH_FLAGS_SIZE; + match owner { + None => {} + Some(RefundOwner::Identity(_)) => size += OWNER_ID_SIZE, + Some(RefundOwner::ContractBucket { .. }) => size += CONTRACT_BUCKET_OWNER_SIZE, + } + if let Some((approximate_change_count, bytes_changed_required_size)) = + approximate_changes_and_bytes_count + { + size += (approximate_change_count as u32) * (2 + bytes_changed_required_size as u32) + } + size + } + + /// Wrap Storage Flags into optional owned cow + pub fn into_optional_cow<'a>(self) -> Option> { + Some(Cow::Owned(self)) + } + + /// Builds the flags for a base epoch, an optional epoch map and an + /// optional typed owner. An empty epoch map gives a single epoch variant. + pub(super) fn from_parts( + base_epoch: BaseEpoch, + epochs: Option>, + owner: Option, + ) -> Self { + let epochs = epochs.filter(|epochs| !epochs.is_empty()); + match (owner, epochs) { + (None, None) => SingleEpoch(base_epoch), + (None, Some(epochs)) => MultiEpoch(base_epoch, epochs), + (Some(RefundOwner::Identity(identity_id)), None) => { + SingleEpochOwned(base_epoch, identity_id.to_buffer()) + } + (Some(RefundOwner::Identity(identity_id)), Some(epochs)) => { + MultiEpochOwned(base_epoch, epochs, identity_id.to_buffer()) + } + ( + Some(RefundOwner::ContractBucket { + contract_id, + position, + }), + None, + ) => SingleEpochContractBucket(base_epoch, contract_id.to_buffer(), position), + ( + Some(RefundOwner::ContractBucket { + contract_id, + position, + }), + Some(epochs), + ) => MultiEpochContractBucket(base_epoch, epochs, contract_id.to_buffer(), position), + } + } + + /// Splits the flags into their base epoch, epoch map and typed owner, + /// moving the map out. + pub(super) fn into_parts( + self, + ) -> ( + BaseEpoch, + Option>, + Option, + ) { + let owner = self.refund_owner(); + match self { + SingleEpoch(base_epoch) + | SingleEpochOwned(base_epoch, _) + | SingleEpochContractBucket(base_epoch, ..) => (base_epoch, None, owner), + MultiEpoch(base_epoch, epochs) + | MultiEpochOwned(base_epoch, epochs, _) + | MultiEpochContractBucket(base_epoch, epochs, ..) => (base_epoch, Some(epochs), owner), + } + } + + /// The crate's value with every typed owner replaced by its removal key, + /// moving the epoch map instead of cloning it. See + /// [`Self::to_crate_flags_keyed_by_removal_key`] for why this is sound. + pub(super) fn into_crate_flags_keyed_by_removal_key(self) -> CrateStorageFlags { + let (base_epoch, epochs, owner) = self.into_parts(); + let key = owner.map(|owner| owner.removal_key()); + match (key, epochs) { + (None, None) => CrateStorageFlags::SingleEpoch(base_epoch), + (None, Some(epochs)) => CrateStorageFlags::MultiEpoch(base_epoch, epochs), + (Some(key), None) => CrateStorageFlags::SingleEpochOwned(base_epoch, key), + (Some(key), Some(epochs)) => { + CrateStorageFlags::MultiEpochOwned(base_epoch, epochs, key) + } + } + } + + /// The crate's value with every typed owner replaced by its removal key. + /// + /// This is what lets the crate's epoch arithmetic (split and combine) run + /// unchanged over the bucket variants: the crate only ever compares owner + /// ids for equality and sections removed bytes under them, and the + /// removal key is exactly the key those bytes must be sectioned under. + /// Callers map the result back through [`Self::refund_owner`] of the + /// inputs so the kind is never read from the key. + pub(super) fn to_crate_flags_keyed_by_removal_key(&self) -> CrateStorageFlags { + match self { + SingleEpoch(base_epoch) => CrateStorageFlags::SingleEpoch(*base_epoch), + MultiEpoch(base_epoch, epochs) => { + CrateStorageFlags::MultiEpoch(*base_epoch, epochs.clone()) + } + SingleEpochOwned(base_epoch, owner_id) => { + CrateStorageFlags::SingleEpochOwned(*base_epoch, *owner_id) + } + MultiEpochOwned(base_epoch, epochs, owner_id) => { + CrateStorageFlags::MultiEpochOwned(*base_epoch, epochs.clone(), *owner_id) + } + SingleEpochContractBucket(base_epoch, contract_id, position) => { + CrateStorageFlags::SingleEpochOwned( + *base_epoch, + bucket_removal_key(contract_id, *position), + ) + } + MultiEpochContractBucket(base_epoch, epochs, contract_id, position) => { + CrateStorageFlags::MultiEpochOwned( + *base_epoch, + epochs.clone(), + bucket_removal_key(contract_id, *position), + ) + } + } + } +} + +/// The carrier key of a contract bucket owner +fn bucket_removal_key( + contract_id: &ContractId, + position: ContractCreditBucketPosition, +) -> [u8; 32] { + RefundOwner::ContractBucket { + contract_id: Identifier::from(*contract_id), + position, + } + .removal_key() +} + +#[cfg(test)] +mod tests; diff --git a/packages/rs-drive/src/util/storage_flags/split.rs b/packages/rs-drive/src/util/storage_flags/split.rs new file mode 100644 index 00000000000..9c5b627fd83 --- /dev/null +++ b/packages/rs-drive/src/util/storage_flags/split.rs @@ -0,0 +1,90 @@ +//! Splitting removed bytes across epochs and owners when an element shrinks +//! or is deleted. + +use super::StorageFlags; +use crate::error::drive::DriveError; +use crate::error::Error; +use dpp::fee::refund_owner::{RefundOwner, RefundOwnersByIdentifier, SYSTEM_REFUND_CARRIER_KEY}; +use grovedb::ElementFlags; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes::BasicStorageRemoval; + +impl StorageFlags { + /// Sections removed bytes per epoch, taking from the latest epochs + /// first, under the owner's carrier key (or the system key when the + /// bytes are unowned). + /// + /// The carrier key alone does not say what kind of owner it names; use + /// [`Self::split_removal_bytes_typed`] on a path that must route the + /// refund, so that the owner is recorded alongside. + pub fn split_storage_removed_bytes( + &self, + removed_key_bytes: u32, + removed_value_bytes: u32, + ) -> (StorageRemovedBytes, StorageRemovedBytes) { + self.to_crate_flags_keyed_by_removal_key() + .split_storage_removed_bytes(removed_key_bytes, removed_value_bytes) + } + + /// The batch split closure for typed owners. + /// + /// Decodes all six flag types, sections the removed bytes under the + /// owner's carrier key and records `(carrier key, owner)` in + /// `refund_owners` for every owned removal, identities included. Within + /// one batch a carrier key may name only one owner; a second owner for + /// the same key, or a bucket whose key equals the system key, fails the + /// batch. Nothing is guessed and nothing is re-derived. + pub fn split_removal_bytes_typed( + flags: &mut ElementFlags, + removed_key_bytes: u32, + removed_value_bytes: u32, + refund_owners: &mut RefundOwnersByIdentifier, + ) -> Result<(StorageRemovedBytes, StorageRemovedBytes), Error> { + let maybe_storage_flags = Self::from_element_flags_ref(flags)?; + match maybe_storage_flags { + None => Ok(( + BasicStorageRemoval(removed_key_bytes), + BasicStorageRemoval(removed_value_bytes), + )), + Some(storage_flags) => { + if let Some(owner) = storage_flags.refund_owner() { + Self::record_refund_owner(owner, refund_owners)?; + } + Ok(storage_flags + .into_crate_flags_keyed_by_removal_key() + .split_storage_removed_bytes(removed_key_bytes, removed_value_bytes)) + } + } + } + + fn record_refund_owner( + owner: RefundOwner, + refund_owners: &mut RefundOwnersByIdentifier, + ) -> Result<(), Error> { + let key = owner.removal_key(); + if key == SYSTEM_REFUND_CARRIER_KEY { + return match owner { + // The all-zero identity has always been sectioned as system + // bytes and never refunded; recording nothing keeps that. + RefundOwner::Identity(_) => Ok(()), + RefundOwner::ContractBucket { .. } => { + Err(Error::Drive(DriveError::CorruptedCodeExecution( + "a contract bucket refund owner derived the system carrier key", + ))) + } + }; + } + match refund_owners.get(&key) { + Some(existing) if *existing != owner => { + Err(Error::Drive(DriveError::CorruptedCodeExecution( + "two different refund owners share one storage removal carrier key within a batch", + ))) + } + Some(_) => Ok(()), + None => { + refund_owners.insert(key, owner); + Ok(()) + } + } + } +} diff --git a/packages/rs-drive/src/util/storage_flags/tests.rs b/packages/rs-drive/src/util/storage_flags/tests.rs new file mode 100644 index 00000000000..f4adc2020b2 --- /dev/null +++ b/packages/rs-drive/src/util/storage_flags/tests.rs @@ -0,0 +1,918 @@ +//! Tests for the Drive storage flags: codec, split, combine and the batch +//! closure entry points. + +use super::*; +use crate::error::drive::DriveError; +use crate::error::Error; +use dpp::fee::refund_owner::{RefundOwnersByIdentifier, SYSTEM_REFUND_CARRIER_KEY}; +use grovedb_costs::storage_cost::removal::StorageRemovalPerEpochByIdentifier; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes; +use grovedb_costs::storage_cost::removal::StorageRemovedBytes::{ + BasicStorageRemoval, NoStorageRemoval, SectionedStorageRemoval, +}; +use grovedb_costs::storage_cost::StorageCost; +use grovedb_epoch_based_storage_flags::error::StorageFlagsError; +use intmap::IntMap; + +const CONTRACT_ID: ContractId = [0x11; 32]; +const OWNER_ID: OwnerId = [0x22; 32]; + +fn bucket_owner(position: ContractCreditBucketPosition) -> RefundOwner { + RefundOwner::ContractBucket { + contract_id: Identifier::from(CONTRACT_ID), + position, + } +} + +fn epochs(entries: &[(EpochIndex, BytesAddedInEpoch)]) -> BTreeMap { + entries.iter().copied().collect() +} + +fn every_variant() -> Vec { + vec![ + SingleEpoch(3), + MultiEpoch(3, epochs(&[(4, 10), (5, 300)])), + SingleEpochOwned(3, OWNER_ID), + MultiEpochOwned(3, epochs(&[(4, 10), (5, 300)]), OWNER_ID), + SingleEpochContractBucket(3, CONTRACT_ID, 7), + MultiEpochContractBucket(3, epochs(&[(4, 10), (5, 300)]), CONTRACT_ID, 7), + MultiEpochContractBucket(0, epochs(&[(65535, u32::MAX)]), CONTRACT_ID, u16::MAX), + ] +} + +fn sectioned(key: [u8; 32], entries: &[(u16, u32)]) -> StorageRemovedBytes { + let mut map = StorageRemovalPerEpochByIdentifier::new(); + map.insert(key, IntMap::from_iter(entries.iter().copied())); + SectionedStorageRemoval(map) +} + +mod codec { + use super::*; + + #[test] + fn should_round_trip_every_variant() { + for flags in every_variant() { + let bytes = flags.serialize(); + let decoded = StorageFlags::deserialize(&bytes) + .expect("should decode") + .expect("should be some"); + assert_eq!(decoded, flags, "{}", flags); + assert_eq!(bytes.first().copied(), Some(flags.type_byte())); + } + } + + #[test] + fn should_encode_the_historical_variants_exactly_as_the_crate() { + let pairs = [ + (SingleEpoch(3), CrateStorageFlags::SingleEpoch(3)), + ( + MultiEpoch(3, epochs(&[(4, 10)])), + CrateStorageFlags::MultiEpoch(3, epochs(&[(4, 10)])), + ), + ( + SingleEpochOwned(3, OWNER_ID), + CrateStorageFlags::SingleEpochOwned(3, OWNER_ID), + ), + ( + MultiEpochOwned(3, epochs(&[(4, 10), (5, 300)]), OWNER_ID), + CrateStorageFlags::MultiEpochOwned(3, epochs(&[(4, 10), (5, 300)]), OWNER_ID), + ), + ]; + for (ours, theirs) in pairs { + assert_eq!(ours.serialize(), theirs.serialize(), "{}", ours); + assert_eq!(ours.serialized_size(), theirs.serialized_size()); + assert_eq!( + StorageFlags::deserialize(&theirs.serialize()).expect("should decode"), + Some(ours) + ); + } + } + + #[test] + fn should_pin_the_byte_layout_of_every_variant() { + assert_eq!(SingleEpoch(0x0102).serialize(), vec![0, 1, 2]); + assert_eq!( + MultiEpoch(0x0102, epochs(&[(0x0304, 300)])).serialize(), + vec![1, 1, 2, 3, 4, 0xac, 0x02] + ); + + let mut owned = vec![2]; + owned.extend_from_slice(&OWNER_ID); + owned.extend_from_slice(&[1, 2]); + assert_eq!(SingleEpochOwned(0x0102, OWNER_ID).serialize(), owned); + assert_eq!(owned.len(), 35); + + let mut multi_owned = vec![3]; + multi_owned.extend_from_slice(&OWNER_ID); + multi_owned.extend_from_slice(&[1, 2, 3, 4, 0xac, 0x02]); + assert_eq!( + MultiEpochOwned(0x0102, epochs(&[(0x0304, 300)]), OWNER_ID).serialize(), + multi_owned + ); + + let mut bucket = vec![4]; + bucket.extend_from_slice(&CONTRACT_ID); + bucket.extend_from_slice(&[0, 7]); + bucket.extend_from_slice(&[1, 2]); + assert_eq!( + SingleEpochContractBucket(0x0102, CONTRACT_ID, 7).serialize(), + bucket + ); + assert_eq!(bucket.len(), 37); + + let mut multi_bucket = vec![5]; + multi_bucket.extend_from_slice(&CONTRACT_ID); + multi_bucket.extend_from_slice(&[0, 7]); + multi_bucket.extend_from_slice(&[1, 2, 3, 4, 0xac, 0x02]); + assert_eq!( + MultiEpochContractBucket(0x0102, epochs(&[(0x0304, 300)]), CONTRACT_ID, 7).serialize(), + multi_bucket + ); + } + + #[test] + fn should_report_the_serialized_size_of_every_variant() { + for flags in every_variant() { + assert_eq!( + flags.serialized_size() as usize, + flags.serialize().len(), + "{}", + flags + ); + } + } + + #[test] + fn should_reject_an_unknown_type_byte() { + let mut bytes = vec![6]; + bytes.extend_from_slice(&[0; 36]); + let error = StorageFlags::deserialize(&bytes).expect_err("should reject"); + assert!(matches!( + error, + StorageFlagsError::DeserializeUnknownStorageFlagsType(_) + )); + assert!(matches!( + StorageFlags::deserialize(&[255, 1, 2]), + Err(StorageFlagsError::DeserializeUnknownStorageFlagsType(_)) + )); + } + + #[test] + fn should_reject_truncated_and_oversized_bucket_flags() { + let single = SingleEpochContractBucket(3, CONTRACT_ID, 7).serialize(); + for cut in [1usize, 32, 33, 34, 35, 36] { + assert!( + matches!( + StorageFlags::deserialize(&single[..cut]), + Err(StorageFlagsError::StorageFlagsWrongSize(_)) + ), + "single bucket cut at {}", + cut + ); + } + let mut too_long = single.clone(); + too_long.push(0); + assert!(matches!( + StorageFlags::deserialize(&too_long), + Err(StorageFlagsError::StorageFlagsWrongSize(_)) + )); + + let multi = MultiEpochContractBucket(3, epochs(&[(4, 300)]), CONTRACT_ID, 7).serialize(); + // header only, header plus one epoch byte, and a cut inside the varint + for cut in [37usize, 38, 39, multi.len() - 1] { + assert!( + matches!( + StorageFlags::deserialize(&multi[..cut]), + Err(StorageFlagsError::StorageFlagsWrongSize(_)) + ), + "multi bucket cut at {}", + cut + ); + } + // a dangling epoch index with no byte count + let mut dangling = multi.clone(); + dangling.extend_from_slice(&[0, 9]); + assert!(matches!( + StorageFlags::deserialize(&dangling), + Err(StorageFlagsError::StorageFlagsWrongSize(_)) + )); + } + + #[test] + fn should_decode_empty_flags_as_none() { + assert_eq!(StorageFlags::deserialize(&[]).expect("should decode"), None); + assert_eq!( + StorageFlags::map_some_element_flags_ref(&None).expect("should decode"), + None + ); + } + + #[test] + fn should_size_typed_owners_for_estimation() { + let identity = RefundOwner::Identity(Identifier::from(OWNER_ID)); + let bucket = bucket_owner(7); + + assert_eq!( + StorageFlags::approximate_size_for_owner(None, None), + SingleEpoch(0).serialized_size() + ); + assert_eq!( + StorageFlags::approximate_size_for_owner(Some(&identity), None), + SingleEpochOwned(0, OWNER_ID).serialized_size() + ); + assert_eq!( + StorageFlags::approximate_size_for_owner(Some(&bucket), None), + SingleEpochContractBucket(0, CONTRACT_ID, 7).serialized_size() + ); + assert_eq!( + StorageFlags::approximate_size_for_owner(Some(&identity), Some((2, 1))), + StorageFlags::approximate_size(true, Some((2, 1))) + ); + assert_eq!( + StorageFlags::approximate_size_for_owner(Some(&bucket), Some((2, 1))), + 37 + 2 * 3 + ); + } + + #[test] + fn should_expose_the_typed_owner_of_every_variant() { + let identity = RefundOwner::Identity(Identifier::from(OWNER_ID)); + let bucket = bucket_owner(7); + + assert_eq!(SingleEpoch(1).refund_owner(), None); + assert_eq!(MultiEpoch(1, epochs(&[(2, 3)])).refund_owner(), None); + assert_eq!(SingleEpochOwned(1, OWNER_ID).refund_owner(), Some(identity)); + assert_eq!( + MultiEpochOwned(1, epochs(&[(2, 3)]), OWNER_ID).refund_owner(), + Some(identity) + ); + assert_eq!( + SingleEpochContractBucket(1, CONTRACT_ID, 7).refund_owner(), + Some(bucket) + ); + assert_eq!( + MultiEpochContractBucket(1, epochs(&[(2, 3)]), CONTRACT_ID, 7).refund_owner(), + Some(bucket) + ); + + // the identity accessor never reads a bucket as an identity + assert_eq!( + SingleEpochContractBucket(1, CONTRACT_ID, 7).owner_id(), + None + ); + assert_eq!(SingleEpochOwned(1, OWNER_ID).owner_id(), Some(&OWNER_ID)); + + assert_eq!( + StorageFlags::new_single_epoch_for_owner(9, Some(bucket)), + SingleEpochContractBucket(9, CONTRACT_ID, 7) + ); + assert_eq!( + StorageFlags::new_single_epoch_for_owner(9, Some(identity)), + SingleEpochOwned(9, OWNER_ID) + ); + assert_eq!( + StorageFlags::new_single_epoch_for_owner(9, None), + SingleEpoch(9) + ); + } +} + +mod split { + use super::*; + + #[test] + fn should_section_a_deleted_bucket_element_under_its_carrier_key_and_record_the_owner() { + let owner = bucket_owner(7); + let mut flags = SingleEpochContractBucket(5, CONTRACT_ID, 7).to_element_flags(); + let mut refund_owners = RefundOwnersByIdentifier::new(); + + let (key_removal, value_removal) = + StorageFlags::split_removal_bytes_typed(&mut flags, 50, 150, &mut refund_owners) + .expect("should split"); + + assert_eq!(key_removal, sectioned(owner.removal_key(), &[(5, 50)])); + assert_eq!(value_removal, sectioned(owner.removal_key(), &[(5, 150)])); + assert_eq!( + refund_owners, + RefundOwnersByIdentifier::from([(owner.removal_key(), owner)]) + ); + } + + #[test] + fn should_take_from_the_latest_epochs_first_when_a_bucket_element_shrinks() { + let owner = bucket_owner(7); + let flags = MultiEpochContractBucket(5, epochs(&[(6, 300), (7, 400)]), CONTRACT_ID, 7); + let mut refund_owners = RefundOwnersByIdentifier::new(); + + let (key_removal, value_removal) = StorageFlags::split_removal_bytes_typed( + &mut flags.to_element_flags(), + 0, + 700, + &mut refund_owners, + ) + .expect("should split"); + + assert_eq!(key_removal, NoStorageRemoval); + // the same LIFO sectioning the crate applies to identity-owned flags + assert_eq!( + value_removal, + sectioned(owner.removal_key(), &[(5, 6), (6, 297), (7, 397)]) + ); + assert_eq!(refund_owners.get(&owner.removal_key()), Some(&owner)); + + let crate_equivalent = CrateStorageFlags::MultiEpochOwned( + 5, + epochs(&[(6, 300), (7, 400)]), + owner.removal_key(), + ) + .split_storage_removed_bytes(0, 700); + assert_eq!(value_removal, crate_equivalent.1); + } + + #[test] + fn should_record_an_identity_owner() { + let owner = RefundOwner::Identity(Identifier::from(OWNER_ID)); + let mut refund_owners = RefundOwnersByIdentifier::new(); + + let (key_removal, value_removal) = StorageFlags::split_removal_bytes_typed( + &mut SingleEpochOwned(2, OWNER_ID).to_element_flags(), + 10, + 20, + &mut refund_owners, + ) + .expect("should split"); + + assert_eq!(key_removal, sectioned(OWNER_ID, &[(2, 10)])); + assert_eq!(value_removal, sectioned(OWNER_ID, &[(2, 20)])); + assert_eq!( + refund_owners, + RefundOwnersByIdentifier::from([(OWNER_ID, owner)]) + ); + } + + #[test] + fn should_section_unowned_bytes_under_the_system_key_and_record_nothing() { + let mut refund_owners = RefundOwnersByIdentifier::new(); + + let (key_removal, value_removal) = StorageFlags::split_removal_bytes_typed( + &mut SingleEpoch(2).to_element_flags(), + 10, + 20, + &mut refund_owners, + ) + .expect("should split"); + + assert_eq!( + key_removal, + sectioned(SYSTEM_REFUND_CARRIER_KEY, &[(2, 10)]) + ); + assert_eq!( + value_removal, + sectioned(SYSTEM_REFUND_CARRIER_KEY, &[(2, 20)]) + ); + assert!(refund_owners.is_empty()); + + let (key_removal, value_removal) = + StorageFlags::split_removal_bytes_typed(&mut vec![], 10, 20, &mut refund_owners) + .expect("should split"); + assert_eq!(key_removal, BasicStorageRemoval(10)); + assert_eq!(value_removal, BasicStorageRemoval(20)); + assert!(refund_owners.is_empty()); + } + + #[test] + fn should_keep_the_all_zero_identity_as_system_bytes_and_record_nothing() { + let mut refund_owners = RefundOwnersByIdentifier::new(); + + let (key_removal, _) = StorageFlags::split_removal_bytes_typed( + &mut SingleEpochOwned(2, SYSTEM_REFUND_CARRIER_KEY).to_element_flags(), + 10, + 20, + &mut refund_owners, + ) + .expect("should split"); + + assert_eq!( + key_removal, + sectioned(SYSTEM_REFUND_CARRIER_KEY, &[(2, 10)]) + ); + assert!(refund_owners.is_empty()); + } + + #[test] + fn should_reject_one_carrier_key_recorded_for_two_owners_within_a_batch() { + let owner = bucket_owner(7); + let mut refund_owners = RefundOwnersByIdentifier::from([( + owner.removal_key(), + RefundOwner::Identity(Identifier::from(owner.removal_key())), + )]); + + let result = StorageFlags::split_removal_bytes_typed( + &mut SingleEpochContractBucket(5, CONTRACT_ID, 7).to_element_flags(), + 1, + 1, + &mut refund_owners, + ); + + assert!(matches!( + result, + Err(Error::Drive(DriveError::CorruptedCodeExecution(_))) + )); + } + + #[test] + fn should_keep_the_same_owner_recorded_once_across_two_elements() { + let owner = bucket_owner(7); + let mut refund_owners = RefundOwnersByIdentifier::new(); + for _ in 0..2 { + StorageFlags::split_removal_bytes_typed( + &mut SingleEpochContractBucket(5, CONTRACT_ID, 7).to_element_flags(), + 1, + 1, + &mut refund_owners, + ) + .expect("should split"); + } + assert_eq!(refund_owners.len(), 1); + assert_eq!(refund_owners.get(&owner.removal_key()), Some(&owner)); + } + + /// The shipped batch apply generations bind their closures to the crate + /// type, so this is the property they rely on. + #[test] + fn should_be_rejected_by_the_crate_closure_entry_points_the_shipped_generations_use() { + let mut flags = SingleEpochContractBucket(5, CONTRACT_ID, 7).to_element_flags(); + assert!(matches!( + CrateStorageFlags::split_removal_bytes(&mut flags, 1, 1), + Err(StorageFlagsError::DeserializeUnknownStorageFlagsType(_)) + )); + assert!(matches!( + CrateStorageFlags::from_element_flags_ref(&flags), + Err(StorageFlagsError::DeserializeUnknownStorageFlagsType(_)) + )); + + let cost = StorageCost { + added_bytes: 10, + replaced_bytes: 1, + removed_bytes: NoStorageRemoval, + }; + let mut new_flags = SingleEpochContractBucket(6, CONTRACT_ID, 7).to_element_flags(); + assert!(matches!( + CrateStorageFlags::update_element_flags( + &cost, + Some(SingleEpochContractBucket(5, CONTRACT_ID, 7).to_element_flags()), + &mut new_flags + ), + Err(StorageFlagsError::DeserializeUnknownStorageFlagsType(_)) + )); + + // the crate still serves the historical variants Drive writes today + let mut owned = SingleEpochOwned(5, OWNER_ID).to_element_flags(); + let (key_removal, _) = + CrateStorageFlags::split_removal_bytes(&mut owned, 3, 4).expect("should split"); + assert_eq!(key_removal, sectioned(OWNER_ID, &[(5, 3)])); + } +} + +mod combine { + use super::*; + + #[test] + fn should_keep_the_bucket_owner_when_combining_the_same_base_epoch() { + let ours = SingleEpochContractBucket(1, CONTRACT_ID, 7); + let theirs = SingleEpochContractBucket(1, CONTRACT_ID, 7); + + let combined = ours + .combine_added_bytes(theirs, 10, MergingOwnersStrategy::RaiseIssue) + .expect("should combine"); + + assert_eq!(combined, SingleEpochContractBucket(1, CONTRACT_ID, 7)); + } + + #[test] + fn should_transfer_ownership_across_kinds_with_use_theirs() { + let identity_to_bucket = SingleEpochOwned(1, OWNER_ID) + .combine_added_bytes( + SingleEpochContractBucket(1, CONTRACT_ID, 7), + 10, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!( + identity_to_bucket, + SingleEpochContractBucket(1, CONTRACT_ID, 7) + ); + + let bucket_to_identity = SingleEpochContractBucket(1, CONTRACT_ID, 7) + .combine_added_bytes( + SingleEpochOwned(1, OWNER_ID), + 10, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!(bucket_to_identity, SingleEpochOwned(1, OWNER_ID)); + + let kept_ours = SingleEpochContractBucket(1, CONTRACT_ID, 7) + .combine_added_bytes( + SingleEpochOwned(1, OWNER_ID), + 10, + MergingOwnersStrategy::UseOurs, + ) + .expect("should combine"); + assert_eq!(kept_ours, SingleEpochContractBucket(1, CONTRACT_ID, 7)); + } + + #[test] + fn should_raise_an_issue_across_kinds_and_across_buckets() { + assert!(matches!( + SingleEpochOwned(1, OWNER_ID).combine_added_bytes( + SingleEpochContractBucket(1, CONTRACT_ID, 7), + 10, + MergingOwnersStrategy::RaiseIssue, + ), + Err(StorageFlagsError::MergingStorageFlagsFromDifferentOwners(_)) + )); + assert!(matches!( + SingleEpochContractBucket(1, CONTRACT_ID, 7).combine_added_bytes( + SingleEpochContractBucket(1, CONTRACT_ID, 8), + 10, + MergingOwnersStrategy::RaiseIssue, + ), + Err(StorageFlagsError::MergingStorageFlagsFromDifferentOwners(_)) + )); + } + + #[test] + fn should_never_merge_an_identity_whose_id_equals_a_bucket_carrier_key() { + let bucket = bucket_owner(7); + let colliding_identity = SingleEpochOwned(1, bucket.removal_key()); + + let result = colliding_identity.combine_added_bytes( + SingleEpochContractBucket(1, CONTRACT_ID, 7), + 10, + MergingOwnersStrategy::UseTheirs, + ); + + assert!(matches!( + result, + Err(StorageFlagsError::MergingStorageFlagsFromDifferentOwners(_)) + )); + } + + #[test] + fn should_keep_the_typed_owner_when_adding_bytes_in_a_higher_epoch() { + let combined = SingleEpochContractBucket(1, CONTRACT_ID, 7) + .combine_added_bytes( + SingleEpochContractBucket(2, CONTRACT_ID, 7), + 10, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!( + combined, + MultiEpochContractBucket(1, epochs(&[(2, 10)]), CONTRACT_ID, 7) + ); + + let again = combined + .combine_added_bytes( + SingleEpochContractBucket(2, CONTRACT_ID, 7), + 5, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!( + again, + MultiEpochContractBucket(1, epochs(&[(2, 15)]), CONTRACT_ID, 7) + ); + + // same arithmetic as the crate performs for an identity owner + let crate_combined = CrateStorageFlags::SingleEpochOwned(1, OWNER_ID) + .combine_added_bytes( + CrateStorageFlags::SingleEpochOwned(2, OWNER_ID), + 10, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!( + crate_combined, + CrateStorageFlags::MultiEpochOwned(1, epochs(&[(2, 10)]), OWNER_ID) + ); + } + + /// A single epoch element that shrinks in a later epoch has no epoch map + /// to subtract from. The crate returns the old flags before it looks at + /// the strategy; the typed path must still honour it, or a shrinking + /// replace would keep the old owner while a growing one transfers. + #[test] + fn should_transfer_ownership_across_kinds_when_a_single_epoch_element_shrinks_later() { + let identity = RefundOwner::Identity(Identifier::from(OWNER_ID)); + let removed = sectioned(identity.removal_key(), &[(1, 2)]); + + let bucket_to_identity = SingleEpochContractBucket(1, CONTRACT_ID, 7) + .combine_removed_bytes( + SingleEpochOwned(2, OWNER_ID), + &removed, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!(bucket_to_identity, SingleEpochOwned(1, OWNER_ID)); + + let identity_to_bucket = SingleEpochOwned(1, OWNER_ID) + .combine_removed_bytes( + SingleEpochContractBucket(2, CONTRACT_ID, 7), + &removed, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!( + identity_to_bucket, + SingleEpochContractBucket(1, CONTRACT_ID, 7) + ); + + let kept_ours = SingleEpochContractBucket(1, CONTRACT_ID, 7) + .combine_removed_bytes( + SingleEpochOwned(2, OWNER_ID), + &removed, + MergingOwnersStrategy::UseOurs, + ) + .expect("should combine"); + assert_eq!(kept_ours, SingleEpochContractBucket(1, CONTRACT_ID, 7)); + + assert!(matches!( + SingleEpochContractBucket(1, CONTRACT_ID, 7).combine_removed_bytes( + SingleEpochOwned(2, OWNER_ID), + &removed, + MergingOwnersStrategy::RaiseIssue, + ), + Err(StorageFlagsError::MergingStorageFlagsFromDifferentOwners(_)) + )); + + // unowned on either side yields to the owned side, as in the crate + let gained = SingleEpoch(1) + .combine_removed_bytes( + SingleEpochContractBucket(2, CONTRACT_ID, 7), + &removed, + MergingOwnersStrategy::RaiseIssue, + ) + .expect("should combine"); + assert_eq!(gained, SingleEpochContractBucket(1, CONTRACT_ID, 7)); + + // the same owner on both sides is unchanged, matching the crate + let same = SingleEpochOwned(1, OWNER_ID) + .combine_removed_bytes( + SingleEpochOwned(2, OWNER_ID), + &removed, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + let crate_same = CrateStorageFlags::SingleEpochOwned(1, OWNER_ID) + .combine_removed_bytes( + CrateStorageFlags::SingleEpochOwned(2, OWNER_ID), + &removed, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!(same, StorageFlags::from(crate_same)); + } + + #[test] + fn should_keep_the_typed_owner_when_removing_bytes_in_a_higher_epoch() { + let owner = bucket_owner(7); + let flags = MultiEpochContractBucket(1, epochs(&[(2, 20)]), CONTRACT_ID, 7); + + let removed = sectioned(owner.removal_key(), &[(2, 5)]); + let combined = flags + .clone() + .combine_removed_bytes( + SingleEpochContractBucket(2, CONTRACT_ID, 7), + &removed, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!( + combined, + MultiEpochContractBucket(1, epochs(&[(2, 15)]), CONTRACT_ID, 7) + ); + + // removing the whole epoch collapses back to the single epoch variant + let removed_all = sectioned(owner.removal_key(), &[(2, 18)]); + let collapsed = flags + .combine_removed_bytes( + SingleEpochContractBucket(2, CONTRACT_ID, 7), + &removed_all, + MergingOwnersStrategy::UseTheirs, + ) + .expect("should combine"); + assert_eq!(collapsed, SingleEpochContractBucket(1, CONTRACT_ID, 7)); + } + + #[test] + fn should_reject_a_newer_base_epoch_merging_into_an_older_one() { + assert!(matches!( + SingleEpochContractBucket(2, CONTRACT_ID, 7).combine_added_bytes( + SingleEpochContractBucket(1, CONTRACT_ID, 7), + 10, + MergingOwnersStrategy::UseTheirs, + ), + Err(StorageFlagsError::MergingStorageFlagsWithDifferentBaseEpoch(_)) + )); + } +} + +mod update { + use super::*; + + fn bigger(added_bytes: u32) -> StorageCost { + StorageCost { + added_bytes, + replaced_bytes: 1, + removed_bytes: NoStorageRemoval, + } + } + + #[test] + fn should_behave_like_the_crate_for_identity_owned_flags() { + let old = SingleEpochOwned(1, OWNER_ID).to_element_flags(); + let mut typed_new = SingleEpochOwned(2, OWNER_ID).to_element_flags(); + let mut crate_new = typed_new.clone(); + + let typed_changed = StorageFlags::update_element_flags_typed( + &bigger(10), + Some(old.clone()), + &mut typed_new, + ) + .expect("should update"); + let crate_changed = + CrateStorageFlags::update_element_flags(&bigger(10), Some(old), &mut crate_new) + .expect("should update"); + + assert_eq!(typed_changed, crate_changed); + assert_eq!(typed_new, crate_new); + assert_eq!( + StorageFlags::deserialize(&typed_new).expect("should decode"), + Some(MultiEpochOwned(1, epochs(&[(2, 10)]), OWNER_ID)) + ); + } + + #[test] + fn should_grow_a_bucket_owned_element_into_a_multi_epoch_bucket() { + let old = SingleEpochContractBucket(1, CONTRACT_ID, 7).to_element_flags(); + let mut new_flags = SingleEpochContractBucket(2, CONTRACT_ID, 7).to_element_flags(); + + let changed = + StorageFlags::update_element_flags_typed(&bigger(10), Some(old), &mut new_flags) + .expect("should update"); + + assert!(changed); + assert_eq!( + StorageFlags::deserialize(&new_flags).expect("should decode"), + Some(MultiEpochContractBucket( + 1, + epochs(&[(2, 10)]), + CONTRACT_ID, + 7 + )) + ); + } + + #[test] + fn should_transfer_a_replaced_element_to_the_new_typed_owner() { + let old = SingleEpochOwned(1, OWNER_ID).to_element_flags(); + let mut new_flags = SingleEpochContractBucket(1, CONTRACT_ID, 7).to_element_flags(); + + let changed = + StorageFlags::update_element_flags_typed(&bigger(10), Some(old), &mut new_flags) + .expect("should update"); + + assert!( + changed, + "the owner already matches but the header grew from 35 to 37 bytes, so \ + GroveDB has to price the element again" + ); + assert_eq!( + StorageFlags::deserialize(&new_flags).expect("should decode"), + Some(SingleEpochContractBucket(1, CONTRACT_ID, 7)) + ); + + // the same owner kind on both sides keeps the width, so an unchanged + // merge is reported as unchanged + let old = SingleEpochOwned(1, OWNER_ID).to_element_flags(); + let mut same_kind = SingleEpochOwned(1, OWNER_ID).to_element_flags(); + let changed = + StorageFlags::update_element_flags_typed(&bigger(10), Some(old), &mut same_kind) + .expect("should update"); + assert!(!changed); + } + + /// GroveDB may price one replace as a shrink first and, once the flags + /// changed width, as same size. Both passes must name the same owner. + #[test] + fn should_resolve_a_same_size_update_the_same_way_as_a_shrink() { + let identity = RefundOwner::Identity(Identifier::from(OWNER_ID)); + let old = SingleEpochOwned(1, OWNER_ID).to_element_flags(); + let proposed = SingleEpochContractBucket(2, CONTRACT_ID, 7).to_element_flags(); + + let shrink = StorageCost { + added_bytes: 0, + replaced_bytes: 1, + removed_bytes: sectioned(identity.removal_key(), &[(1, 2)]), + }; + let mut after_shrink = proposed.clone(); + StorageFlags::update_element_flags_typed(&shrink, Some(old.clone()), &mut after_shrink) + .expect("should update"); + + let same_size = StorageCost { + added_bytes: 0, + replaced_bytes: 1, + removed_bytes: NoStorageRemoval, + }; + let mut after_same_size = proposed; + let changed = + StorageFlags::update_element_flags_typed(&same_size, Some(old), &mut after_same_size) + .expect("should update"); + + assert!(changed); + assert_eq!(after_shrink, after_same_size); + assert_eq!( + StorageFlags::deserialize(&after_same_size).expect("should decode"), + Some(SingleEpochContractBucket(1, CONTRACT_ID, 7)) + ); + + // a multi epoch element keeps its whole map on a same size update + let old = MultiEpochOwned(1, epochs(&[(2, 20)]), OWNER_ID).to_element_flags(); + let mut new_flags = SingleEpochContractBucket(3, CONTRACT_ID, 7).to_element_flags(); + StorageFlags::update_element_flags_typed(&same_size, Some(old), &mut new_flags) + .expect("should update"); + assert_eq!( + StorageFlags::deserialize(&new_flags).expect("should decode"), + Some(MultiEpochContractBucket( + 1, + epochs(&[(2, 20)]), + CONTRACT_ID, + 7 + )) + ); + } + + #[test] + fn should_shrink_a_bucket_owned_element_against_its_sectioned_removal() { + let owner = bucket_owner(7); + let old = + MultiEpochContractBucket(1, epochs(&[(2, 20)]), CONTRACT_ID, 7).to_element_flags(); + let mut new_flags = SingleEpochContractBucket(2, CONTRACT_ID, 7).to_element_flags(); + let cost = StorageCost { + added_bytes: 0, + replaced_bytes: 1, + removed_bytes: sectioned(owner.removal_key(), &[(2, 5)]), + }; + + let changed = StorageFlags::update_element_flags_typed(&cost, Some(old), &mut new_flags) + .expect("should update"); + + assert!(changed); + assert_eq!( + StorageFlags::deserialize(&new_flags).expect("should decode"), + Some(MultiEpochContractBucket( + 1, + epochs(&[(2, 15)]), + CONTRACT_ID, + 7 + )) + ); + } + + #[test] + fn should_keep_old_epochs_on_a_same_size_update_and_pass_through_inserts() { + let old = SingleEpochContractBucket(9, CONTRACT_ID, 7).to_element_flags(); + let mut new_flags = SingleEpochContractBucket(1, CONTRACT_ID, 7).to_element_flags(); + let same_size = StorageCost { + added_bytes: 0, + replaced_bytes: 1, + removed_bytes: NoStorageRemoval, + }; + let changed = + StorageFlags::update_element_flags_typed(&same_size, Some(old.clone()), &mut new_flags) + .expect("should update"); + assert!(changed); + assert_eq!(new_flags, old); + + let mut inserted = SingleEpochContractBucket(1, CONTRACT_ID, 7).to_element_flags(); + let changed = StorageFlags::update_element_flags_typed(&bigger(10), None, &mut inserted) + .expect("should update"); + assert!(!changed); + } + + #[test] + fn should_reject_removing_flags_from_a_flagged_element() { + let old = SingleEpochContractBucket(1, CONTRACT_ID, 7).to_element_flags(); + let mut new_flags = vec![]; + let result = + StorageFlags::update_element_flags_typed(&bigger(10), Some(old), &mut new_flags); + assert!(matches!( + result, + Err(Error::StorageFlags(StorageFlagsError::RemovingFlagsError( + _ + ))) + )); + } +} diff --git a/packages/rs-drive/src/util/storage_flags/update.rs b/packages/rs-drive/src/util/storage_flags/update.rs new file mode 100644 index 00000000000..ddde89bc8fa --- /dev/null +++ b/packages/rs-drive/src/util/storage_flags/update.rs @@ -0,0 +1,104 @@ +//! Updating the flags of an element that is replaced in a batch. + +use super::{MergingOwnersStrategy, StorageFlags}; +use crate::error::Error; +use grovedb::ElementFlags; +use grovedb_costs::storage_cost::transition::OperationStorageTransitionType; +use grovedb_costs::storage_cost::StorageCost; +use grovedb_epoch_based_storage_flags::error::StorageFlagsError; + +impl StorageFlags { + /// The batch flag update closure for typed owners: the crate's rule + /// expressed over the typed flags. + /// + /// A replace merges the old flags into the new ones with `UseTheirs`, so + /// new flags naming a different owner transfer the bytes to that owner. + /// That holds across kinds: an identity can hand bytes to a contract + /// bucket and a bucket to an identity, and it holds for every replace + /// shape, including one that moves no bytes (the crate keeps the old + /// owner there; this generation transfers, so that a replace GroveDB + /// first priced as a shrink resolves the same way when it is priced + /// again as same size). + /// + /// GroveDB prices a replace with the old flags attached and re-prices + /// only when this closure reports a change, so a change of header width + /// (35 bytes for an identity owner, 37 for a bucket owner) is reported + /// even when the merged flags already equal the proposed ones. + pub fn update_element_flags_typed( + cost: &StorageCost, + old_flags: Option, + new_flags: &mut ElementFlags, + ) -> Result { + // if there were no flags before then the new flags are used + let Some(old_flags) = old_flags else { + return Ok(false); + }; + + let maybe_old_storage_flags = Self::from_element_flags_ref(&old_flags)?; + let new_storage_flags = Self::from_element_flags_ref(new_flags)?.ok_or( + StorageFlagsError::RemovingFlagsError( + "removing flags from an item with flags is not allowed".to_string(), + ), + )?; + let Some(old_storage_flags) = maybe_old_storage_flags else { + return Err(StorageFlagsError::RemovingFlagsError( + "old storage flags missing during update".to_string(), + ) + .into()); + }; + + match &cost.transition_type() { + OperationStorageTransitionType::OperationUpdateBiggerSize => { + let combined_storage_flags = old_storage_flags.combine_added_bytes( + new_storage_flags, + cost.added_bytes, + MergingOwnersStrategy::UseTheirs, + )?; + Ok(Self::replace_and_report( + &old_flags, + combined_storage_flags, + new_flags, + )) + } + OperationStorageTransitionType::OperationUpdateSmallerSize => { + let combined_storage_flags = old_storage_flags.combine_removed_bytes( + new_storage_flags, + &cost.removed_bytes, + MergingOwnersStrategy::UseTheirs, + )?; + Ok(Self::replace_and_report( + &old_flags, + combined_storage_flags, + new_flags, + )) + } + OperationStorageTransitionType::OperationUpdateSameSize => { + let combined_storage_flags = old_storage_flags + .combine_same_size(new_storage_flags, MergingOwnersStrategy::UseTheirs)?; + Ok(Self::replace_and_report( + &old_flags, + combined_storage_flags, + new_flags, + )) + } + _ => Ok(false), + } + } + + /// Writes the combined flags over the proposed ones and reports whether + /// GroveDB has to price the element again: when the bytes changed, or + /// when the flags kept their bytes but differ in width from the old + /// flags the replace was priced with. + fn replace_and_report( + old_flags: &ElementFlags, + combined: StorageFlags, + new_flags: &mut ElementFlags, + ) -> bool { + let combined_flags = combined.to_element_flags(); + let changed = combined_flags != *new_flags; + if changed { + *new_flags = combined_flags; + } + changed || old_flags.len() != new_flags.len() + } +}