This project studies multi-agent aggregation and reliability modeling across multimodal behavioral prediction benchmarks.
benchmarks/
webpage_likability/ Web aesthetics screenshots and human ratings
tweet_engagement/ Content Behavior Corpus / tweet-like engagement data
lambda_ad_memorability/ Ad memorability metadata
ad_ctr_prediction/ Placeholder for Ad CTR data if a public source is found
methods/
webpage_likability/ Main benchmark: generation, aggregators, summaries
content_behavior_corpus/ CBC generation and aggregation experiments
lambda_ad_memorability/ LAMBDA generation and MVP evaluation
third_party/
MM-TPP/ Clean copy of the official MM-TPP implementation
results/
README.md Reserved for cross-benchmark summaries
docs/
papers/ Relevant papers
notes/ Project notes and experiment interpretation
scripts/ Stable entry scripts
- Each active benchmark keeps its own generation scripts, raw agent outputs, aggregation code, and summaries under
methods/<benchmark>/. webpage_likabilityis the current main line because its multi-agent oracle and learned aggregation results are strongest.content_behavior_corpusandlambda_ad_memorabilityare exploratory transfer benchmarks.- Old DanmakuTPP and TAXI-PRO workspaces have been retired from this repo.
Within each mature benchmark directory:
methods/<benchmark>/
scripts/ Benchmark-specific generation and evaluation scripts
data_outputs/ Cached raw agent outputs and small smoke/MVP outputs
aggregators/ Aggregation and reliability modeling methods
summaries/ Result tables and experiment notes
webpage_likability/aggregators/ is organized by method family:
01_oracle/ Mean, best single agent, oracle upper bounds
02_math_aggregators/ Calibration, routers, cross-attention style aggregators
03_agent_modeling/ EM and reliability modeling methods
04_llm_finetuning/ VLM/LLM judge and LoRA experiments