Skip to content

Feat/bot leaderboard/v2.3#4435

Draft
lsabor wants to merge 19 commits into
mainfrom
feat/bot-leaderboard/v2.3
Draft

Feat/bot leaderboard/v2.3#4435
lsabor wants to merge 19 commits into
mainfrom
feat/bot-leaderboard/v2.3

Conversation

@lsabor
Copy link
Copy Markdown
Contributor

@lsabor lsabor commented Feb 26, 2026

No description provided.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Feb 26, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b6816165-2865-4f7c-a3cd-b6cc32b9e52f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/bot-leaderboard/v2.3

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 26, 2026

🚀 Preview Environment

Your preview environment is ready!

Resource Details
🌐 Preview URL https://metaculus-pr-4435-feat-bot-leaderboard-v2-3-preview.mtcl.cc
📦 Docker Image ghcr.io/metaculus/metaculus:feat-bot-leaderboard-v2.3-842c814
🗄️ PostgreSQL NeonDB branch preview/pr-4435-feat-bot-leaderboard-v2-3
Redis Fly Redis mtc-redis-pr-4435-feat-bot-leaderboard-v2-3

Details

  • Commit: 4ce24ff8f19705f8ccf55af9b703d301a8ac39ec
  • Branch: feat/bot-leaderboard/v2.3
  • Fly App: metaculus-pr-4435-feat-bot-leaderboard-v2-3

ℹ️ Preview Environment Info

Isolation:

  • PostgreSQL and Redis are fully isolated from production
  • Each PR gets its own database branch and Redis instance
  • Changes pushed to this PR will trigger a new deployment

Limitations:

  • Background workers and cron jobs are not deployed in preview environments
  • If you need to test background jobs, use Heroku staging environments

Cleanup:

  • This preview will be automatically destroyed when the PR is closed

Metac bots with metac_bot metadata but used as internal agents (metac-azimuth, metac-agent) should not be included in leaderboard calculations.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Copy Markdown
Contributor Author

@lsabor lsabor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing looked out of place to me in what @colesussmeier added. I'm the initial author, so I can't approve, but feel free to merge as is or address the tiny nit I added.

) -> float:
include_discrimination: bool = False,
min_matches_per_question_for_disc: int = 30,
) -> float | tuple[float, float]:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I would avoid conditionally returning different structure outputs. just return tuple[float, float | None], only populating the second float conditionally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FutureEval Leaderboard Additions

2 participants