Ask your questions here :)

[PGijsbers]

The Q&A discussion board is for asking any OpenML questions! Whether they are about our organisation, hackathons, websites, or any of our programmatic interfaces! We're here to help 🤗 Just "create a discussion" and ask away!

[BradKML]

Do you think using AI to assist open research for amateurs starting out a bad idea? Or would that be valid for those without a conventional academic background? https://github.com/orgs/openml/discussions/17#discussioncomment-11897058

[PGijsbers]

Hi @BradKML! Sorry for the delay - you can imagine we have been busy lately :)

AI is a tool, in itself it is not bad or good. It's about learning about how and when to use it. Personally I use agents sparingly, mostly for brainstorming, feedback on text, or debugging, all situations where I have the knowledge to judge the output of the models. It is also possible that I might lack the experience working with LLMs to access their full potential, as so far results have been mostly underwhelming to me.¹

I definitely think there is a place for AI in making open science more accessible to "amateurs" or the general public. It can help with programming, data visualization, feedback, and so on. However, there are also dangers. Current agents are frequently giving answers that are incorrect or incomplete, even for simple things like counting. This is why I would generally use the models when I can judge the output as one cannot trust agents blindly, or where accuracy of the output is not important. This means that novices probably should use the models more sparingly in some areas, such as learning new material.²

When it comes to doing "actual science", there is no substitute to knowing what you are doing. That does not mean you need a university degree, but it does mean finding reliable sources to learn the niche you are working on. The good news is that there is a lot of well-made information available for free, whether that's courses, youtube videos, books, and so on. It doesn't even mean you have to know everything when you start, everyone learns along the way, and you can also start with finding more experienced collaborators.

Conversely, AI may also be used by the experts to help make their science more accessible to the general public, by e.g., letting it help come up with layman explanations to the research, as the experts will be able to judge the accuracy of the simplification!

Footnotes

1. It's definitely been great to see the process over the last decade or so, and the agents are a great technological feat. I mean that their usefulness to my personal work has been underwhelming, which in turn leads me to use them less. ↩

2. One promising thing, that I do not have that much experience with, is using LLMs to create quizzes about material you want to learn. By first providing it with the text you want to be quizzed on agents are much less likely to make errors when they quiz you. ↩

[BradKML]

Thanks for the response @PGijsbers and I would like to add one more point: scientific education (including communication to the average populace) and "citizen science" (grassroots-first research) is seriously underexplored. Non-textbook STEM education resources (e.g. physical access for the poor, social labor for hands-on work, time-agnostic low-friction media) are increasingly underfunded in many places, and sometimes not even supported in others.
Conventional academia fails in easing the fears of the masses, while "PopSci" often fails in representing innovation in a sufficient manner. AGI panic through the imagination of scifi is a good example of that. These traditional media makes psuedoscience look more attractive, and we do see a resurgence of that.
LLMs and RAGs should be filling that gap by accelerating engagement with new information through larger error-tolerance and stronger self-correction. The only issue here is that there needs to be a strategy or framework that makes this feasible.

[PGijsbers]

I hope I didn't come off wrong with the "actual science", it was very much in quotes a reason. Citizen science is very meaningful and impactful, and not in any way inferior to a researcher working in a lab -- in fact they can often end up working together! Allowing people to classify galaxies, count birds with their phones, analyze conjectures and many more examples show that there's so much scientific value while also increasing engagement and education of the general public. It's fantastic we live in an age where that's possible, and it's a wonderful thing if we can use AI to enhance or accelerate that - provided we exercise care.

[BradKML]

That is true, the more concerning part is how "care" became a euphemism to limit citizen access through red tape. Streamlining safety controls and make that specifically more accessible would help in that regard. Rather than established fields like ecology and astronomy (hobbyist delegation), or places where it is bona-fide "compute donation" like mathematics or biochemistry (assumption of resources), it would be even more important to focus on frontier work, like social science and AI technology, lending voice to the masses and bridging the gaps in understanding. "Soft science" implies the tenderness of community care rather than bias on "lab coats".

[fingoldo]

It feels like openml was created with the goal of building a superior (to "standard" optimizers like Optuna, Hyperopt etc) metalearning-based HPO/HPT solution in mind, but nowhere on the site or in the docs that idea can't be traced. Is it entirely dropped or never existed? Also, for classification tasks, are raw predicted probabilities stored somewhere? run.predictions seems to only return labels (on the other hand, metrics like ROC AUC are there). Also, when trying to sign up for an API key, confirmation email is not arriving.

[PGijsbers]

There's definitely been work in that area! As early as 2015, auto-sklearn used meta-learning to warm-start Bayesian optimization¹. More recent example for generic warm-starting is work with building portfolios, this is about finding a set of generally good but complementary pipelines, as in e.g., tabrepo² and version 2 of auto-sklearn³. The latter also uses meta-learning to self-configure the optimizer's hyperparameters. There's even work in transferring priors for Bayesian optimization⁴. I also came across this recently though haven't read it yet myself, but seems relevant to your interests.

Footnotes

1. https://proceedings.neurips.cc/paper_files/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf ↩

2. https://arxiv.org/pdf/2311.02971 ↩

3. https://arxiv.org/pdf/2007.04074 ↩

4. https://proceedings.mlr.press/v151/tighineanu22a/tighineanu22a.pdf and https://www.researchgate.net/profile/Matthias-Feurer/publication/323003916_Scalable_Meta-Learning_for_Bayesian_Optimization/links/5f7c3349a6fdccfd7b4a87d9/Scalable-Meta-Learning-for-Bayesian-Optimization.pdf ↩

[fingoldo]

Yes I was asking myself why auto-sklearn only uses meta-learning for a warm start. Imho it there's predictability of final performance depending on meta-features, no Bayesian optimization is needed. But look at this recent paper, curated by the famous Frank Hutter.

In 2.8 s, TabPFN outperforms an ensemble of the strongest baselines tuned for 4 h in a classifcation setting. As a generative transformer-based foundation model, this model also allows fne-tuning, data generation, density estimation and learning reusable embeddings. TabPFN is a learning algorithm that is itself learned across millions of synthetic datasets, demonstrating the power of this approach for algorithm development. By improving modelling abilities across diverse felds, TabPFN has the potential to accelerate scientifc discovery and enhance important decision-making in various domains.

Such speed & accuracy gains I did not have even in my boldest meta-learning dreams )

[PGijsbers]

I agree, the results of TabPFN are remarkable. It's very interesting and impressive they can get such results from training on (cleverly) generated synthetic data.

[fingoldo]

Wondering what's the contributions of metalearning and transformer architecture to that achievements, percentwise.

[BradKML]

@fingoldo maybe code agents and creative agents can work together towards something more efficient, data-to-paper or AI-scientist being two examples

[stheid]

I would like to publish a dataset, but the sign up process seems broken. I have not recieved a activation email. is that semi-automatic or currently broken?

[stheid]

ok, but the "edit dataset" api doesnt allow editing the dataset, but only the metadata. that was a bit confusing to me. i managed it.

[joaquinvanschoren]

Cool 💪
Let me know if we can improve the documentation ❤️

[stheid]

Imho the documentation should state:

edit_dataset:

• Edits the metadata of a dataset. for substituting the dataset look at "create dataset"

create_dataset:

• create a new dataset or a new version of a dataset. If the name of the dataset already exists, the dataset will created as a new version. In case of new version, all provided metadata will be overwritten.

Version parameter

I also think providing a "version" does not make a difference actually, because I accidentally provided the new upload version with the same version, but still the version on the website updated as expected. I think the version argument gets dropped and not used at all. maybe that parameter should also be documented accordingly.

Function names

I think thats all my comments. For the next major version i think it would be good to change the names of the functions to "edit_dataset_metadata" and "upload_dataset" accordingly.

[joaquinvanschoren]

@stheid update: the upload via the website is also fixed

[joaquinvanschoren]

The 'version_label' parameter is a label you can freely choose. It's stored alongside OpenML's version (which is always an incrementing int). I see you indeed set "version_label": "0.1" for both dataset versions.