Skip to content

Showcase ERDDAP new features - #353

Open
ocefpaf wants to merge 6 commits into
ioos:mainfrom
ocefpaf:ERDDAP_new_features
Open

ocefpaf wants to merge 6 commits into
ioos:mainfrom
ocefpaf:ERDDAP_new_features

Conversation

@ocefpaf

@ocefpaf ocefpaf commented Oct 7, 2026

Copy link
Copy Markdown
Member

Closes #350

@MathewBiddle this is just a first draft to test if you are OK with us going in this direction for #350. Let me know what you think.

PS: Do you know any croissant expert? It would be nice to do something with the metadata we download beyond spec checking. Maybe attach the data and prepare a pytorch/tensorflow workflow? (Not really running anything, just get as close to running as possible to demonstrate it.)

@ocefpaf ocefpaf changed the title Erddap new features Showcase ERDDAP new features Oct 7, 2026
@MathewBiddle

Copy link
Copy Markdown
Contributor

I like it so far. It would be nice to provide an example pulling the data into pandas as .csv and pulling into pandas as .parquet and highlighting any efficiencies? Maybe amount of bits sent over the wire is less? Resultant data is smaller? Something to show the benefit of having parquet as an option.

@ocefpaf

ocefpaf commented Oct 8, 2026 •

Copy link
Copy Markdown
Member Author

I like it so far. It would be nice to provide an example pulling the data into pandas as .csv and pulling into pandas as .parquet and highlighting any efficiencies? Maybe amount of bits sent over the wire is less? Resultant data is smaller? Something to show the benefit of having parquet as an option.

So, the parquet file is 1 order of magnitude smaller, no surprises there. However, the time it took for the server-side creation of the file + the download was longer than the csv! This is not a valid benchmark at all and your mileage may vary with different datasets, slices/filters, etc. I just wanted to mention this b/c when thinking about cloud infrastructure, if only the download is of concern then parquet is a huge win, if server computing time should be taken into account, then things are not so simple.

@MathewBiddle

Copy link
Copy Markdown
Contributor

ah, I see. Let's just highlight the download difference between the two responses.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[New Notebook]: Demonstrate new ERDDAP endpoints and features

2 participants