Layering HTTP Client

[mookums]

This allows us to write HTTP Client layers to add functionality without messing with the core HTTPClient too much. This is just an experiment right now so don't merge but would love to hear thoughts.

This ideally should make cache revalidation easier to implement as it would be entirely local to the Cache Layer and transparent to the other layers.

[karlseguin]

Something like this is needed. A bit of rambling thoughts...

layers

We need singleflight protection (both module import and robots implement their own mechanism for this) and request interception handling (and we should probably make sure we handle robots.txt and caches correct with request interception).

ergonomics

I don't know it this would be implemented as a "layer", but I'd like synchronous request handling to better. Right now ScriptManager handles it, but it should be pushed down to the http client library, since we need it in a few other places:

const res = try http.requestSync(.{.arena = arena, ...});
// res complete

And probably improve the ergonomics for async/callback too so that they can just register 1 doneCallback and get the whole response (only XHR (and in theory fetch) care about streaming, and even they just want the whole body collected at the end).

script caching

The concern with this type of layering is that it makes some things a little harder. I was thinking of how Script caching would work with layers. I think you might end up losing a bit of efficiency (and maybe even consistency), because script caching might need to happen separately from http caching (e.g. in the ScriptManager). Maybe not. But, it did make me realize that the new caching code could itself be split into Storage and HTTP caching.

script manager

Tha Robots and Cache layers are nice since they cover recent additions, I think going back to the original users of HttClient (ScriptManager, XHR and fetch) and see how they can be improved is worthwhile. That's where I think of things like singleflight and synchronous requests and there are probably other things that I'm missing from their code that would be nice to streamline if possible.