No updates since July, website stuck?

[leahneukirchen]

planet.scheme.org hasn't been updated since July 24, but a few blogs had posts in the meantime, does the server need a restart?

[lassik]

The server is fine but the blog aggregator, Planet Venus, is held together with duct tape and bubble gum. It keeps breaking in various ways. In case you have spare cycles to try to fix it, I can give SSH access to the box. Beware that the work is not fun.

[leahneukirchen]

Uh, I'll give it a try, my key is at https://github.com/leahneukirchen.keys

[lassik]

Thanks. Please try ssh leahneukirchen@tuonela.scheme.org

My bash history says that I last tried the following command line:

cd /production/planet/ && sudo -u prod-planet tar -cf - cache | docker run --rm -i prod-planet | sudo -u prod-planet tar -xvf -

and the container stopped with the following error:

Downloading https://i.snap.as/hrje8kg1.png
Saving into images/cache/9f529e65913c774bac185bac4bffdc58acc7a761.png
Downloading https://i.snap.as/vp7SwJDz.png
Saving into images/cache/30fb7e963a45d1d6e9adbc7dbe8f6184ac93bc37.png
*** SYSTEM-ERROR: write failed on #<oport /planet/images-cache.scm 0x7f99e771a900>: Bad file descriptor
    While loading "./../images.scm" at line 91

There have been at least two different ways to have this contraption run as a cron job, and there were problems with each. There have also been many other, non-cron-related problems with the aggregator that I've mercifully forgotten.

[lassik]

The above error comes from images.scm. It's a custom script I wrote that runs using Gauche after Planet Venus has done its job. images.scm downloads all the images from the blog posts into a local cache since Planet Venus doesn't do that.

[leahneukirchen]

I'll have a look, thanks!

[lassik]

Thank you for your service. I wasn't kidding about the duct tape.

The simplest "patch on top of a patch" fix is probably to comment out the shell command that runs images.scm. In that case, the web page generated by Planet Venus ought to use off-site URLs in the <img ...> tags. However, there's no clear reason why images.scm should encounter Bad file descriptor. I hope it's not some obscure feature of Docker.

[leahneukirchen]

So when I ran it, it worked fine... One post has mochibake but that seems unrelated.

[lassik]

lol. It works in mysterious ways.

It's good to know that it's not systematically broken. We've been reluctant to spend time on it since the architecture is pretty much broken beyond repair. It would be best to write a new blog aggregator in Scheme or Racket, but no one has had the time to do it yet.

[leahneukirchen]

The mojibake is a result of this imo weird behavior of feedparser:

    But, if
    the HTTP Content-Type is text/xml, text/*+xml, or
    text/xml-external-parsed-entity, the encoding given in the XML prefix
    within the document is ALWAYS IGNORED and only the encoding given in
    the charset parameter of the HTTP Content-Type header should be
    respected, and it defaults to 'us-ascii' if not specified.

[leahneukirchen]

Where is the image-cache supposed to be stored anyway?

[lassik]

The mojibake is a result of this imo weird behavior of feedparser:

Good to know. Potentially relevant to #5.

Where is the image-cache supposed to be stored anyway?

As regular files under /production/planet/www/images/cache/.

[lassik]

https://planet.scheme.org/ is simply the contents of /production/planet/www/ served as a static site by nginx.

[leahneukirchen]

Those are the images, but I mean the cache file that script writes.

We could try updating Gauche, perhaps there was a bug in it.

[leahneukirchen]

Also, how was this meant to be executed periodically? The cronjob in /etc/cron.d doesn't exist.

[lassik]

Those are the images, but I mean the cache file that script writes.

My mistake. images-cache.scm is not shipped out of the container. It should be part of the tarball.

The culprit is probably this line In planet-container.sh:

tar --sort name -cf - www >&3

should be

tar --sort name -cf - images-cache.scm www >&3

We could try updating Gauche, perhaps there was a bug in it.

No idea.

Also, how was this meant to be executed periodically? The cronjob in /etc/cron.d doesn't exist.

$ sudo find /etc/ -iname '*planet*'
/etc/cron.d/scheme-prod-planet