Extract the file-field abstraction into the library

CHANGES

@@ -1,3 +1,16 @@
+0.10
+----
+* added FlexImageField/FlexImageInput, the canonical image field: its value is a `flexfile:<uuid>`
+  reference to a payload the project stores, not the payload itself
+* added hope_flex_fields.references with is_reference, parse_reference, format_reference,
+  is_data_uri, flex_file_src and the DATA_URI_FORMAT/DATA_URI_PREFIX/DEFAULT_MIMETYPE constants
+* added the FILE_URL_NAME configuration entry, the route flex_file_src builds payload URLs from
+* deprecated Base64ImageField/Base64ImageInput, which inline the payload as a `data:` URI. They are
+  kept for one release and are not registered by default: a project still migrating away from them
+  registers them itself, so that swapping the field type and migrating the data can happen in
+  either order
+* added Pillow as a dependency: FlexImageField validates uploads through django.forms.ImageField, which needs it
+
 0.9
 ---
 * enhanced DataChecker's split_data method to accept optional file field names

docs/src/.pages

@@ -2,5 +2,6 @@ nav:
   - Home: index.md
   - install.md
   - contributing.md
+  - file_fields.md
   - reference.md
   - examples.md

docs/src/file_fields.md

@@ -0,0 +1,93 @@
+---
+title: File fields
+---
+
+A flex field can hold a file rather than text. Because a `Fieldset` is normally
+validated against data that ends up serialised as JSON, carrying image bytes
+inline makes those records large and slow to read. File-typed fields therefore
+store a short **reference** instead, and the payload lives wherever the project
+using this library decided to put it.
+
+```
+flexfile:1b4e28ba-2fa1-11d2-883f-0016d3cca427
+```
+
+The library owns the field type, the reference format, and the API for finding
+file fields. It deliberately does **not** own storage: writing the bytes, deciding
+who is allowed to read them back, and serving them are the project's job, since
+those depend on the project's own models and permissions.
+
+## Declaring a file field
+
+Register a `FieldDefinition` with `FlexImageField` as its field type. Every flex
+field built from it renders as a thumbnail with the usual *clear* checkbox, and
+reports itself as a file field to the rest of the library.
+
+`FlexImageField.clean()` validates the upload and then returns the reference that
+was already on the record, discarding the uploaded file. This is intentional: only
+the project's save layer holds both the uploaded file and the record to attach it
+to, so it is the only place that can store the payload and produce a new reference.
+A `clean()` that returned the file would put bytes into data meant to stay JSON.
+
+!!! note
+    Uploads are validated by `django.forms.ImageField`, which relies on
+    [Pillow](https://pypi.org/project/pillow/). It is installed as a dependency
+    of this library.
+
+## Serving payloads
+
+`flex_file_src()` turns a stored value into something usable as an `<img src>`.
+It reverses the route named by `FILE_URL_NAME`, so point that at the view the
+project serves payloads from:
+
+```python
+FLEX_FIELDS_CONFIG = {
+    "FILE_URL_NAME": "myapp:flex_file",
+}
+```
+
+The route is reversed with the file id as its only argument. When it is not
+configured, or does not reverse, `flex_file_src()` returns an empty string and
+logs a warning; the widget then shows the raw value as text instead of a broken
+image.
+
+Values that are not references pass through unchanged when they are inline
+`data:` URIs, so records that predate the move to references keep rendering while
+a project migrates them.
+
+## Finding file fields
+
+Consumers usually need to treat file fields differently from the rest: excluding
+them from an export template, refusing to bulk-edit them, skipping them in a list
+display. Two members cover that, so nobody has to inspect field types by hand:
+
+- [`FlexField.is_file`][hope_flex_fields.models.FlexField.is_file] — whether this
+  field holds file data, that is, whether its type derives from `forms.FileField`.
+- [`DataChecker.get_file_field_names`][hope_flex_fields.models.DataChecker.get_file_field_names]
+  — the names of those fields. Pass `with_prefix=False` for bare names as declared
+  on the fieldset; the default returns them prefixed exactly as
+  `get_form_class()` names them, so they can be used as keys into cleaned data.
+- [`DataChecker.split_data`][hope_flex_fields.models.DataChecker.split_data] —
+  splits a mapping into `fields` and `files` using the above.
+
+## Deprecated inline storage
+
+`Base64ImageField` and `Base64ImageInput` are the previous behaviour: they encode
+the upload into the record as a `data:` URI. They are kept for one release and are
+**not** registered by default, so a project still moving away from them registers
+them itself. The deprecated widget inherits the new rendering, which handles both
+references and `data:` URIs, so swapping the field type and migrating the stored
+data can happen in either order.
+
+## Reference
+
+### ::: hope_flex_fields.references
+    options:
+        show_root_heading: false
+        show_source: true
+
+### ::: hope_flex_fields.fields.FlexImageField
+    options:
+        show_bases: false
+        show_root_heading: true
+        show_source: true

pyproject.toml

@@ -41,6 +41,7 @@ dependencies = [
   "jmespath",
   "jsonpickle",
   "mini-racer",
+  "pillow",
   "python-calamine",
   "requests",
   "xlsxwriter",

src/hope_flex_fields/config.py

@@ -6,6 +6,8 @@
     "API_AUTHENTICATION_CLASSES": ["rest_framework.authentication.BasicAuthentication"],
     "API_TOKEN": "",
     "MASTER_URL": None,
+    # route serving the payload of a file-typed field, reversed with its file id
+    "FILE_URL_NAME": None,
 }
 
 CONFIG.update(**getattr(settings, "FLEX_FIELDS_CONFIG", {}))

src/hope_flex_fields/fields.py

@@ -1,8 +1,14 @@
-from typing import TYPE_CHECKING
+from base64 import b64encode
+from typing import Literal, TYPE_CHECKING
 
 from django import forms
 
+from hope_flex_fields.references import DATA_URI_FORMAT
+from hope_flex_fields.widgets import Base64ImageInput, FlexImageInput
+
 if TYPE_CHECKING:
+    from django.core.files.uploadedfile import UploadedFile
+
     from hope_flex_fields.models import FlexField
 
 
@@ -24,3 +30,44 @@ class IdentityField(forms.CharField):
     def __init__(self, *args, **kwargs):
         kwargs.setdefault("disabled", True)
         super().__init__(*args, **kwargs)
+
+
+class FlexImageField(forms.ImageField):
+    """Image flex field whose value is a reference to an offloaded payload.
+
+    Uploads are validated here but stored elsewhere: only the consuming project
+    holds both the uploaded file and the record to attach it to. ``clean`` keeps
+    the reference already on the record and leaves it to that project's save
+    layer to swap in the new one once the payload has been written.
+    """
+
+    widget = FlexImageInput
+
+    def clean(self, data: "UploadedFile | Literal[False] | None", initial: str | None = None) -> str:
+        cleaned_data = super().clean(data, initial)
+        if not cleaned_data:
+            return ""
+        if hasattr(cleaned_data, "read"):
+            return initial or ""
+        return cleaned_data
+
+
+class Base64ImageField(forms.ImageField):
+    """Deprecated: superseded by :class:`FlexImageField`, kept for one release.
+
+    Encodes the upload inline as a ``data:`` URI, which is what records looked
+    like before payloads were offloaded. It is not registered by default: a
+    project still migrating away from it registers it itself, so that swapping
+    the field type and migrating the data can happen in either order.
+    """
+
+    widget = Base64ImageInput
+
+    def clean(self, data: "UploadedFile | Literal[False] | None", initial: str | None = None) -> str:
+        if cleaned_data := super().clean(data, initial):
+            if hasattr(cleaned_data, "read"):
+                content = b64encode(cleaned_data.read()).decode()
+                return DATA_URI_FORMAT.format(mimetype=data.content_type, content=content)
+            return cleaned_data
+
+        return ""

src/hope_flex_fields/migrations/0018_add_fleximagefield.py

@@ -0,0 +1,44 @@
+# Points the FlexImageField definition at the library implementation of the field.
+#
+# A project that shipped its own FlexImageField before it moved here already has a
+# definition under that name, and FieldDefinition.name is unique, so the existing row
+# is repointed instead of a second one being created. The update goes through the
+# queryset so that the old field_type is never deserialized, which would fail as soon
+# as the project drops its own class.
+#
+# The values are fixed rather than derived from the field class, so that later changes
+# to the class or to the attribute helpers cannot alter what this migration writes.
+
+from django.db import migrations
+
+FIELD_NAME = "FlexImageField"
+FIELD_SLUG = "fleximagefield"
+FIELD_TYPE = "hope_flex_fields.fields.FlexImageField"
+FIELD_ATTRS = {"required": False, "help_text": "", "max_length": None, "allow_empty_file": False}
+
+
+def add_flex_image_field(apps, schema_editor):
+    field_definition = apps.get_model("hope_flex_fields", "FieldDefinition")
+    existing = field_definition.objects.filter(name=FIELD_NAME)
+    if existing.exists():
+        existing.update(field_type=FIELD_TYPE)
+        return
+    field_definition.objects.create(
+        name=FIELD_NAME,
+        slug=FIELD_SLUG,
+        field_type=FIELD_TYPE,
+        attrs=FIELD_ATTRS,
+    )
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        (
+            "hope_flex_fields",
+            "0017_add_identityfield",
+        ),
+    ]
+
+    operations = [
+        migrations.RunPython(add_flex_image_field, migrations.RunPython.noop),
+    ]

src/hope_flex_fields/references.py

@@ -0,0 +1,79 @@
+"""Parsing and formatting of the ``flexfile:`` reference scheme.
+
+A file-typed flex field does not keep its payload inline: the stored value is a
+``flexfile:<uuid>`` reference, and the bytes live wherever the consuming project
+decided to put them. Storage and access control are therefore out of scope here.
+This module owns the reference format alone, so that every consumer reads and
+writes it the same way.
+"""
+
+import logging
+from typing import Any
+from uuid import UUID
+
+from django.urls import NoReverseMatch, reverse
+
+logger = logging.getLogger(__name__)
+
+REFERENCE_PREFIX = "flexfile:"
+DATA_URI_FORMAT = "data:{mimetype};base64,{content}"
+DATA_URI_PREFIX = "data:"
+DEFAULT_MIMETYPE = "application/octet-stream"
+
+
+def is_reference(value: Any) -> bool:
+    """Whether ``value`` is a ``flexfile:`` reference, well formed or not."""
+    return isinstance(value, str) and value.startswith(REFERENCE_PREFIX)
+
+
+def is_data_uri(value: Any) -> bool:
+    """Whether ``value`` is an inline ``data:`` URI, the pre-offload storage format."""
+    return isinstance(value, str) and value.startswith(DATA_URI_PREFIX)
+
+
+def format_reference(file_id: UUID | str) -> str:
+    """Build the reference to store in place of the payload of ``file_id``."""
+    return f"{REFERENCE_PREFIX}{file_id}"
+
+
+def parse_reference(value: Any) -> UUID | None:
+    """Return the file id ``value`` points to, or ``None`` when it points nowhere.
+
+    A malformed reference is not an error: values reach this function from user
+    data and from records written before the offload, so callers get ``None``
+    and decide how to render the field.
+    """
+    if not is_reference(value):
+        return None
+    try:
+        return UUID(value.removeprefix(REFERENCE_PREFIX))
+    except ValueError:
+        return None
+
+
+def flex_file_src(value: Any) -> str:
+    """Return something usable as an ``<img src>``, or ``""`` when nothing can be shown.
+
+    References resolve through the route named by ``FILE_URL_NAME``, which the
+    consuming project serves payloads from. Inline ``data:`` URIs pass through
+    untouched, so that migrated and not-yet-migrated records render alike.
+    """
+    if (file_id := parse_reference(value)) is not None:
+        return _reverse_file_url(file_id)
+    if is_data_uri(value):
+        return value
+    return ""
+
+
+def _reverse_file_url(file_id: UUID) -> str:
+    from hope_flex_fields.config import CONFIG  # noqa
+
+    url_name = CONFIG["FILE_URL_NAME"]
+    if not url_name:
+        logger.warning("FLEX_FIELDS_CONFIG['FILE_URL_NAME'] is not set: cannot build an URL for file %s", file_id)
+        return ""
+    try:
+        return reverse(url_name, args=[file_id])
+    except NoReverseMatch:
+        logger.warning("FLEX_FIELDS_CONFIG['FILE_URL_NAME'] = %r does not reverse with a file id", url_name)
+        return ""

src/hope_flex_fields/registry.py

@@ -3,7 +3,7 @@
 from strategy_field.registry import Registry
 from strategy_field.utils import fqn, import_by_name
 
-from .fields import IdentityField
+from .fields import FlexImageField, IdentityField
 
 
 class FieldRegistry(Registry):
@@ -65,4 +65,5 @@ def as_choices(self):
 field_registry.register(forms.URLField)
 field_registry.register(forms.UUIDField)
 field_registry.register(forms.JSONField)
+field_registry.register(FlexImageField)
 field_registry.register(IdentityField)

src/hope_flex_fields/templates/flex_fields/flex_image_widget.html

@@ -0,0 +1,23 @@
+{% if widget.is_initial %}
+    {{ widget.initial_text }}:
+    {% if widget.image_src %}
+        <a href="{{ widget.image_src }}" target="_blank">
+            <img src="{{ widget.image_src }}" style="max-height: 100px; width: auto" />
+        </a>
+    {% else %}
+        <em>{{ widget.value }}</em>
+    {% endif %}
+    {% if not widget.required %}
+        <input type="checkbox"
+               name="{{ widget.checkbox_name }}"
+               id="{{ widget.checkbox_id }}"
+               {% if widget.attrs.disabled %} disabled{% endif %}
+               {% if widget.attrs.checked %} checked{% endif %} />
+        <label for="{{ widget.checkbox_id }}">
+            {{ widget.clear_checkbox_label }}
+        </label>
+    {% endif %}
+    <br />
+    {{ widget.input_text }}:
+{% endif %}
+<input type="{{ widget.type }}" name="{{ widget.name }}"{% include "django/forms/widgets/attrs.html" %} />

src/hope_flex_fields/widgets.py

@@ -1,5 +1,36 @@
+from typing import Any
+
 from django import forms
 
+from hope_flex_fields.references import flex_file_src
+
+
+class FlexImageInput(forms.ClearableFileInput):
+    """Render a file-typed flex field whose value is a reference, not a file.
+
+    ``ClearableFileInput`` expects the initial value to be a ``FieldFile`` it can
+    take a ``url`` from. Here it is a plain reference string, so both the test
+    for an existing value and the preview URL go through the reference helpers.
+    """
+
+    template_name = "flex_fields/flex_image_widget.html"
+
+    def is_initial(self, value: Any) -> bool:
+        return bool(value)
+
+    def get_context(self, name: str, value: Any, attrs: dict[str, Any] | None) -> dict[str, Any]:
+        context = super().get_context(name, value, attrs)
+        context["widget"]["image_src"] = flex_file_src(value)
+        return context
+
+
+class Base64ImageInput(FlexImageInput):
+    """Deprecated: superseded by :class:`FlexImageInput`, kept for one release.
+
+    It inherits the new rendering, which already handles inline ``data:`` URIs, so
+    that a project can swap the field type and migrate its data in either order.
+    """
+
 
 class JavascriptEditor(forms.Textarea):
     template_name = "flex_fields/editor.html"