Scipy

[freakboy3742]

What is the PyPI name of the package you would like to see added?

scipy

Additional details

Major blocker is availability of a Fortran compiler that can target iOS

[freakboy3742]

Via @ydesgagn - he reports that it's possible to get flang working for iOS by running it through a Docker container.

Use this docker file to prepare a flang image:

FROM ubuntu:jammy

# persistent dependencies
RUN set -eux ; \
    apt-get -y update ; \
    apt-get -y --no-install-recommends install ca-certificates build-essential wget libncurses5 ;

# flang
RUN wget https://github.com/flang-compiler/flang/releases/download/flang_20190329/flang-20190329-x86-70.tgz ; \
    tar -xvzf *.tgz ; \
    rm -f *.tgz ; \
    ldconfig ;

Then, use this script (flang.sh) to execute the container:

#!/usr/bin/env bash

# shellcheck disable=SC2068
docker exec -it flang /root/host/proxy.sh "${PWD}" ${@}
cp -rf inbox/* "${PWD}" &> /dev/null
rm -rf inbox/* &> /dev/null

The replacement gfortran is a Python script that fixes command line arguments and calls flang.sh:

Details

#!/usr/bin/env python3

import sys
import os
import shlex
import subprocess
import string
import random
import shutil

def join_args(args):
    return ' '.join(shlex.quote(x) for x in args)

if "-dumpversion" in sys.argv:
    print("10.2.0")
    sys.exit(0)

arguments = []
sys_args = sys.argv
del sys_args[0]

try:
    for arg in os.environ["ARCHFLAGS"].split(" "):
        sys_args.append(arg)
except KeyError:
    pass

if "-h" in sys_args or "--help" in sys_args:
    print("The running executable emulates a Fortran compiler. If '-arch arm64' is passed, the compiler will produce an arm64 object file for iOS from the passed source. If not, gfortran located in '/opt/homebrew/bin/' will be executed.")
    print("Because flang runs in a Docker container, only files under '/Users/', '/var/folders' or '/Library' can be compiled.")
    sys.exit(0)

all_args_are_object_files = True

for arg in sys_args:

    if os.path.isfile(arg) and not arg.endswith(".o"):
        all_args_are_object_files = False

if (not "-c" and not all_args_are_object_files) in sys_args or not "-arch arm64" in " ".join(sys_args):
    print("The executed Fortran compiler only supports producing object files for iOS arm64, falling back to gfortran.", file=sys.stderr)
    print("To compile sources for iOS arm64, make sure to add -arch arm64 and -c.", file=sys.stderr)
    sys.exit(os.system(shlex.join(["/opt/homebrew/bin/gfortran"]+sys_args)))

if "-bundle" in sys_args or all_args_are_object_files:
    args = ["clang", "-undefined", "dynamic_lookup", "-shared"]
    try:
        for arg in os.environ["LDFLAGS"].split(" "):
            args.append(arg)
    except KeyError:
        pass
    for arg in sys_args:
        if arg != "-bundle":
            args.append(arg)
    
    command = shlex.join(args)
    sys.exit(os.system(command))

def convert_to_docker_path(arg):
    if arg.startswith("/"):
        arg = "/¡"+arg
    else:
        arg = os.path.join(os.getcwd(), arg)
    
    return arg

the_previous_parameter_was_dash_o = False
the_previous_parameter_was_dash_c = False
the_previous_parameter_was_dash_arch = False
output_path = None
source = None

for arg in sys_args:

    if arg == "-c":
        the_previous_parameter_was_dash_c = True
    elif the_previous_parameter_was_dash_c:
        the_previous_parameter_was_dash_c = False
        source = arg

    if arg == "-o":
        the_previous_parameter_was_dash_o = True
        continue
    elif the_previous_parameter_was_dash_o:
        the_previous_parameter_was_dash_o = False
        output_path = arg
        continue

    if arg == "-arch":
        the_previous_parameter_was_dash_arch = True
        continue
    elif the_previous_parameter_was_dash_arch:
        the_previous_parameter_was_dash_arch = False
        continue

    if arg == "-fallow-argument-mismatch":
        continue

    if os.path.exists(arg):
        arg = convert_to_docker_path(arg)
    if arg.startswith("-I"):
        path = arg.split("-I")[-1]
        arg = "-I"+convert_to_docker_path(path)

    if arg.startswith("/¡"):
        arg = arg[2:]

    arguments.append(arg)

if output_path is None and source is not None:
    parts = source.split(".")
    del parts[-1]
    output_path = os.getcwd()+"/"+".".join(parts)+".o"

if os.path.isfile(output_path):
    sys.exit(0)

print(output_path)

dir = os.path.dirname(os.path.abspath(__file__))
cwd = os.getcwd()

arguments.insert(0, os.path.abspath(os.path.join(dir, "flang.sh")))
arguments.insert(1, "--save-temps")
flang_command = join_args(arguments)

inbox = os.path.join(cwd, ".inbox"+''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(5)))

try:
    os.mkdir(inbox)
except FileExistsError:
    pass

os.chdir(inbox)

os.system(flang_command)
file_path = None

for file in os.listdir("."):
    if not file.endswith(".ll"):
        try:
            os.remove(file)
        except FileNotFoundError:
            pass
    else:
        file_path = os.path.join(os.getcwd(), file)

os.chdir(cwd)
llc = [os.path.join(dir, "llc"), "-mtriple=arm64-apple-ios", "-filetype=obj", file_path, "-o", output_path]
subprocess.run(llc, stdout=None, stderr=None)

shutil.rmtree(inbox)

The proxy.sh that is referenced is:

#!/usr/bin/env bash

cd /root/host || exit
cd "${1}" || exit
flang "${@:2}"

[mhsmith]

Major blocker is availability of a Fortran compiler that can target iOS

@meow464: You used to have a good summary of this at https://meow464.neocities.org/Fortran/state-of-fortran-on-ios. Is there some reason why you've taken it down?

[meow464]

The url changed https://meow464.neocities.org/blog/state-of-fortran-on-ios/

I added support for iOS cross compilation on LFortran but it still can't compile scipy and lapack. Both are being actively worked on.

lfortran/lfortran#1251

[meow464]

I will add instructions for cross compiling for iOS, there was a pull request for that but I forgot about it.

It's not complicated, just cross compile the runtime with a iOS cmake toolchain file then use --target=apple-iphone-ios

[meow464]

@freakboy3742 is that the old or new flang?

[freakboy3742]

@meow464 I can't say I know - I opened this ticket so I had a store of some details that @ydesgagn provided to me; he may be able to comment. If I had to guess based on the Dockerfile, it seems to be using flang 20190329.

[meow464]

That's the old flang also known as flang classic, it's based on the open sourced PGI compiler and it has a problem generating the apple-arm ABI. I spent some 6 months working on it in 2021 but couldn't fix it.
Yes it can generate code and some stuff will work but other stuff will crash.

The problem is with variadic arguments. On arm they should be put on registers until none are available and then on the stack. On apple they should all be put on the stack.

The flang intrinsic functions are written in C and compiled with the apple abi (as they should) but the fortran code is compiled with the arm abi.

[freakboy3742]

Ah - good to know. Thanks for those details. "Run flang through Docker" seemed like a neat trick, but it also seemed like just enough of a neat trick that there was very likely a land mine in waiting.

[freakboy3742]

It's worth noting that Quansight recently blogged about getting flang working with SciPy.

[ydesgagn]

@meow464 We are executing unit tests on an iPad M2 and they all pass for Scipy. Can you be more specific on has a problem generating the apple-arm ABI? Also we have an iPad only application using written in Swift accepted by the Apple Store extensively calling Python 3.11 with numpy, scikit-learn and scipy and all the reported bugs we got so far are due to our failure in our code. Nothing in Python, numpy, scikit-learn and scipy.

@freakboy3742 one of my guy is about to PR in https://github.com/beeware/mobile-forge to add scikit-learn. Will do more after that and we can try Scipy if you think the Docker way is not too hacky.

[freakboy3742]

@ydesgagn "Working with a weird hack" is better than "not working"... but I guess it depends how much engineering overhead we'll need to add to support the hack. If it's something that we can add now, then remove easily later if/when a "direct" flang approach works, then I don't mind living with a hack.

Suffice to say, I'd much rather see a "direct" flang approach - but I'd also like a pony, and a gold Lamborghini :-)

[mhsmith]

Can you be more specific on has a problem generating the apple-arm ABI?

There are some more details and links here, but they may not be up to date.

[serhii-vlasiuk]

For 3.11
Added recipes/scipy/meta.yaml
package:
name: scipy
version: 1.11.3

Getting error
ERROR: Could not find a version that satisfies the requirement numpy==1.23.2 (from versions: 1.26.0)
ERROR: No matching distribution found for numpy==1.23.2
Traceback (most recent call last):
File "/Users/svlas/ydesgagn/github/mobile-forge/venv3.11/bin/forge", line 8, in
sys.exit(main())
^^^^^^
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/main.py", line 205, in main
builder.prepare(clean=args.clean and (p == 0))
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/build.py", line 407, in prepare
self.cross_venv.pip_install(
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/cross.py", line 375, in pip_install
self.run(
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/cross.py", line 362, in run
return subprocess.run(args, **final_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/svlas/ydesgagn/github/Python-Apple-support/install/macOS/macosx/python-3.11.6/lib/python3.11/subprocess.py", line 575, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '-m', 'pip', 'install', '--disable-pip-version-check', '--only-binary', ':all:', '--find-links', '/Users/svlas/ydesgagn/github/mobile-forge/dist', 'build', 'meson-python>=0.12.1,<0.15.0', 'Cython>=0.29.35,<3.0', 'pybind11>=2.10.4,<2.11.1', 'pythran>=0.12.0,<0.15.0', "numpy==1.22.0; platform_machine=='loongarch64'", "numpy==1.22.3; python_version=='3.9' and platform_system=='Windows' and platform_python_implementation != 'PyPy'", "numpy==1.22.3; python_version=='3.10' and platform_system=='Windows' and platform_python_implementation != 'PyPy'", "numpy==1.21.6; python_version=='3.9' and (platform_system!='Windows' and platform_machine!='loongarch64') and platform_python_implementation != 'PyPy'", "numpy==1.21.6; python_version=='3.10' and (platform_system!='Windows' and platform_machine!='loongarch64') and platform_python_implementation != 'PyPy'", "numpy==1.23.2; python_version=='3.11' and platform_python_implementation != 'PyPy'", "numpy>=1.26.0,<1.27.0; python_version=='3.12'", "numpy; python_version>='3.9' and platform_python_implementation=='PyPy'", "numpy>=1.26.0; python_version>='3.13'"]' returned non-zero exit status 1.

Before built numpy 1.26.0
mobile-forge/dist/numpy-1.26.0-cp311-cp311-ios_12_0_iphoneos_arm64.whl

Will appreciate any advice?

[freakboy3742]

It seems to me that the error message is telling you exactly what is going on:

No matching distribution found for numpy==1.23.2

You need to have numpy 1.23.2 available for installation.

On 3.11, 1.23.2 is a special version of numpy - it's the oldest supported version. A lot of numpy-related tooling will deliberately build against the "oldest-supported-numpy" target to ensure maximum ABI compatibility. You'll need to build this version first (forge iOS numpy:1.23.2)