Thoughts¶
Note
This is a somewhat open and in-progress section, by definition. As soon as development reaches a minimally mature state, it will probably be either dropped or evolve into some kind of roadmap/whish list – you have been warned! :)
Preliminary words on what pup
should do and how it could work.
That should help:
- Determine usage patterns.
- Define fundamental concepts.
- Identify challenges and requirements.
Things we’d like now¶
- Package Mu Editor for current macOS and Windows versions, into commonly used single file downloads for software distribution and installation..
- Packaged distributions should be signed/notarized per platform requirements.
- Packaging with/under CPython 3.7 and 3.8.
- Distributable artifacts optionally ask for license agreement, during the installation.
- Can work offline, as far as platform signing/notarization requirements allow (but will require some initial “online time”).
Things that need thinking now¶
- Support 32-bit Windows or go 64-bit only?
- Go for Windows
setup.exe
-like installers or MSI based ones, on Windows? - The packaging process varies depending on several things:
platform, final artifact “kind”, and probably many more.
Let’s have
pup
be extendable via plugins. - What’s the usage pattern?
Do we want
make
-like behaviour where dependencies are analysed and incremental “builds” are supported, or do we go for something simpler, more along the lines of “just do it all, please, I don’t care about the details”?
Things we’d like later¶
- Linux Snap, Flatpak, AppImage (no idea what I’m talking about here!). :)
- Support current CPython: soon to be 3.9, then 3.10, and so on.
- GUIs built on top of pygame, pgzero, pyside2, kivy, arcade.
- If it’s
pip install
-able, it should bepup
packageable. - Given that
pup
will probably make things relocatable, producing “portable Windows applications” (no installation needed) should be easy – we want that. pup
should not only be a CLI tool, but also a library with a stable API that other programs can use. How cool will it be when Mu itself gains a “package button” to help beginners share their projects with friends and family? :) (yes, huge challenges here, especially WRT signing and cross-platform-ness).- Support running tests on the pre-packaged result: right before the final put-it-all-in-a-single-file stage, running under the to-be-distributed Python environment.
- Installer and license agreement localization.
Things that need thinking later¶
- Direct to “store” publishing.
- Cross-platform packaging.
Challenges:
- Cross-platform
pip install
-ing. - Cross-platform signing, notarizing and, ultimately, “store” publishing.
- Cross-platform
Packaging a Python GUI Application¶
…aka, the famous last “how hard can it be?” words.
A possible way to package a Python GUI application is as follows:
- Lay a relocatable Python distribution in a newly created build directory.
- Use that Python distribution’s
pip
to install the application. - Create an GUI level thing such that (double)-clicking it launches the application.
- Package the whole thing in a single distributable and installable file.
Easy, see? :)
We might even add another step that distributes the packaged application to the application stores of the world… But let’s not get ahead of ourselves and dig a bit deeper, instead. Of course the devil will be in the details – let’s see…
Really Packaging a Python GUI Application¶
Stage 1 | Works from the CLI¶
Lay a relocatable Python distribution in a build directory.
Candidates:
- Gregory Szorc’s Python Standalone Builds. Preliminary ad-hoc testing showed it to work nicely on both macOS and Windows – should work on Linux, too.
- The Python environment that’s running
pup
itself. Would avoid an external dependency (and download!). Maybe making it relocatable is non trivial. Not sure if it could be distributed, license-wise.
Needs thinking:
- How will we know where to get a Python Standalone Build from?
- We should (optionally?) cache such downloads.
Use that Python distribution’s
pip
to install the application. By definition, it will be installed within that environment’ssite-packages
, along with its dependencies.Implies:
- Packaging for the same platform we’re running on.
Otherwise, that Python distribution wouldn’t be executable locally.
More, maybe
pip
does not install non-native binary wheels from PyPI: Windows PyQt5, while packaging on macOS? - The application must be
pip install
-able, - …declare one setuptools entry point,
- …and/or is runnable with
python -m <application.
.
- Packaging for the same platform we’re running on.
Otherwise, that Python distribution wouldn’t be executable locally.
More, maybe
At this point,
the application should be launchable from a CLI,
by either running the pip install
-ed entry point executable
or python -m <application>
within the relocatable Python environment.
Stage 2 | Works from the GUI¶
On macOS:
Create a bare-bones application bundle, maybe off of a template.
Application bundles are directories containing a set of predefined files and sub-directories, holding everything needed to run the application: executable, libraries, resources, etc. For end users, application bundles look like files in the Finder.
Thoughts:
- Maybe cookiecutter can be of use.
- Such template could be somehow bundled, with no need for downloads. Overriding it should be possible, though.
Copy the relocatable Python distribution from Stage 1 | Works from the CLI into the application bundle.
No thoughts, should be trivial: shutil is our friend.
Create the application bundle’s executable, under the
MacOS
sub-directory, as a shell-script that runs the application’s entry point executable.Thoughts:
- Care must be taken to avoid having absolute paths, which the application’s entry point executable probably includes.
- Maybe the templated application bundle can include this from the start.
At this point, the application should be launchable from the Finder.
On Windows:
Create an appliction directory layout, somewhat inspired on macOS’s application bundles.
Thoughts:
- Let’s use the same templated solution as for macOS, if any.
Copy the relocatable Python distribution from Stage 1 | Works from the CLI into it.
Create the application’s explorer clickable thing as a shell link in the root of the application directory layout.
Thoughts:
- It should either run the application’s entry point executable or
the relocatable
python
binary with a-m <application>
argument. Either of these options still needs evaluation: on one hand, we’re not sure yet if the application’s entry point executable is relocatable (thus failing to run when the whole application directory is moved); on the other, goingpython -m <application>
imposes further requirements on the way the application is created. - All in all, this step may not be strictly needed, given that on a later stage everything will be bundled into an installable file that itself will be responsible for creating a Start Menu link, much like this one.
How to do this:
- Use winshell by none other than Tim Golden, a Mu contributor. :)
- Maybe get some ideas from this stack overflow link.
- It should either run the application’s entry point executable or
the relocatable
Stage 3 | Single-file distributable artifact¶
On macOS:
There are two common ways of distributing applications: via DMG (disk image) or PKG (packaged installer) files. The former tends to be simpler and that’s what we’ll be initially targeting. The latter can be supported later, maybe via a plugin.
Create DMG file containing the application bundle from the previous stage.
For consideration:
- Include a link to the
Applications
directory. - Have it displayed in a nice visual layout.
- When opened, optionally require accepting a license before mounting.
How to do this:
- Run the
hdiutil
command with the proper arguments. - Use dmgbuild.
Eventually useful:
- Include a link to the
On Windows:
There are two common ways of distributing installable programs:
setup.exe
-like installers, and MSI files.
I tend to prefer the latter because they are
natively supported,
but I suppose supporting both is feasible given that,
up until now,
Mu has been distributed with a setup.exe
-like installer –
so there’s that!
Food for thought.
AFAICT,
there are no native installer creation tools on Windows.
To create setup.exe
-like installers
either NSIS or
Inno Setup
can be used,
and create MSI installer files,
the WiX toolset
can be used, instead.
Regardless of the case, the general procedure will be:
- Create an input file with the necessary packaging specification, per the selected tool, maybe off a template.
- Run the tool with that.
Possible Usage¶
Packaging¶
Keeping things simple and aligned with the things we want, the basic, “it just works” usage shall be:
$ pup package <src>
Where:
<src>
will be a directory containing apip install
-able application source. If omitted, the CWD will be assumed.- The final single-file artifact will be put into
<src>/dist
, in a file named after the application, including its version and, maybe, some other meta-data: Python version, 32/64 bitness, Portable ZIP- vs. NSIS- vs. Inno Setup-packaged, more?… - Packaging will default to produce a packaged application for the same platform where the packaging process is run, bundling the same major version of Python.
- As the process progresses,
one per-stage sub-directory will probably be created under
<src>/build/pup
and left alone for later analysis, maybe.
Some optional/required parameters can be envisioned:
- If there are multiple output format options for the platform
(Windows MSI vs.
setup.exe
-based installer, for example), maybe a--output-format=<format>
will be required (unless a default one is implemented). - Signing will very probably require some kind of
--signer=<whom>
argument so that the correct code-signing certificate can be used (again, unless an effective default certificate identification mechanism is put in place) - Not sure if a
--preview
/--flight-check
option will be useful: maybe it could not only run a few validations, but also report on its decisions, displaying the metadata for the application to be packaged, which relocatable Python will be downloaded, (if any, otherwise, which cached version will be used), which final packaging format will be used, whether or not required third-party tools are found (say, XCode Command Line Tools, needed for signing on macOS, or NSIS or the WiX toolset for final Windows packaging, and more). On possibly very useful output could be the “packaging plan”, as a series of stages and steps.
Interesting future options:
--platform=<platform>
should support the production of a foreign-platform single-file application distribution artifact.
Some thoughts:
- Running the packaging process up until a specific stage or step might be useful.
- Tracking cross-stage/step dependencies is very non-trivial:
pup
will probably not implement amake
-like behaviour. It will probably always “start from scratch”. This raises the question: how should it behave when it finds the left-overs of a previous run, say, under the<src>/build/pup
directory?
Testing¶
Testing the application under the packaged environment will be a valuable capability. It might be triggered with:
$ pup test <src>
…where the <src>
argument will be optional as with the package
command.
The process will probably need to:
- Complete Stage 1 | Works from the CLI.
- Then install any test dependencies on that environment.
- Finally running them.
The way to handle the test dependency installation and test running is non-obvious, for now. One possibility could be:
- Requiring that the application source specifies a pre-defined
setuptools extra
called
dev
(ortest
, maybe?), declaring the test running dependencies. - Having an optional
--run-tests-with=<python-args>
that would havepup
pick up on<python-args>
and run the tests by calling the Python executable in the Stage 1 | Works from the CLI output directory, like.../python <python-args>
. This would support running:- unittest-based tests
with
--run-tests-with='-m unittest'
, - pytest-based tests
with
--run-tests-with='-m pytest'
, - Twisted Trial-based tests
with
--run-tests-with='-m twisted.trial ...'
, - …and more, I suppose.
- unittest-based tests
with
Cleaning Up¶
We want pup
to be a nice citizen to your filesystem layout and disk utilization so
a single command will used to remove files and directories produced by itself:
$ pup cleanup
Removes the intermediate build directories.
Then, maybe the following options could be useful:
--caches
removing all cached files that were previously downloaded.--artifacts
removing allpup
-produced artifacts.--build
(would be the implicit default, per the note above: do we want that?)--all
might be friendly, too.
More Thoughts¶
As progress is made, maybe more thoughts are written down here. Or maybe in other, more specific sections. We will see…