Bundle-python

Version 27 (Adrian Georgescu, 11/08/2016 02:58 pm)

1 1 Saúl Ibarra Corretgé
h1. Building a Python Framework to bundle inside Blink
2 1 Saúl Ibarra Corretgé
3 1 Saúl Ibarra Corretgé
In order to avoid using the system Python a custom Framework build is needed. Using a bundled Python version will make the package bigger in size, but all package versions are controlled and not up to the environment. Also, we can use the latest Python version, with latest bugfixes and features, since Apple only updates the system Python version on every major OS release.
4 1 Saúl Ibarra Corretgé
5 13 Saúl Ibarra Corretgé
The following instructions only apply for 64bit builds, 32bit builds are no longer supported.
6 7 Adrian Georgescu
7 22 Adrian Georgescu
Blink dependencies must be installed under the following directory structure:
8 22 Adrian Georgescu
9 22 Adrian Georgescu
* Distribution/Frameworks/
10 22 Adrian Georgescu
* Distribution/Resources/lib
11 1 Saúl Ibarra Corretgé
 
12 7 Adrian Georgescu
h2. Building the Python Framework itself
13 1 Saúl Ibarra Corretgé
14 18 Saúl Ibarra Corretgé
* Install it using Homebrew
15 1 Saúl Ibarra Corretgé
16 1 Saúl Ibarra Corretgé
<pre>
17 18 Saúl Ibarra Corretgé
brew install python
18 1 Saúl Ibarra Corretgé
</pre>
19 1 Saúl Ibarra Corretgé
20 18 Saúl Ibarra Corretgé
The framework will be installed and linked with Homebrew supplied OpenSSL and SQLite versions. Those libraries will need to be copied too.
21 1 Saúl Ibarra Corretgé
22 18 Saúl Ibarra Corretgé
*NOTE*: Be careful when copying the framework around, it contains symlinks and if @cp -r@ is used the size will we doubled, use @cp -a@ instead.
23 22 Adrian Georgescu
24 22 Adrian Georgescu
The Python framework is found in
25 22 Adrian Georgescu
26 22 Adrian Georgescu
<pre>
27 27 Adrian Georgescu
cp -a /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework ~/work/blink/Distribution/Frameworks/
28 22 Adrian Georgescu
</pre>
29 22 Adrian Georgescu
30 22 Adrian Georgescu
*NOTE*: Python.framework as well as all other libraries must be signed using command line tools. Make sure when building Blink that "Code sign on copy" option is disabled for Python.framework. This script can be used to sign all libraries and frameworks
31 22 Adrian Georgescu
32 22 Adrian Georgescu
<pre>
33 22 Adrian Georgescu
sos=`find ./Resources/lib -name *.so`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
34 22 Adrian Georgescu
sos=`find ./Frameworks -name *.dylib`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
35 22 Adrian Georgescu
sos=`find ./Frameworks -name *.so`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
36 22 Adrian Georgescu
sos=`find ./Frameworks -name *.o`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
37 22 Adrian Georgescu
sos=`find ./Frameworks -name *.a`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
38 22 Adrian Georgescu
</pre>
39 13 Saúl Ibarra Corretgé
40 1 Saúl Ibarra Corretgé
41 2 Saúl Ibarra Corretgé
* Reduce the size of the Python Framework:
42 2 Saúl Ibarra Corretgé
43 1 Saúl Ibarra Corretgé
There are a number of things that can (and must when submitting a sandbox app to Mac App Store) be removed from the framework directory to make it smaller in size:
44 1 Saúl Ibarra Corretgé
45 1 Saúl Ibarra Corretgé
<pre>
46 27 Adrian Georgescu
cd ~/work/blink/Distribution/Frameworks//Python.framework
47 26 Adrian Georgescu
find . -name *.pyc -exec rm -r "{}" \; 
48 26 Adrian Georgescu
find . -name *.pyo -exec rm -r "{}" \; 
49 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/config/python.o
50 24 Adrian Georgescu
rm -r Versions/Current/Mac
51 24 Adrian Georgescu
rm -r Versions/Current/bin
52 24 Adrian Georgescu
rm -r Versions/Current/share
53 24 Adrian Georgescu
rm -r Versions/Current/Resources/*
54 24 Adrian Georgescu
rm -r Versions/Current/Resources/*.app
55 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/test
56 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/plat-*
57 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/idlelib
58 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/curses
59 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib2to3
60 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-tk
61 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/bsddb
62 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-dynload/gdbm.so
63 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-dynload/readline.so
64 1 Saúl Ibarra Corretgé
</pre>
65 3 Saúl Ibarra Corretgé
66 4 Saúl Ibarra Corretgé
* Prevent system paths from being used with this bundle
67 4 Saúl Ibarra Corretgé
68 17 Saúl Ibarra Corretgé
Replace @Versions/Current/lib/python2.7/site.py@ with an empty file.
69 5 Saúl Ibarra Corretgé
70 5 Saúl Ibarra Corretgé
h2. Compiling PyObjC
71 5 Saúl Ibarra Corretgé
72 13 Saúl Ibarra Corretgé
In order to get a PyObjC version that will work with the framework created above (Python 2.7, 64bits) an equivalent Python must be used to compile it. That is, if has to be a Python 2.7 version (it doesn't have to be the exact version) and it has to be a 64bit version. The MACOSX_DEPLOYMENT_TARGET must also be set to the appropriate value.
73 5 Saúl Ibarra Corretgé
74 5 Saúl Ibarra Corretgé
PyObjcC can be installed with easy_install or pip. We install it in 2 steps to save some compilation time due to a bug in the build system:
75 1 Saúl Ibarra Corretgé
76 5 Saúl Ibarra Corretgé
<pre>
77 5 Saúl Ibarra Corretgé
pip install pyobjc-core
78 5 Saúl Ibarra Corretgé
pip install pyobjc
79 23 Adrian Georgescu
pip install pycrypto
80 1 Saúl Ibarra Corretgé
</pre>
81 1 Saúl Ibarra Corretgé
82 5 Saúl Ibarra Corretgé
When compiling PyObjC a Python package will be created for every system framework, but not all of them are needed (at the moment), so just pick the ones we use:
83 5 Saúl Ibarra Corretgé
84 5 Saúl Ibarra Corretgé
<pre>
85 5 Saúl Ibarra Corretgé
AddressBook
86 5 Saúl Ibarra Corretgé
AppKit
87 1 Saúl Ibarra Corretgé
Cocoa
88 5 Saúl Ibarra Corretgé
CoreFoundation
89 5 Saúl Ibarra Corretgé
Foundation
90 13 Saúl Ibarra Corretgé
JavaScriptCore
91 5 Saúl Ibarra Corretgé
LaunchServices
92 1 Saúl Ibarra Corretgé
PyObjCTools
93 1 Saúl Ibarra Corretgé
Quartz
94 13 Saúl Ibarra Corretgé
ScriptingBridge
95 13 Saúl Ibarra Corretgé
StoreKit
96 1 Saúl Ibarra Corretgé
WebKit
97 8 Adrian Georgescu
objc
98 6 Saúl Ibarra Corretgé
</pre>
99 6 Saúl Ibarra Corretgé
100 1 Saúl Ibarra Corretgé
101 21 Adrian Georgescu
For example this is the content of a Resources/lib bundled with Blink Cocoa as of November 3rd, 2016 (including sipsimple dependencies & all):
102 20 Adrian Georgescu
103 1 Saúl Ibarra Corretgé
<pre>
104 1 Saúl Ibarra Corretgé
AVFoundation
105 1 Saúl Ibarra Corretgé
AddressBook
106 1 Saúl Ibarra Corretgé
AppKit
107 1 Saúl Ibarra Corretgé
Cocoa
108 1 Saúl Ibarra Corretgé
CoreFoundation
109 1 Saúl Ibarra Corretgé
Crypto
110 1 Saúl Ibarra Corretgé
Foundation
111 1 Saúl Ibarra Corretgé
LaunchServices
112 1 Saúl Ibarra Corretgé
PyObjCTools
113 1 Saúl Ibarra Corretgé
Quartz
114 1 Saúl Ibarra Corretgé
ScriptingBridge
115 1 Saúl Ibarra Corretgé
WebKit
116 1 Saúl Ibarra Corretgé
_cffi_backend.so
117 1 Saúl Ibarra Corretgé
_ldap.so
118 1 Saúl Ibarra Corretgé
_markerlib
119 1 Saúl Ibarra Corretgé
application
120 1 Saúl Ibarra Corretgé
cffi
121 1 Saúl Ibarra Corretgé
cjson.so
122 1 Saúl Ibarra Corretgé
cryptography
123 1 Saúl Ibarra Corretgé
cryptography-1.5.1.dist-info
124 1 Saúl Ibarra Corretgé
dateutil
125 1 Saúl Ibarra Corretgé
dns
126 1 Saúl Ibarra Corretgé
dsml.py
127 1 Saúl Ibarra Corretgé
enum
128 1 Saúl Ibarra Corretgé
eventlib
129 1 Saúl Ibarra Corretgé
formencode
130 1 Saúl Ibarra Corretgé
gmpy2.so
131 1 Saúl Ibarra Corretgé
gnutls
132 1 Saúl Ibarra Corretgé
greenlet.so
133 1 Saúl Ibarra Corretgé
idna
134 1 Saúl Ibarra Corretgé
ipaddress.py
135 1 Saúl Ibarra Corretgé
ldap
136 1 Saúl Ibarra Corretgé
ldapurl.py
137 1 Saúl Ibarra Corretgé
ldif.py
138 1 Saúl Ibarra Corretgé
lxml
139 1 Saúl Ibarra Corretgé
msrplib
140 1 Saúl Ibarra Corretgé
objc
141 1 Saúl Ibarra Corretgé
otr
142 1 Saúl Ibarra Corretgé
pkg_resources
143 1 Saúl Ibarra Corretgé
pyasn1
144 1 Saúl Ibarra Corretgé
pycparser
145 1 Saúl Ibarra Corretgé
pydispatch
146 1 Saúl Ibarra Corretgé
pytz
147 1 Saúl Ibarra Corretgé
service_identity
148 1 Saúl Ibarra Corretgé
sipsimple
149 1 Saúl Ibarra Corretgé
six.py
150 1 Saúl Ibarra Corretgé
sqlobject
151 1 Saúl Ibarra Corretgé
twisted
152 1 Saúl Ibarra Corretgé
xcaplib
153 21 Adrian Georgescu
</pre>
154 21 Adrian Georgescu
155 21 Adrian Georgescu
156 21 Adrian Georgescu
*NOTE:* The _objc_ package is located inside a _PyObjC_ directory, just copy it from there, without the parent directory.
157 21 Adrian Georgescu
158 21 Adrian Georgescu
*NOTE:* _PyObjCTools_ is not a valid Python package, as it lacks a @__init__.py@ file, an empty one needs to be manually created with this content:
159 21 Adrian Georgescu
160 21 Adrian Georgescu
<pre>
161 21 Adrian Georgescu
__import__('pkg_resources').declare_namespace(__name__)
162 21 Adrian Georgescu
</pre>
163 21 Adrian Georgescu
164 25 Adrian Georgescu
h2. Fix library paths
165 25 Adrian Georgescu
166 25 Adrian Georgescu
All libraries must have their relative path change to the Framework path bundled within Blink.app
167 25 Adrian Georgescu
168 25 Adrian Georgescu
<pre>
169 25 Adrian Georgescu
#!/bin/sh
170 25 Adrian Georgescu
171 25 Adrian Georgescu
old_path="local/lib/\|local/Cellar/\|/usr/local/opt/libmpc/lib/\|/usr/local/opt/mpfr/lib/\|Frameworks/Frameworks/\|/Users/adigeo/work/ag-projects/video/local/lib/"
172 25 Adrian Georgescu
new_path="@executable_path/../Frameworks/"
173 25 Adrian Georgescu
174 25 Adrian Georgescu
for library in $@; do
175 25 Adrian Georgescu
  install_name_tool -id $new_path$library $library
176 25 Adrian Georgescu
  dependencies=$(otool -L $library | grep $old_path | awk '{print $1}')
177 25 Adrian Georgescu
  for dependency in $dependencies; do
178 25 Adrian Georgescu
      new_basename=$(basename $dependency)
179 25 Adrian Georgescu
      new_name="$new_path$new_basename"
180 25 Adrian Georgescu
      echo $dependency $new_name $library
181 25 Adrian Georgescu
      install_name_tool -change $dependency $new_name $library
182 25 Adrian Georgescu
  done
183 25 Adrian Georgescu
done
184 25 Adrian Georgescu
</pre>
185 25 Adrian Georgescu
186 21 Adrian Georgescu
h2. Module exceptions
187 21 Adrian Georgescu
188 21 Adrian Georgescu
When copying built Python modules into the distribution folder, care must be taken with the 2 following packages:
189 21 Adrian Georgescu
190 21 Adrian Georgescu
* zope: an empty @__init__.py@ file must be created in the @zope@ directory
191 21 Adrian Georgescu
* cryptography: the @*-dist.info@ must be copied too
192 21 Adrian Georgescu
193 21 Adrian Georgescu
h1. Creating a sandbox (Python virtualenv)
194 21 Adrian Georgescu
195 21 Adrian Georgescu
<pre>
196 21 Adrian Georgescu
sudo easy_install pip
197 21 Adrian Georgescu
sudo pip install virtualenv virtualenvwrapper
198 21 Adrian Georgescu
</pre>
199 21 Adrian Georgescu
200 21 Adrian Georgescu
Add to ~.bashrc
201 21 Adrian Georgescu
202 21 Adrian Georgescu
<pre>
203 21 Adrian Georgescu
# Virtualenv
204 21 Adrian Georgescu
export WORKON_HOME=$HOME/.virtualenvs
205 21 Adrian Georgescu
export PIP_VIRTUALENV_BASE=$WORKON_HOME
206 21 Adrian Georgescu
export PIP_RESPECT_VIRTUALENV=true
207 21 Adrian Georgescu
export VIRTUALENVWRAPPER_SCRIPT=/usr/local/bin/virtualenvwrapper.sh
208 21 Adrian Georgescu
[[ -f /usr/local/bin/virtualenvwrapper_lazy.sh ]] && source /usr/local/bin/virtualenvwrapper_lazy.sh
209 21 Adrian Georgescu
</pre>
210 21 Adrian Georgescu
211 21 Adrian Georgescu
Creating a sandbox:
212 21 Adrian Georgescu
213 21 Adrian Georgescu
<pre>
214 21 Adrian Georgescu
mkvirtualenv -p $(which python2.7) sandbox
215 21 Adrian Georgescu
</pre>
216 21 Adrian Georgescu
217 21 Adrian Georgescu
Exiting the sandbox:
218 21 Adrian Georgescu
219 21 Adrian Georgescu
<pre>
220 21 Adrian Georgescu
deactivate
221 21 Adrian Georgescu
</pre>
222 21 Adrian Georgescu
223 21 Adrian Georgescu
Entering the sandbox:
224 21 Adrian Georgescu
225 21 Adrian Georgescu
<pre>
226 21 Adrian Georgescu
workon sandbox
227 19 Adrian Georgescu
</pre>