Bundle-python

Version 38 (Adrian Georgescu, 11/08/2016 03:52 pm)

1 1 Saúl Ibarra Corretgé
h1. Building a Python Framework to bundle inside Blink
2 1 Saúl Ibarra Corretgé
3 1 Saúl Ibarra Corretgé
In order to avoid using the system Python a custom Framework build is needed. Using a bundled Python version will make the package bigger in size, but all package versions are controlled and not up to the environment. Also, we can use the latest Python version, with latest bugfixes and features, since Apple only updates the system Python version on every major OS release.
4 1 Saúl Ibarra Corretgé
5 13 Saúl Ibarra Corretgé
The following instructions only apply for 64bit builds, 32bit builds are no longer supported.
6 7 Adrian Georgescu
7 22 Adrian Georgescu
Blink dependencies must be installed under the following directory structure:
8 22 Adrian Georgescu
9 22 Adrian Georgescu
* Distribution/Frameworks/
10 22 Adrian Georgescu
* Distribution/Resources/lib
11 1 Saúl Ibarra Corretgé
 
12 7 Adrian Georgescu
h2. Building the Python Framework itself
13 1 Saúl Ibarra Corretgé
14 18 Saúl Ibarra Corretgé
* Install it using Homebrew
15 1 Saúl Ibarra Corretgé
16 1 Saúl Ibarra Corretgé
<pre>
17 18 Saúl Ibarra Corretgé
brew install python
18 1 Saúl Ibarra Corretgé
</pre>
19 1 Saúl Ibarra Corretgé
20 18 Saúl Ibarra Corretgé
The framework will be installed and linked with Homebrew supplied OpenSSL and SQLite versions. Those libraries will need to be copied too.
21 1 Saúl Ibarra Corretgé
22 18 Saúl Ibarra Corretgé
*NOTE*: Be careful when copying the framework around, it contains symlinks and if @cp -r@ is used the size will we doubled, use @cp -a@ instead.
23 22 Adrian Georgescu
24 22 Adrian Georgescu
The Python framework is found in
25 22 Adrian Georgescu
26 22 Adrian Georgescu
<pre>
27 27 Adrian Georgescu
cp -a /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework ~/work/blink/Distribution/Frameworks/
28 22 Adrian Georgescu
</pre>
29 22 Adrian Georgescu
30 2 Saúl Ibarra Corretgé
* Reduce the size of the Python Framework:
31 2 Saúl Ibarra Corretgé
32 1 Saúl Ibarra Corretgé
There are a number of things that can (and must when submitting a sandbox app to Mac App Store) be removed from the framework directory to make it smaller in size:
33 1 Saúl Ibarra Corretgé
34 1 Saúl Ibarra Corretgé
<pre>
35 27 Adrian Georgescu
cd ~/work/blink/Distribution/Frameworks//Python.framework
36 26 Adrian Georgescu
find . -name *.pyc -exec rm -r "{}" \; 
37 26 Adrian Georgescu
find . -name *.pyo -exec rm -r "{}" \; 
38 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/config/python.o
39 24 Adrian Georgescu
rm -r Versions/Current/bin
40 24 Adrian Georgescu
rm -r Versions/Current/Resources/*
41 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/test
42 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/plat-*
43 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/idlelib
44 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/curses
45 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib2to3
46 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-tk
47 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/bsddb
48 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-dynload/gdbm.so
49 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-dynload/readline.so
50 35 Adrian Georgescu
rm -r Versions/2.7/lib/python2.7/site-packages
51 1 Saúl Ibarra Corretgé
</pre>
52 3 Saúl Ibarra Corretgé
53 1 Saúl Ibarra Corretgé
Replace @Versions/Current/lib/python2.7/site.py@ with an empty file.
54 28 Adrian Georgescu
55 28 Adrian Georgescu
<pre>
56 38 Adrian Georgescu
rm ~/work/blink/Distribution/Frameworks//Python.framework/Versions/Current/lib/python2.7/site.py
57 38 Adrian Georgescu
touch ~/work/blink/Distribution/Frameworks//Python.framework/Versions/Current/lib/python2.7/site.py
58 28 Adrian Georgescu
</pre>
59 5 Saúl Ibarra Corretgé
60 34 Adrian Georgescu
Python Framework needs file a Info.plist file under Resources in order to be compatible with latest OSX bundle structure:
61 34 Adrian Georgescu
62 34 Adrian Georgescu
<pre>
63 34 Adrian Georgescu
cp build_scripts/PythonFramework.plist Distribution/Frameworks/Python.framework/Resources/Info.plist           
64 34 Adrian Georgescu
</pre>
65 34 Adrian Georgescu
66 34 Adrian Georgescu
67 5 Saúl Ibarra Corretgé
h2. Compiling PyObjC
68 5 Saúl Ibarra Corretgé
69 13 Saúl Ibarra Corretgé
In order to get a PyObjC version that will work with the framework created above (Python 2.7, 64bits) an equivalent Python must be used to compile it. That is, if has to be a Python 2.7 version (it doesn't have to be the exact version) and it has to be a 64bit version. The MACOSX_DEPLOYMENT_TARGET must also be set to the appropriate value.
70 5 Saúl Ibarra Corretgé
71 5 Saúl Ibarra Corretgé
PyObjcC can be installed with easy_install or pip. We install it in 2 steps to save some compilation time due to a bug in the build system:
72 1 Saúl Ibarra Corretgé
73 5 Saúl Ibarra Corretgé
<pre>
74 5 Saúl Ibarra Corretgé
pip install pyobjc-core
75 5 Saúl Ibarra Corretgé
pip install pyobjc
76 23 Adrian Georgescu
pip install pycrypto
77 1 Saúl Ibarra Corretgé
</pre>
78 1 Saúl Ibarra Corretgé
79 5 Saúl Ibarra Corretgé
When compiling PyObjC a Python package will be created for every system framework, but not all of them are needed (at the moment), so just pick the ones we use:
80 5 Saúl Ibarra Corretgé
81 5 Saúl Ibarra Corretgé
<pre>
82 5 Saúl Ibarra Corretgé
AddressBook
83 5 Saúl Ibarra Corretgé
AppKit
84 1 Saúl Ibarra Corretgé
Cocoa
85 5 Saúl Ibarra Corretgé
CoreFoundation
86 5 Saúl Ibarra Corretgé
Foundation
87 13 Saúl Ibarra Corretgé
JavaScriptCore
88 5 Saúl Ibarra Corretgé
LaunchServices
89 1 Saúl Ibarra Corretgé
PyObjCTools
90 1 Saúl Ibarra Corretgé
Quartz
91 13 Saúl Ibarra Corretgé
ScriptingBridge
92 13 Saúl Ibarra Corretgé
StoreKit
93 1 Saúl Ibarra Corretgé
WebKit
94 8 Adrian Georgescu
objc
95 6 Saúl Ibarra Corretgé
</pre>
96 6 Saúl Ibarra Corretgé
97 1 Saúl Ibarra Corretgé
98 21 Adrian Georgescu
For example this is the content of a Resources/lib bundled with Blink Cocoa as of November 3rd, 2016 (including sipsimple dependencies & all):
99 20 Adrian Georgescu
100 1 Saúl Ibarra Corretgé
<pre>
101 1 Saúl Ibarra Corretgé
AVFoundation
102 1 Saúl Ibarra Corretgé
AddressBook
103 1 Saúl Ibarra Corretgé
AppKit
104 1 Saúl Ibarra Corretgé
Cocoa
105 1 Saúl Ibarra Corretgé
CoreFoundation
106 1 Saúl Ibarra Corretgé
Crypto
107 1 Saúl Ibarra Corretgé
Foundation
108 1 Saúl Ibarra Corretgé
LaunchServices
109 1 Saúl Ibarra Corretgé
PyObjCTools
110 1 Saúl Ibarra Corretgé
Quartz
111 1 Saúl Ibarra Corretgé
ScriptingBridge
112 1 Saúl Ibarra Corretgé
WebKit
113 1 Saúl Ibarra Corretgé
_cffi_backend.so
114 1 Saúl Ibarra Corretgé
_ldap.so
115 1 Saúl Ibarra Corretgé
_markerlib
116 1 Saúl Ibarra Corretgé
application
117 1 Saúl Ibarra Corretgé
cffi
118 1 Saúl Ibarra Corretgé
cjson.so
119 1 Saúl Ibarra Corretgé
cryptography
120 1 Saúl Ibarra Corretgé
cryptography-1.5.1.dist-info
121 1 Saúl Ibarra Corretgé
dateutil
122 1 Saúl Ibarra Corretgé
dns
123 1 Saúl Ibarra Corretgé
dsml.py
124 1 Saúl Ibarra Corretgé
enum
125 1 Saúl Ibarra Corretgé
eventlib
126 1 Saúl Ibarra Corretgé
formencode
127 1 Saúl Ibarra Corretgé
gmpy2.so
128 1 Saúl Ibarra Corretgé
gnutls
129 1 Saúl Ibarra Corretgé
greenlet.so
130 1 Saúl Ibarra Corretgé
idna
131 1 Saúl Ibarra Corretgé
ipaddress.py
132 1 Saúl Ibarra Corretgé
ldap
133 1 Saúl Ibarra Corretgé
ldapurl.py
134 1 Saúl Ibarra Corretgé
ldif.py
135 1 Saúl Ibarra Corretgé
lxml
136 1 Saúl Ibarra Corretgé
msrplib
137 1 Saúl Ibarra Corretgé
objc
138 1 Saúl Ibarra Corretgé
otr
139 1 Saúl Ibarra Corretgé
pkg_resources
140 1 Saúl Ibarra Corretgé
pyasn1
141 1 Saúl Ibarra Corretgé
pycparser
142 1 Saúl Ibarra Corretgé
pydispatch
143 1 Saúl Ibarra Corretgé
pytz
144 1 Saúl Ibarra Corretgé
service_identity
145 1 Saúl Ibarra Corretgé
sipsimple
146 1 Saúl Ibarra Corretgé
six.py
147 1 Saúl Ibarra Corretgé
sqlobject
148 1 Saúl Ibarra Corretgé
twisted
149 1 Saúl Ibarra Corretgé
xcaplib
150 21 Adrian Georgescu
</pre>
151 21 Adrian Georgescu
152 21 Adrian Georgescu
153 21 Adrian Georgescu
*NOTE:* The _objc_ package is located inside a _PyObjC_ directory, just copy it from there, without the parent directory.
154 21 Adrian Georgescu
155 21 Adrian Georgescu
*NOTE:* _PyObjCTools_ is not a valid Python package, as it lacks a @__init__.py@ file, an empty one needs to be manually created with this content:
156 21 Adrian Georgescu
157 21 Adrian Georgescu
<pre>
158 21 Adrian Georgescu
__import__('pkg_resources').declare_namespace(__name__)
159 21 Adrian Georgescu
</pre>
160 21 Adrian Georgescu
161 25 Adrian Georgescu
h2. Fix library paths
162 25 Adrian Georgescu
163 25 Adrian Georgescu
All libraries must have their relative path change to the Framework path bundled within Blink.app
164 25 Adrian Georgescu
165 25 Adrian Georgescu
<pre>
166 25 Adrian Georgescu
#!/bin/sh
167 25 Adrian Georgescu
168 25 Adrian Georgescu
old_path="local/lib/\|local/Cellar/\|/usr/local/opt/libmpc/lib/\|/usr/local/opt/mpfr/lib/\|Frameworks/Frameworks/\|/Users/adigeo/work/ag-projects/video/local/lib/"
169 25 Adrian Georgescu
new_path="@executable_path/../Frameworks/"
170 25 Adrian Georgescu
171 25 Adrian Georgescu
for library in $@; do
172 25 Adrian Georgescu
  install_name_tool -id $new_path$library $library
173 25 Adrian Georgescu
  dependencies=$(otool -L $library | grep $old_path | awk '{print $1}')
174 25 Adrian Georgescu
  for dependency in $dependencies; do
175 25 Adrian Georgescu
      new_basename=$(basename $dependency)
176 25 Adrian Georgescu
      new_name="$new_path$new_basename"
177 25 Adrian Georgescu
      echo $dependency $new_name $library
178 25 Adrian Georgescu
      install_name_tool -change $dependency $new_name $library
179 25 Adrian Georgescu
  done
180 1 Saúl Ibarra Corretgé
done
181 1 Saúl Ibarra Corretgé
</pre>
182 1 Saúl Ibarra Corretgé
183 1 Saúl Ibarra Corretgé
A script is available in ./build_scripts/ directory
184 1 Saúl Ibarra Corretgé
185 1 Saúl Ibarra Corretgé
<pre>
186 1 Saúl Ibarra Corretgé
./build_scripts/change_lib_names.sh Distribution/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/*.so
187 1 Saúl Ibarra Corretgé
chmod +w Distribution/Frameworks/Python.framework/Versions/Current/Python
188 1 Saúl Ibarra Corretgé
./build_scripts/change_lib_names.sh Distribution/Frameworks/Python.framework/Versions/Current/Python
189 1 Saúl Ibarra Corretgé
</pre>
190 1 Saúl Ibarra Corretgé
191 33 Adrian Georgescu
*NOTE*: Python.framework as well as all other libraries must be signed using command line tools. Make sure when building Blink that "Code sign on copy" option is disabled for Python.framework. This script can be used to sign all libraries and frameworks
192 33 Adrian Georgescu
193 33 Adrian Georgescu
<pre>
194 33 Adrian Georgescu
sos=`find ./Resources/lib -name *.so`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
195 33 Adrian Georgescu
sos=`find ./Frameworks -name *.dylib`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
196 33 Adrian Georgescu
sos=`find ./Frameworks -name *.so`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
197 33 Adrian Georgescu
sos=`find ./Frameworks -name *.o`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
198 33 Adrian Georgescu
sos=`find ./Frameworks -name *.a`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
199 33 Adrian Georgescu
</pre>
200 33 Adrian Georgescu
201 33 Adrian Georgescu
A script is available in ./build_scripts/ directory
202 33 Adrian Georgescu
203 33 Adrian Georgescu
<pre>
204 33 Adrian Georgescu
./build_scripts/codesign.sh 
205 33 Adrian Georgescu
</pre>
206 29 Adrian Georgescu
207 21 Adrian Georgescu
h2. Module exceptions
208 21 Adrian Georgescu
209 21 Adrian Georgescu
When copying built Python modules into the distribution folder, care must be taken with the 2 following packages:
210 21 Adrian Georgescu
211 21 Adrian Georgescu
* zope: an empty @__init__.py@ file must be created in the @zope@ directory
212 21 Adrian Georgescu
* cryptography: the @*-dist.info@ must be copied too
213 21 Adrian Georgescu
214 21 Adrian Georgescu
h1. Creating a sandbox (Python virtualenv)
215 21 Adrian Georgescu
216 21 Adrian Georgescu
<pre>
217 21 Adrian Georgescu
sudo easy_install pip
218 21 Adrian Georgescu
sudo pip install virtualenv virtualenvwrapper
219 21 Adrian Georgescu
</pre>
220 21 Adrian Georgescu
221 21 Adrian Georgescu
Add to ~.bashrc
222 21 Adrian Georgescu
223 21 Adrian Georgescu
<pre>
224 21 Adrian Georgescu
# Virtualenv
225 21 Adrian Georgescu
export WORKON_HOME=$HOME/.virtualenvs
226 21 Adrian Georgescu
export PIP_VIRTUALENV_BASE=$WORKON_HOME
227 21 Adrian Georgescu
export PIP_RESPECT_VIRTUALENV=true
228 21 Adrian Georgescu
export VIRTUALENVWRAPPER_SCRIPT=/usr/local/bin/virtualenvwrapper.sh
229 21 Adrian Georgescu
[[ -f /usr/local/bin/virtualenvwrapper_lazy.sh ]] && source /usr/local/bin/virtualenvwrapper_lazy.sh
230 21 Adrian Georgescu
</pre>
231 21 Adrian Georgescu
232 21 Adrian Georgescu
Creating a sandbox:
233 21 Adrian Georgescu
234 21 Adrian Georgescu
<pre>
235 21 Adrian Georgescu
mkvirtualenv -p $(which python2.7) sandbox
236 21 Adrian Georgescu
</pre>
237 21 Adrian Georgescu
238 21 Adrian Georgescu
Exiting the sandbox:
239 21 Adrian Georgescu
240 21 Adrian Georgescu
<pre>
241 21 Adrian Georgescu
deactivate
242 21 Adrian Georgescu
</pre>
243 21 Adrian Georgescu
244 21 Adrian Georgescu
Entering the sandbox:
245 21 Adrian Georgescu
246 21 Adrian Georgescu
<pre>
247 21 Adrian Georgescu
workon sandbox
248 19 Adrian Georgescu
</pre>