Deploying Flog (this Flask-Powered Blog) on a Raspberry Pi Model B

Raspberry Pi | Python | Linux | Flask | swap

For now this blog is hosted on a headless Raspberry Pi running Arch Linux. When it came to deploying the application for the first time I ran into issues pulling in lxml, a dependency of the project.

~ $ cd py_toy/
~/py_toy $ virtualenv venv
Using base prefix '/usr'
New python executable in venv/bin/python3
Also creating executable in venv/bin/python
Installing setuptools, pip...done.
~/py_toy $ . venv/bin/activate
(venv)~/py_toy $ time pip install lxml
[ output truncated ]
    creating build/temp.linux-armv6l-3.4/src/lxml

    gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=armv6 -mfloat-abi=hard -mfpu=vfp -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-3ikmlcvh/lxml/src/lxml/includes -I/usr/include/python3.4m -c src/lxml/lxml.etree.c -o build/temp.linux-armv6l-3.4/src/lxml/lxml.etree.o -w

    gcc: internal compiler error: Killed (program cc1)

    Please submit a full bug report,

    with preprocessed source if appropriate.

    See <> for instructions.

    /usr/lib/python3.4/distutils/ UserWarning: Unknown distribution option: 'bugtrack_url'


    error: command 'gcc' failed with exit status 4

    Command "/root/py_toy/venv/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-3ikmlcvh/lxml/';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))" install --record /tmp/pip-ytt52_hh-record/install-record.txt --single-version-externally-managed --compile --install-headers /root/py_toy/venv/include/site/python3.4" failed with error code 1 in /tmp/pip-build-3ikmlcvh/lxml

real    27m18.069s
user    26m55.390s
sys     0m11.620s
(venv)~/py_toy $

So what was happening? Turns out I was exhausting all 512 MB of memory. So as a workaround, I temporarily employed a 512 MB swapfile on the SD card.

(venv)~/py_toy $ sudo dd if=/dev/zero of=/swapfile bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 132.228 s, 4.1 MB/s
(venv)~/py_toy $ sudo mkswap /swapfile
Setting up swapspace version 1, size = 524284 KiB
no label, UUID=c45c43c4-00dd-463b-a5ca-30ddeb0b102e
(venv)~/py_toy $ sudo chmod 600 /swapfile
(venv)~/py_toy $ sudo swapon /swapfile

Another attempt:

(venv)~/py_toy $ time pip install lxml 
Collecting lxml
  Using cached lxml-3.4.2.tar.gz
    Building lxml version 3.4.2.
    Building without Cython.
    Using build configuration of libxslt 1.1.28
    Building against libxml2/libxslt in the following directory: /usr/lib
    /usr/lib/python3.4/distutils/ UserWarning: Unknown distribution option: 'bugtrack_url'
Installing collected packages: lxml
  Running install for lxml
    Building lxml version 3.4.2.
    Building without Cython.
    Using build configuration of libxslt 1.1.28
    Building against libxml2/libxslt in the following directory: /usr/lib
    building 'lxml.etree' extension
    gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=armv6 -mfloat-abi=hard -mfpu=vfp -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-8m8hfu47/lxml/src/lxml/includes -I/usr/include/python3.4m -c src/lxml/lxml.etree.c -o build/temp.linux-armv6l-3.4/src/lxml/lxml.etree.o -w
    gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro build/temp.linux-armv6l-3.4/src/lxml/lxml.etree.o -L/usr/lib -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -lpython3.4m -o build/lib.linux-armv6l-3.4/lxml/
    building 'lxml.objectify' extension
    gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=armv6 -mfloat-abi=hard -mfpu=vfp -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-8m8hfu47/lxml/src/lxml/includes -I/usr/include/python3.4m -c src/lxml/lxml.objectify.c -o build/temp.linux-armv6l-3.4/src/lxml/lxml.objectify.o -w
    gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro build/temp.linux-armv6l-3.4/src/lxml/lxml.objectify.o -L/usr/lib -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -lpython3.4m -o build/lib.linux-armv6l-3.4/lxml/
    /usr/lib/python3.4/distutils/ UserWarning: Unknown distribution option: 'bugtrack_url'
Successfully installed lxml-3.4.2

real    33m46.351s
user    32m43.210s
sys 0m16.700s
(venv)~/py_toy $


Sideloading Nexus Devices with the Android SDK

Android | Lollipop | Nexus 7

Android 5.0 "Lollipop" was released yesterday and I wanted to check it out on my Nexus 7 2013 WiFi tablet without having to wait for the update to be pushed. I've done this several times with various Nexus devices now, but so infrequently that I always have to look up how to do it. So this time I'm taking some notes. It should be noted that most of this information is just scraped from the flash-all.{bat,sh} included in the factory image.

First off, Nexus factory images are listed here. Start off by pulling the proper one down, verifying the integrity, and then extracting the tarball:

~ $ mkdir tmp && cd tmp && wget -q
~/tmp $ md5sum razor-lrx21p-factory-ba55c6ab.tgz 
fd868b03bd00074dd7f70e31ad73b25a  razor-lrx21p-factory-ba55c6ab.tgz
~/tmp $ tar xf razor-lrx21p-factory-ba55c6ab.tgz 
~/tmp $ ls razor-lrx21p
bootloader-flo-flo-04.04.img  flash-all.bat

To start the bootloader will need to be unlocked. This will wipe the data on the device, but once unlocked this is no longer an issue when flashing in the future. Sideloading the image requires the Android SDK platform tools.

First thing's first: get into the bootloader. Begin by plugging the tablet into the PC and verifying that it's connected.

~/android-studio/sdk/platform-tools $ ./adb devices -l
List of devices attached 
0a16e983               device usb:1-1.2 product:razor model:Nexus_7 device:flo

Now reboot into the bootloader:

~/android-studio/sdk/platform-tools $ ./adb reboot bootloader

Once in the bootloader make sure the device is connected/detected:

~/android-studio/sdk/platform-tools $ sudo ./fastboot devices -l
0a16e983               fastboot usb:1-1.2

If the bootloader is not already unlocked this will need to be done first (note: this wipes the device, so be sure to back things up beforehand). Unlock the bootloader with the following command (follow the prompts on the device; it'll warn about wiping the device clean):

~/android-studio/sdk/platform-tools $ sudo ./fastboot oem unlock
(bootloader) Unlocking bootloader...
(bootloader) erasing userdata...
(bootloader) erasing userdata done
(bootloader) erasing cache...
(bootloader) erasing cache done
(bootloader) Unlocking bootloader done!
OKAY [ 65.559s]
finished. total time: 65.559s

Now flash the bootloader image downloaded earlier:

~/android-studio/sdk/platform-tools $ sudo ./fastboot  flash bootloader ~/tmp/razor-lrx21p/bootloader-flo-flo-04.04.img 
sending 'bootloader' (3911 KB)...
OKAY [  0.192s]
writing 'bootloader'...
OKAY [  1.936s]
finished. total time: 2.128s

And reboot into the bootloader again:

~/android-studio/sdk/platform-tools $ sudo ./fastboot reboot-bootloader
rebooting into bootloader...
OKAY [  0.006s]
finished. total time: 0.006s

Next flash the zip image:

~/android-studio/sdk/platform-tools $ sudo ./fastboot -w update ~/tmp/razor-lrx21p/ 
archive does not contain 'boot.sig'
archive does not contain 'recovery.sig'
archive does not contain 'system.sig'
Bootloader Version...: FLO-04.04
Baseband Version.....: none
Serial Number........: 0a16e983
checking product...
OKAY [  0.003s]
checking version-bootloader...
OKAY [  0.004s]
sending 'boot' (7154 KB)...
OKAY [  0.347s]
writing 'boot'...
OKAY [  0.420s]
sending 'recovery' (7740 KB)...
OKAY [  0.369s]
writing 'recovery'...
OKAY [  0.297s]
erasing 'system'...
OKAY [  1.510s]
sending 'system' (799121 KB)...
OKAY [ 37.893s]
writing 'system'...
OKAY [ 35.392s]
erasing 'userdata'...
OKAY [ 23.679s]
formatting 'userdata' partition...
Creating filesystem with parameters:
    Size: 28856791040
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 8192
    Inode size: 256
    Journal blocks: 32768
    Blocks: 7045115
    Block groups: 215
    Reserved block group size: 1024
Created filesystem with 11/1761280 inodes and 154578/7045115 blocks
sending 'userdata' (139085 KB)...
writing 'userdata'...
OKAY [ 13.485s]
erasing 'cache'...
OKAY [  0.406s]
formatting 'cache' partition...
Creating filesystem with parameters:
    Size: 587202560
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 7168
    Inode size: 256
    Journal blocks: 2240
    Blocks: 143360
    Block groups: 5
    Reserved block group size: 39
Created filesystem with 11/35840 inodes and 4616/143360 blocks
sending 'cache' (10984 KB)...
writing 'cache'...
OKAY [  1.033s]

finished. total time: 114.857s

And now it's finished. If the bootloader was already locked previously, then you can remove the -w flag in the last command to avoid wiping the device (note: I've not tried this yet).

Debugging CGI / CGit

cgit | cgi

I've been using CGit as a git web front-end for a bit now and just recently started looking into customizing the root header. Assuming I'd have to hack the code to do so, I decided to pull down the source and build it.

~ $ git clone
~ $ cd cgit/
~/cgit $ make get-git && make

After perusing cgit.c and ui-shared.c I was pleasantly surprised to find I wouldn't have to hack anything; it seems that just about everything I wanted to change was configurable via the cgit config file (default: /etc/cgitrc). For example for changing the root title and description, just set root-title and root-desc:

~ $ grep 'root' /etc/cgitrc
root-desc=git repository browser

This worked fine, but what about changing the logo? Just overwriting the logo.png in the resources directory should work, but I want to use an animated gif. The file extension then would be technically incorrect, the worst kind of incorrect. Fortunately, it too looks like this could be set with logo in cgitrc as well. But this didn't seem to work. What's going on?

To run cgit from the CLI, simply do the following. HTML output is printed to stdout and errors are printed to stderr. This will generate the root index content.

~/cgit $ CGIT_CONFIG="./cgitrc" ./cgit 1>stdout.html 2>stderr.log

As an example, to see the content generated for a specific repo, do the following:

~/cgit $ CGIT_CONFIG="./cgitrc" QUERY_STRING="url=bunkergate" ./cgit 1>stdout.html 2>stderr.log
~/cgit $ CGIT_CONFIG="./cgitrc" QUERY_STRING="url=bunkergate/tree/index.html" ./cgit 1>stdout.html 2>stderr.log

It's worth noting that this is probably a basic template for debugging any CGI.

Setting logo in the configuration file didn't seem to have the desired effect. How can we see what's going on? Simple, run it in GDB:

~/cgit $ CGIT_CONFIG="./cgitrc" QUERY_STRING="url=bunkergate/tree/index.html" gdb ./cgit

So what was happening? Well after setting a watchpoint on ctx.cfg.logo I noticed something peculiar. The variable was getting set to the value I expected, but then later it was getting set yet again to the undesired value. Turns out that the problem was that I had set logo twice in the configuration file. Ugh.

Shell Wildcard Channel Attacks

vim | gvim | ibus

Using wildcards with a shell can open you up to channeling attacks (for example, sql injection is a type of channeling attack). When using the * wildcard in particular in a directory containing argument-like-filenames (e.g. -rf) can lead to wild results.

zum Beispiel:

[root@defensecode public]# ls -al
total 20
drwxrwxr-x.  5 leon   leon   4096 Oct 28 17:04 .
drwx------. 22 leon   leon   4096 Oct 28 16:15 ..
drwxrwxr-x.  2 leon   leon   4096 Oct 28 17:04 DIR1
drwxrwxr-x.  2 leon   leon   4096 Oct 28 17:04 DIR2
drwxrwxr-x.  2 leon   leon   4096 Oct 28 17:04 DIR3
-rw-rw-r--.  1 leon   leon      0 Oct 28 17:03 file1.txt
-rw-rw-r--.  1 leon   leon      0 Oct 28 17:03 file2.txt
-rw-rw-r--.  1 leon   leon      0 Oct 28 17:03 file3.txt
-rw-rw-r--.  1 nobody nobody    0 Oct 28 16:38 -rf

[root@defensecode public]# rm *
[root@defensecode public]# ls -al
total 8
drwxrwxr-x.  2 leon   leon   4096 Oct 28 17:05 .
drwx------. 22 leon   leon   4096 Oct 28 16:15 ..
-rw-rw-r--.  1 nobody nobody    0 Oct 28 16:38 -rf

Because rm * expands to:

[user@defensecode WILD]$ rm DIR1 DIR2 DIR3 file1.txt file2.txt file3.txt -rf

This type of attack used in conjunction with seemingly innocuous utilities like tar can lead to execution of arbitrary commands.

gvim + ibus woes / building gvim from source

ibus | gvim | vim

My main development machine is running Ubuntu 12.04 and for a while now I've had issues with using gvim 7.3.x in conjunction with ibus. Specifically upon starting gvim there's a long start up latency (30s+) while the ibus daemon is running. Shutdown the ibus daemon and gvim starts up near instantly. Here is the ticket describing the issue in more detail in Launchpad

Yesterday vim 7.4 was just released and according to the aforementioned ticket the issue was resolved in 7.3.530. Unfortunately neither vim 7.4 nor 7.3.530 are yet in the Ubuntu repos, so I decided to build 7.4 from source. Along the way I'll attempt to jot down some notes here for my future self.

I didn't know before I began, but gvim is actually built from the same source tree as vim source. Before we start, we'll need to make sure we have all the necessary development libraries required to build gvim. Ubuntu docs suggest the following command should work -- I can't say for sure as I had a lot of requisite development libraries installed as other projects I'm working on depend on them as well.

~ $ sudo apt-get build-dep vim

We'll need mercurial to pull down the latest vim sources. I'm going to build the sources in my ~/working directory. Since I doubt I'll need the repo metadata (the included .hg directory) so I'll remove that to reclaim a little space.

~ $ sudo apt-get install mercurial
~ $ cd ~/working/
~/working $ hg clone vim
~/working $ cd vim
~/working/vim $ du -sh
139M    .
~/working/vim $ rm -rf .hg
~/working/vim $ du -sh
65M .

Then to build, change into the src directory and run configure with the options below. I want to build with pretty much all the features so I'll go with the option --with-features=huge; optionally you can use tiny, small, normal, or big instead; a breakdown of the differences is given here. The options of the form --enable-XXXinterp are support for vim plugins written in XXX. The --enable-gui=gtk2 option is obviously for gvim support.

~/working/vim $ cd src
~/working/vim $ ./configure --with-features=huge --enable-rubyinterp  \
                           --enable-pythoninterp --enable-perlinterp \
                           --enable-luainterp --enable-gui=gtk2

Once the configuration script runs without errors, we can then build with make. Note the VMRUNTIMEDIR variable passed to make. This is the location of default system runtime director for vim plugins, docs, dictionaries, etc... for Ubuntu.

~/working/vim $ make VIMRUNTIMEDIR=/usr/share/vim/vim73

At this point one could optionally install vim on their system using the typical make install target. But this feels dirty to me. Plus I prefer to keep all my development tools installed within my home directory. This makes migrating my my development environment much less painful. So I'll define an alias in my ~/.bashrc and be good to go.

~/working/vim $ alias | grep gvim
alias gvim='~/working/vim/src/vim -g -p'