Category Archives: Hardware and infrastructure

20 Rasperry Pi’s – one massive art installation

A couple of internationally renowned artists asked me for some help with their largest installation to date. As part of Hull’s City of Culture, Davy and Kristin McGuire created a large cardboard city and brought it to life with video projections.

They needed nearly 20 video players, so I created a bootable linux image for the Raspberry Pi which automatically plays a video from a standard location. I copied this to 20 memory cards, and tested them all.

The installation looked amazing and was a great success.

Namepy – on the shoulders of giants

Whilst my core skill/tool is Python, I’m always learning new things, either inside or outside the Python ecosystem. I recently had the pleasure of working with Angular and Python/Flask. Here is a playful application based on these, plus Highcharts.

Going through “Python for Data Analysis”, some of the examples use a database of frequency of (US) baby names since 1880. I thought I’d combine this with a bit of Scrabble™.

In the Python world it’s common to add “py” to a word when making up names, so I’m calling this project “namepy”.

Since I’ll be using various frameworks and libraries, all created by others, I’ve subtitled this “On the shoulders of giants”.

Taking small steps often results in faster progress, so that’s what I’m be doing here.

Technical set up

The source code is at https://github.com/CoachCoen/namepy, with one branch per step.

Many production sites Content Delivery Networks for serving Javascript frameworks and libraries, usually minified, which helps to take the load of the server and may speed up first page load. To keep things simple and stable over time, I’m using full-sized, downloaded, copies.

I’m using WebFaction (affiliate link) as the host, since they make it easy to create Flask, Django and similar projects. And, as a popular host for developers, you’ll find lots of helpful documentation for developers online.

Getting started

Create a project folder

mkdir namepy
cd namepy

At the start of each of the steps

cd (my folder for personal projects)
cd namepy
git clone https://github.com/CoachCoen/namepy.git -b step1 step1

Note: “-b step1” specifies the name of the branch to clone. The second “step1” is the target folder, i.e. namepy/step1.

Next

Continue to Step 1 – Angular “Hello World”

VirtualBox – moving virtual disks

Having installed VirtualBox I created a few virtual machines, with 20GB virtual disks each. After all, I’ve got a 1TB hard disk in my computer.

When Linux started complaining about running out of disk space I realised that my main drive is a 256GB Solid State Drive, and that is where VirtualBox was storing the virtual disks. Time to move the virtual disks. Please note that it is a bit of a manual process and, whilst it worked fine for me, if you’ve already invested a lot of time setting up your virtual machines, I suggest that you test this out by moving one or two VMs first.

  1. Stop all VMs, stop VirtualBox
  2. Find a folder called “VirtualBox VMs” (probably in your home folder), and move it to the new location
    1. Or move just a few files first, to make sure it works
  3. Restart VirtualBox
    1. In File -> Preferences -> General, point the Default Machine Folder to the new location
    2. All your existing VMs are now broken. Delete them
    3. One at a time: Machine -> Add
      1. Select the .vbox file from the new folder

Virtual machines using VirtualBox

Tapping into all the wonderful tools and languages that make our lives as developers so interesting can throw up some fun (?) challenges.

You start installing the latest linter or library, and before you know it you’re in dependency hell. Required libraries require even more libraries, with difference versions numbers and clashing with your existing setup.  If you’re really unlucky you break some essential software along the way.

A safer approach is to use virtual machines, for experimentation, and to keep different (and clashing) environments separate.

For more detail see the step by step instructions at Everyday Linux User.

These instructions are for installing VirtualBox on Linux Mint (17.2), and then creating a Linux Mint virtual machine

Installation and setup

I used mostly used the default settings. This may not be right for everyone

  1. Use a package manager to install VirtualBox.
    1. I used Synaptic, and selected virtualbox-5.0
  2. When I started VirtualBox and tried to create my first virtual machine (VM), it only let me create 32 bit VMs, even though my computer is fully 64 bit
    1. Fix this by going into the BIOS and enabling (Intel) Virtualisation
  3. If like me you have multiple drives (SSDs and/or HDDs) or multiple partitions, specify where VirtualBox stores the virtual drives (files):
    1. File -> Preferences -> General -> Default Machine Folder

Your first virtual machine

  1. Download a copy of the relevant linux Distro
  2. Start VirtualBox: Menu -> Administrate -> Oracle VM VirtualBox
  3. Click on “New”
    1. Name: E.g. “Linux Mint 1”
    2. Type: “Linux”
    3. Version: Linux 2.6/3.x (64 bit) or Other Linux (64 bit)
    4. Memory: Recommended size or higher (note: you can always change it later)
      1. During my first attemp I used the default of 256MB. When trying to boot Linux Mint off the install ‘disk’, the virtual machine ground down to a halt. Increasing this to 1GB fixed this)
      2. 512MB minimum, 1GB+ is probably better
  4. Hard drive: Create a virtual drive now
    1. Note: VirtualBox creates a file and pretends that it is a whole hard disk (i.e. a “virtual drive”)
    2. Type: leave as is (VDI)
    3. Size: Dynamically allocated
    4. Limit: leave as is or increase
    5. Create
  5. The machine has been created, but isn’t running yet. Click on the “New” button
    1. Click on the folder with green ‘arrow’ icon
    2. Select the previously download distro
    3. Click Start
      1. This will start the VM, which will boot off the downloaded distro
    4. Follow the instructions to install the new OS
    5. Suggestion: Use a different password from your main password. This is generally good security practice, but in this case it may stop you doing something on your actual Linux install whilst thinking you’re working on a VM. I realised this when I tried to stripdown all s/w on a VM to only that relevant to development work, and nearly removed some software of the host operating system
  6. If you get a large error message “Running in software rendering mode”:
    1. With VM stopped, go into settings -> Display -> Enable 3D accelleration

I suggest you clone this VM before you start experimenting, to save you having to reinstall the OS should you need another (clean) VM

Installing Python 2.10 on Linux Mint 17.2

Warning: You may have some packages on your machine which rely on Python, and which may no longer work after installing a different version of Python. Ideally you should use a virtual machine for this

From https://slobaexpert.wordpress.com/2015/07/09/upgrade-python-2-7-6-to-python-2-7-10-on-linux-mint-os/:

  1. Download Gzipped source tarball from the Python website
  2. Unzip the downloaded file
  3. Switch to new folder (e.g. /Downloads/Python-2.7.10)
  4. In a terminal window:
    1. apt-get install libc-dev
      1. If you miss this, you’ll get an error: error: C compiler cannot create executables
    2. ./configure
    3. sudo make install
  5. Run “python –version”
    1. You should now see 2.7.10

Flask and Angular on Heroku

I am working my way through this excellent tutorial, covering Python3, Flask, Angular, Heroku, SQLAlchemy, Alembic, requests, Beautiful Soup, NLTK, Redis and D3. Here are some extra notes

  • To stop me from blindly copying/pasting the code, I printed off the tutorial and worked from the paper version.
  • I had some problems installing Virtualenvwrapper (on Linux Mint 17.2), until I followed these instructions
  • I had some clashes with Anaconda
    • Virtualenvwrapper’s deactivate clashed with Anaconda’s deactivate. Prompted by these instructions I renamed ~/anaconda/bin/activate and ~/anaconda/bin/deactivate
    • “pip install psycopg2” resulted in:
      Error: “setuptools must be installed to install from a source distribution”
      After much experimentation I guessed that this might be due to Anaconda. I created a new virtual machine (without Anaconda) and re-started the tutorial. This fixed the psycopg2 problem

Part 1, set up Heroku

  • I used a free Heroku account. Between a dedicated server, a WebFaction account and a HotDrupal account I’m already paying enough for hosting
  • “heroku create wordcounts-pro” gave me an error “Name is already taken”. According to this Heroku page,  app names are in the global namespace, so I guess I’m not the first one to follow this tutorial. To work around this, I prepended the app name with my initials, i.e. “heroku create cdg-wordcounts-pro”, etc
  • So I can push the changes to heroku I set up public key access
  • Before running “git push stage/pro master”, make sure to check in the changes to git (git add, git commit)

Part 2, set up databases

  • To create the Postgres database:
    • sudo su — postgres
    • psql
      • # CREATE DATABASE wordcount_dev;
      • # CREATE USER ‘<your user name>’
      • # GRANT ALL PRIVILEGES ON wordcount_dev TO <your user name>;
  • After running “heroku run python manage.py db upgrade …” I got the error message:
    No such file or directory: ‘/app/migrations/versions’

    • Locally I had an empty directory <app folder>/migrations/versions. However, git ignores empty directories. This is why I could run “.. manage.py db upgrade” locally but not on heroku
    • Oops, I’d forgotten to run
      python manage.py db migrate
      Now it worked fine
    • If you make the same mistake, remember to propagate the changes to heroku and then re-run db migrate on heroku

Part 3, requests, Beautiful Soup and NLTK

  • At one stage I got a server error. To sort this I looked at the heroku log:
    heroku logs –app <heroku app name>
  • When I ran the nltk downloader I didn’t get the usual gui but a “tui” (text user interface). It was fairly simple to navigate, but I didn’t bother to specify the location of the tokenizers. Instead I used the default (~/nltk_data) and then moved nltk_data into my app folder
  • The links to Bootstrap and jQuery didn’t work, either because I mistyped them or because they are out of date. The Bootstrap and jQuery websites give you up-to-date CDN links, so use those instead

Part 4, Redis task queue

  • I used these instructions to install Redis on Linux Mint
  • Apart from the inevitable few typing mistakes, everything worked remarkably smoothly. Nothing else to add

Part 5, Adding in Angular

  • It all went well, until I added the getWordCount function to the controller. I’d put the function inside the main.js file, but outside of the controller. When poller got called, none of the dependencies were included, so it couldn’t find $http (first line of poller function)
    • The error was: $http not defined
    • Despite comparing my version with the author’s GitHub one, I couldn’t see the difference. In the end I used the author’s version (of main.js) instead of mine. That worked fine. It took another line by line comparison to find the problem
  • The word/frequency list is no longer sorted. jsonify loses the order

Part 6, Staging the changes, including Redis

  • So far I’ve been using a free account. When I tried to add on Redis, heroku tells me: Please verify your account to install this add-on plan (please enter a credit card)
    • If I understand it correctly, it is still free (but don’t take my word for it – and don’t come back to me if you end up getting charged)
    • I entered my credit card details for my Heroku again. Now I can add Redis
  • “heroku addons:add redistogo –app” gave a warning to say that “addons:add” has been deprecated.
    • I used “addons:create” instead

 

Python, CherryPy and Websockets – hosted

To set up websockets on a host can take a few extra steps. The host needs to know what to do with the websocket traffic.

It took a few attempts to get a Python/websockets sample up and running. The first websocket library I used (for Python/Bottle) worked very well on my local machine, but the host didn’t have the required libraries installed and didn’t give me enough access rights to fix this. I switched to a different framework and library, which worked much better.

  1. Create a new (WebFaction) application, wstest, custom websocket
    1. Save it, attach it to a domain/website, and make a note of the port
  2. ssh into the host
  3. Set up the requirements:
    1. pip install cherrypy
    2. pip install ws4py
    3. pip install argparse
  4. Switch to the application’s folder (cd webapps/wstest)
  5. Create index.html and site.py as created by Sylvain Hellegouarch, but with following changes:
    ….
    return index_page {d34bf16ac7b745ad0d2811187511ec8954163ba9b5dbe9639d7e21cc4b3adbdb} {‘username’: “User{d34bf16ac7b745ad0d2811187511ec8954163ba9b5dbe9639d7e21cc4b3adbdb}d” {d34bf16ac7b745ad0d2811187511ec8954163ba9b5dbe9639d7e21cc4b3adbdb} random.randint(50, 1000),
    ‘ws_addr’: ‘ws://<url>/ws’}
    ….
    cherrypy.config.update({
    ‘server.socket_host’: ‘127.0.0.1’,
    ‘server.socket_port’: <the application’s port – see above>
    })

    1. Important (this took me a while to figure out): the original websocket url (first/second line, above) in the sample code is “ws://localhost:9000/ws”. Having tried a bunch of different ports (80, 9000, the application’s port, etc), the correct solution is to leave the port off completely, so just use “ws://<url>/ws” instead
  6. Start CherryPy: python2.7 site.py
    1. Since you’re running this still attached to your terminal session, when you close the (ssh) terminal the process will stop as well.
  7. Now open the url in two separate browsers. You should be able to enter messages in either browser and see it appearing in both

Electronics and Robotics – Teaching

Our children’s school asked me to teach the 14 – 16 year old pupils some electronics. To make it more interesting I threw in some robotics.

During an intense teaching period I spent morning sessions teaching the theory of electronics and programming, and afternoon sessions soldering the PIC-based control boards.

The challenge was to create a programmable line following robot, from electronic components and Fischertechnik parts (see the picture above), and to race it.

The groups completed their control boards, and each created a different style robot. We had an exciting race, with last minute improvements to the control logic, and all robots finished the course.

Computer building – teaching

Our children’s school (a Waldorf/Steiner school) believes that ideally children should build a computer before they use one. So I offered to work with the 15-16 year old pupils on this.

My first computer was a 1802 CPU based single board computer, which I had to solder together. It had a hexadecimal keyboard (0-9, A-F), a double-digit display (not a monitor, just two digits similar to the picture above), 256 bytes of memory (yes, bytes, not kilo bytes, not mega bytes). It had a very simple instruction set, which gave me an excellent insight into the workings of the CPU.

That is the experience I was aiming for: getting the pupils to get a real feel for how computers work.

I designed a simple Z80-based computer, and created a curriculum consisting of building computers in small groups, programming, and an introduction to the core elements of a CPU.

With a little help all groups managed to complete their computer and run some simple programs.

Managing the repositories in Linux

Much of Linux’s software is stored in repositories. Your Linux installation has a list of repositories, with the most stable recent packages for your distro and possibly some others.

To view, add or delete the repos

In Synaptic, go to Settings -> Repositories

  1. Official Repositories
  2. Upgrade to the latest version – PPAs – Personal Package Archives
    • The official repo contains a fully tested set of packages – both tested individually and together. So you can be confident that all packages work together, that they don’t clash. Many of the packages in the official repo are one or more versions behind the latest version. PPAs contain other versions, and may include the latest version. Installing a package from a PPA increases the risk of packages clashing
    • For more information on PPAs see this article on How To Geek
  • Additional Repositories – for software that isn’t part of the official repo. Often the software’s website will give you step-by-step instructions on how to install it, including the command line for adding the repo. Or you can use Synaptic – Click on “Add a new repository” and enter the name and the URL