Category Archives: DevOps

Namepy – on the shoulders of giants

Whilst my core skill/tool is Python, I’m always learning new things, either inside or outside the Python ecosystem. I recently had the pleasure of working with Angular and Python/Flask. Here is a playful application based on these, plus Highcharts.

Going through “Python for Data Analysis”, some of the examples use a database of frequency of (US) baby names since 1880. I thought I’d combine this with a bit of Scrabble™.

In the Python world it’s common to add “py” to a word when making up names, so I’m calling this project “namepy”.

Since I’ll be using various frameworks and libraries, all created by others, I’ve subtitled this “On the shoulders of giants”.

Taking small steps often results in faster progress, so that’s what I’m be doing here.

Technical set up

The source code is at https://github.com/CoachCoen/namepy, with one branch per step.

Many production sites Content Delivery Networks for serving Javascript frameworks and libraries, usually minified, which helps to take the load of the server and may speed up first page load. To keep things simple and stable over time, I’m using full-sized, downloaded, copies.

I’m using WebFaction (affiliate link) as the host, since they make it easy to create Flask, Django and similar projects. And, as a popular host for developers, you’ll find lots of helpful documentation for developers online.

Getting started

Create a project folder

mkdir namepy
cd namepy

At the start of each of the steps

cd (my folder for personal projects)
cd namepy
git clone https://github.com/CoachCoen/namepy.git -b step1 step1

Note: “-b step1” specifies the name of the branch to clone. The second “step1” is the target folder, i.e. namepy/step1.

Next

Continue to Step 1 – Angular “Hello World”

VirtualBox – moving virtual disks

Having installed VirtualBox I created a few virtual machines, with 20GB virtual disks each. After all, I’ve got a 1TB hard disk in my computer.

When Linux started complaining about running out of disk space I realised that my main drive is a 256GB Solid State Drive, and that is where VirtualBox was storing the virtual disks. Time to move the virtual disks. Please note that it is a bit of a manual process and, whilst it worked fine for me, if you’ve already invested a lot of time setting up your virtual machines, I suggest that you test this out by moving one or two VMs first.

  1. Stop all VMs, stop VirtualBox
  2. Find a folder called “VirtualBox VMs” (probably in your home folder), and move it to the new location
    1. Or move just a few files first, to make sure it works
  3. Restart VirtualBox
    1. In File -> Preferences -> General, point the Default Machine Folder to the new location
    2. All your existing VMs are now broken. Delete them
    3. One at a time: Machine -> Add
      1. Select the .vbox file from the new folder

Virtual machines using VirtualBox

Tapping into all the wonderful tools and languages that make our lives as developers so interesting can throw up some fun (?) challenges.

You start installing the latest linter or library, and before you know it you’re in dependency hell. Required libraries require even more libraries, with difference versions numbers and clashing with your existing setup.  If you’re really unlucky you break some essential software along the way.

A safer approach is to use virtual machines, for experimentation, and to keep different (and clashing) environments separate.

For more detail see the step by step instructions at Everyday Linux User.

These instructions are for installing VirtualBox on Linux Mint (17.2), and then creating a Linux Mint virtual machine

Installation and setup

I used mostly used the default settings. This may not be right for everyone

  1. Use a package manager to install VirtualBox.
    1. I used Synaptic, and selected virtualbox-5.0
  2. When I started VirtualBox and tried to create my first virtual machine (VM), it only let me create 32 bit VMs, even though my computer is fully 64 bit
    1. Fix this by going into the BIOS and enabling (Intel) Virtualisation
  3. If like me you have multiple drives (SSDs and/or HDDs) or multiple partitions, specify where VirtualBox stores the virtual drives (files):
    1. File -> Preferences -> General -> Default Machine Folder

Your first virtual machine

  1. Download a copy of the relevant linux Distro
  2. Start VirtualBox: Menu -> Administrate -> Oracle VM VirtualBox
  3. Click on “New”
    1. Name: E.g. “Linux Mint 1”
    2. Type: “Linux”
    3. Version: Linux 2.6/3.x (64 bit) or Other Linux (64 bit)
    4. Memory: Recommended size or higher (note: you can always change it later)
      1. During my first attemp I used the default of 256MB. When trying to boot Linux Mint off the install ‘disk’, the virtual machine ground down to a halt. Increasing this to 1GB fixed this)
      2. 512MB minimum, 1GB+ is probably better
  4. Hard drive: Create a virtual drive now
    1. Note: VirtualBox creates a file and pretends that it is a whole hard disk (i.e. a “virtual drive”)
    2. Type: leave as is (VDI)
    3. Size: Dynamically allocated
    4. Limit: leave as is or increase
    5. Create
  5. The machine has been created, but isn’t running yet. Click on the “New” button
    1. Click on the folder with green ‘arrow’ icon
    2. Select the previously download distro
    3. Click Start
      1. This will start the VM, which will boot off the downloaded distro
    4. Follow the instructions to install the new OS
    5. Suggestion: Use a different password from your main password. This is generally good security practice, but in this case it may stop you doing something on your actual Linux install whilst thinking you’re working on a VM. I realised this when I tried to stripdown all s/w on a VM to only that relevant to development work, and nearly removed some software of the host operating system
  6. If you get a large error message “Running in software rendering mode”:
    1. With VM stopped, go into settings -> Display -> Enable 3D accelleration

I suggest you clone this VM before you start experimenting, to save you having to reinstall the OS should you need another (clean) VM

Installing Python 2.10 on Linux Mint 17.2

Warning: You may have some packages on your machine which rely on Python, and which may no longer work after installing a different version of Python. Ideally you should use a virtual machine for this

From https://slobaexpert.wordpress.com/2015/07/09/upgrade-python-2-7-6-to-python-2-7-10-on-linux-mint-os/:

  1. Download Gzipped source tarball from the Python website
  2. Unzip the downloaded file
  3. Switch to new folder (e.g. /Downloads/Python-2.7.10)
  4. In a terminal window:
    1. apt-get install libc-dev
      1. If you miss this, you’ll get an error: error: C compiler cannot create executables
    2. ./configure
    3. sudo make install
  5. Run “python –version”
    1. You should now see 2.7.10

Managing the repositories in Linux

Much of Linux’s software is stored in repositories. Your Linux installation has a list of repositories, with the most stable recent packages for your distro and possibly some others.

To view, add or delete the repos

In Synaptic, go to Settings -> Repositories

  1. Official Repositories
  2. Upgrade to the latest version – PPAs – Personal Package Archives
    • The official repo contains a fully tested set of packages – both tested individually and together. So you can be confident that all packages work together, that they don’t clash. Many of the packages in the official repo are one or more versions behind the latest version. PPAs contain other versions, and may include the latest version. Installing a package from a PPA increases the risk of packages clashing
    • For more information on PPAs see this article on How To Geek
  • Additional Repositories – for software that isn’t part of the official repo. Often the software’s website will give you step-by-step instructions on how to install it, including the command line for adding the repo. Or you can use Synaptic – Click on “Add a new repository” and enter the name and the URL

Linux package management – some common operations

For my own benefit (and maybe others), here are some of the common package management operations and how to do it (in Linux Mint and other Linux distros)

  • Update all install software:
    • apt-get update # refresh repos
    • apt-get upgrade # upgrade all software
    • (note: do both of the above, in that order)
    • Note: In Linux Mint you’ll see a shield with an exclamation mark in the bottom right hand corner when you need to update your software. Click on it and follow the instructions
  • Install a new package, if you know the name: apt-get install <package name>
  • Search for a package: use Synaptic (Menu -> Administration -> Synaptic), enter (partial) name or other keyword (e.g. “game”) in the search box at the top of the screen
  • View installed packages: In Synaptic, bottom left hand corner, click on “Status”, then (top left hand corner) click on “Installed”
  • Get more information about an installed package
    • Use Synaptic, find the package, click on the Properties button (top of the screen)
    • Common: Installed version, status (e.g. installed)
    • Dependencies – what other packages this package needs to work
    • Installed files – what files is this package made up of
    • Versions – what versions are available
    • Description

Linux Mint – family tree

Having used Fedora, CentOS and Debian (and a little of Raspbian and Unix) a couple of years ago I swapped over to Linux Mint. Since it is based on the popular Debian and Ubuntu it has an excellent eco-system –  packages, community and support

See Wikipedia for some more information about Linux Mint. This family tree shows you the relationship between many of the main Linux distros.

DevOps – intro

If I understand it correctly (it looks like the jury is still out), a DevOp is someone who helps developers be more productive by giving them the tools they need.

As a freelance developer, from time to time I do some DevOp work, such as installing a library or some new software. Things can get quite complicated, but understanding how it all fits together, and knowing some tools and commands, can really speed things up.

Since I mainly use Linux Mint and Python, many of the DevOp articles on this website are about Linux Mint and Python. A lot of it will be relevant to other Linux distributions (distros) and languages.