Python is a very popular tool for data extraction, clean up, analysis and visualisation. I’ve recently done some work in this area, and would love to do some more. I particularly enjoy using my maths background and creating pretty, clear and helpful visualisations
- Short client project, analysing sensor data. I took readings from two accelerometers and rotated the readings to get the relative movement between them. Using NumPy, Pandas and MatplotLib, I created a number of different charts, looking for a correlation between the equipment’s setting and the movement. Unfortunately the sensors aren’t sensitive enough to return usable information. Whilst not the outcome they were hoping for, the client told me “You’ve been really helpful and I’ve learned a lot”
- At PyCon UK (Cardiff, September 2018) I attended 14 data analysis sessions. It was fascinating to see the range of tools and applications in Python data analytics. At a Bristol PyData MeetUp I summarised the sessions in a 5 minute lightening talk. This made me pay extra attention and keep useful notes during the conference
- Short client project, researching best way to import a large data set, followed by implementation. The client regularly accesses large datasets using a folder hierarchy\to structure that data. They were looking to replace this with a professional database, i.e. PostgreSQL. I analysed their requirements, researched the different storage methods in PostgreSQL, reported my findings and created an import script.
I recently completed a small project for Zenstores. They simplify the shipping process for ecommerce sites. Their online service lets online businesses use multiple shipping companies for deliveries.
Each shipping companies offers a different own API, for booking shipments, etc. My client uses a separate microservice for each shipping company. These microservices listen to requests from the main system and translate them to the shipping company’s standard.
My client asked me to use Django Rest Framework to create a microservice which supports a new shipping company. DRF is a popular and powerful library to create RESTful APIs using Django.
The supplier provided me with a sandbox API and extensive documentation. The documentation was somewhat incomplete and out of date. Fortunately their support contact was very helpful all along.
I used Test Driven Design for complex functions where I understood the functionality well. For the rest I used a more experimental approach and added unit tests afterwards. Testing coverage was over 90%.
The client has integrated the microservice within their system and the first test shipments have gone through.
A couple of internationally renowned artists asked me for some help with their largest installation to date. As part of Hull’s City of Culture, Davy and Kristin McGuire created a large cardboard city and brought it to life with video projections.
They needed nearly 20 video players, so I created a bootable linux image for the Raspberry Pi which automatically plays a video from a standard location. I copied this to 20 memory cards, and tested them all.
The installation looked amazing and was a great success.
My client, a start up with a lot of experience in their field, had identified an important gap in the market. Large sums of money were being invested, with very long payback periods, without access to effective performance tracking tools.
They designed a tool to cover the gap and asked me to create a demonstration system in preparation for generating interest and raising capital.
The resulting system imports the base data and generates monthly cost and revenue forecasts, taking into account seasonal variations, tax allowances and more.
The main management screen gives quick access to some key performance indicators.
Constraints can be defined, and potential investments can be checked against them.
Actual results can be compared against the projections.
Different heat maps show absolute or relative performance by state or county.
This was an eight month intensive project, resulting in a demo site which generated a lot of interest in the industry and allowed the client to achieve their first round of funding.
In 1999 I took over the running of an online community for business and personal coaches. Initially a simple email discussion list run on the Mailman open source software, once we hit 500+ members I decided to upgrade the systems.
At the time (about 2003) there was very little ready made community software available. So I created my own.
Mailman is written in Python. To migrate the data it seemed easiest to stick with Python, and I selected Zope as the web framework.
I extended the MailBoxer email distribution component and added digests (HTML, text, attachments). And I created the Zope website with a members-only message archive, online payment system, automatic subscription management, profiles and a classified adverts section.
The new system has been running for about 10 years now, seen many 1000s of members and 50,000+ messages.
Our children’s school asked me to teach the 14 – 16 year old pupils some electronics. To make it more interesting I threw in some robotics.
During an intense teaching period I spent morning sessions teaching the theory of electronics and programming, and afternoon sessions soldering the PIC-based control boards.
The challenge was to create a programmable line following robot, from electronic components and Fischertechnik parts (see the picture above), and to race it.
The groups completed their control boards, and each created a different style robot. We had an exciting race, with last minute improvements to the control logic, and all robots finished the course.
Our children’s school (a Waldorf/Steiner school) believes that ideally children should build a computer before they use one. So I offered to work with the 15-16 year old pupils on this.
My first computer was a 1802 CPU based single board computer, which I had to solder together. It had a hexadecimal keyboard (0-9, A-F), a double-digit display (not a monitor, just two digits similar to the picture above), 256 bytes of memory (yes, bytes, not kilo bytes, not mega bytes). It had a very simple instruction set, which gave me an excellent insight into the workings of the CPU.
That is the experience I was aiming for: getting the pupils to get a real feel for how computers work.
I designed a simple Z80-based computer, and created a curriculum consisting of building computers in small groups, programming, and an introduction to the core elements of a CPU.
With a little help all groups managed to complete their computer and run some simple programs.
The client, a highly respected training company, was using an in-house developed system to manage the courses, bookings, delegates, and sales and marketing processes. When the business outgrew the system they asked me to evaluate an online solution (Administrate).
After presenting my in-depth analysis, the Board of Directors decided to proceed, and hired me as the Technical Project Manager.
I created Python scripts to clean the old database and restructure the data for the new system. Under my guidance, the internal developer integrated the client’s website with Administrate. Administrate staff and myself provided the training.
Overall the transition went very smoothly. The only snag was with some of the more advanced marketing lists (used for email marketing campaigns). I had already expressed my concerns to Administrate that the existing system wouldn’t be able to handle these queries, but was reassured by them that it would. Fortunately, when my concerns were confirmed, Administrate agreed to make the necessary changes. They were very good about it and did this without any protest, although this may have been different without my careful preparations. Administrate spent a good few developer months creating a completely new reporting system, in consultation with me.
Meticulous preparations meant a smooth transition and saved the client months of bespoke work.