July 15, 2018

Iain R. Learmonth


Free Software Efforts (2018W28)

Last week was my second week working full time in the Metrics team at Tor Project.

Here’s what I’ve been up to:

Tor Project

Relay Search Fixes

A few fixes for Relay Search. These were all small fixes but hopefully should also be quite high impact for usability.

Onionoo Enhancements

The improvements to Onionoo’s reverse DNS resolver have now been finalised and merged. The plan is that this will be deployed next week.

I have added support to Onionoo to filter relays by operating system. This was the oldest open Onionoo ticket until it was fixed.

I have added support to Onionoo to search for relays that do not have known countries or autonomous system numbers.

New Bridge Authority

The old bridge authority has been retired, and a new bridge authority set up: Serge. I produced a tiny patch for CollecTor to allow it to recognise this new bridge authority and reviewed the pre-release tarball for CollecTor 1.7.0. This is now released and deployed.


I’ve been improving the JavaDoc coverage, and have also increased the maximum length of tweets to match Twitter’s new 280 charachter limit (some autonomous systems have long names).


I reviewed patches for metrics-lib, CollecTor and Onionoo.


I also attended the Metrics team meeting on Thursday.


Since my last weekly update, I have uploaded new Debian packages for:


I reviewed and merged improvements to the installation instructions for hellfire. I also reviewed and merged a rate limiter for hellfire, and made it configurable via a command-line option.

July 15, 2018 05:00 PM

July 09, 2018

Iain R. Learmonth



Last week when I found myself with my Internet connection down, I tried to use my EE LTE connection to work instead. This plan fell apart quite quickly when I discovered that * was blocked as 18+ content.

I found using the Open Rights Group’s tool,, that in fact a large number of UK ISPs have these wildcard blocks in place. This tool also provides a means to report misclassification and so I submitted requests to unblock the following domains:


It is important to note that none of these domains host copies of Tor clients, Tor Browser or anything that would enable to use Tor. They only host data about the public Tor network.

Sky Broadband were the first to respond to say that they had passed on the queries to Symantec, but I’ve not heard anything back from them yet.

Then EE responded today and it seems they also use Symantec to provide their block lists:

I have checked the classification of with Symantec who classify websites on EE’s behalf and they say that the site is correctly classed as a ‘Technology and Anonymizer’ website which means it can only be seen by adult EE users who have turned their parental controls off.

I’m not sure about the classification here, does this include all “Technology” websites?! They go on to say:

Symantec says the website offers Tech solutions and mainly the download of Tor browser. From their website: “Tor is an effective censorship circumvention tool, allowing its users to reach otherwise blocked destinations or content.” There will be no change at this point.

I hope this helps clarify the situation and that you understand that EE does not decide what sites are behind parental controls but follows guidelines drawn up by the BBFC.

This is nonsense. These sites provide recent and historical data relating to the public Tor network and do not provide access to circumvention software.

Symantec has not checked the domains that I have reported. While there may be a case for blocking the homepage, blocking all subdomains is a case of overblocking and general laziness.

I was interested to see the guidelines from the BBFC, so I looked up their website. I noticed that I could participate in a survey to influence the guidelines for the future, so I thought I’d have a go at that:

So that’s not a great start. I click next and then:

Thanks for answering those questions, unfortunately based on the answers provided you do not qualify for this survey. We appreciate the time taken to provide us with your opinions.

I guess that a secure connection wasn’t required after all.

July 09, 2018 05:00 PM

July 08, 2018

Iain R. Learmonth


Summer School on Internet Path Transparency Measurements

This post was originally published at the MAMI Project blog.

On June the 11th the Electronics Research Group hosted the MAMI Summer School on Internet Path Transparency Measurements in Aberdeen, Scotland. This consisted of a few hands-on workshops, with participation both on-site and remote via video conference.

The summer school started with Korian and Justin demonstrating Tracebox through a variety of topologies. The participants then worked on their own trying to uncover middleboxes and hidden topologies using a variety of tools, including tracebox and paris-traceroute.

To follow, I presented the history and development of PATHspider. I also delivered an interactive Scapy tutorial followed by teaching students how to use Scapy to create PATHspider plugins using the Evil Bit as an example. The participants ran plugins against a list of real targets and produced results.

To finish off, Brian Trammell delivered a presentation on the PTO (Path Transparency Observatory), and helped the participants upload to it the results they collected in the previous session. The participants then learned how to query the PTO in order to examine the data they just uploaded. In an interesting twist of events, the measurements found ECN manipulation by a middlebox in the eduroam network in Aberdeen - mission accomplished!

If you missed the summer school, all slides and materials can be found here.

July 08, 2018 02:00 PM

July 07, 2018

Iain R. Learmonth


Free Software Efforts (2018W27)

Last week was my first week working full time in the Metrics team at Tor Project.

On Tuesday, my Internet connection was interrupted for the entire workday which did make it a little more difficult to work. I was able to use my 4G connection but, unfortunately, access to the Tor bug tracker and all of the Metrics websites were blocked requiring me to verify I was over the age of 18 to access. Tor Browser was quite happy to go around the censorship so it was not impossible to work.

Here’s what I’ve been up to:

Tor Project

Working full time for Tor Project means that I’ve mostly done Tor Project things.

Relay Search Integration

In the roadmap for the Metrics team we had said that we wanted to fully integrate Relay Search into Tor Metrics. This week we have completed this integration, although there will still be some documentation to update next week (e.g. contributing documentation on the wiki). As part of the integration, we have now changed the license for Relay Search to 3-clause BSD from the previous MIT license to allow for alignment with the rest of the Tor Metrics codebases.

Later in the week I have triaged some relay search tickets and written patches for some of the easier to fix issues.

Onionoo’s Reverse DNS Resolver

For my first substantial contribution to the Onionoo codebase I accepted ticket #18342. My patch for this is nearing completion and I will make a seperate post about this once it has been merged. I’ve even learnt new things about DNS that I did not know before, and even surprised me, so this has been a great first ticket to work on.

Working on this ticket means I now also have a development environment set up to work on Onionoo, and I’m starting to get to grips with the codebase. In the past I’ve only used data provided by Onionoo via its HTTP API (e.g. for Relay Search or metrics-bot).

Measuring Latency

At the Tor meeting in Rome a few people asked about the latency for connections over Tor. At the time, Tor Metrics only had details about throughput but it looked like OnionPerf could collect the information we needed. To look into this, I filed #25774 and this week I have been reviewing karsten’s work on this ticket. No extensions to OnionPerf were required and so this went quickly and the graphs are already live.

While working on this ticket, I noticed that one of Tor Metrics’ OnionPerf instances (op-us) was failing on 100% of connections, so I asked our services admin to restart it. It looks to now be running again and we will look into better monitoring in the future.


I also attended the Metrics team meeting on Thursday and the Pluggable Transports meeting on Friday. I had attended the Pluggable Transports meeting purely by chance, but it was quite interesting so I followed along.


I reviewed and sponsored packaging for lektor (ITP #892901). This is currently in the NEW queue.

Community and Events

I have tidied up the configuration for the 57North Hacklab Planet. If you’ve noticed that your blog is no longer present, it’s because the feed was 404. Some http:// URLs have been updated to https:// URLs where I saw that redirections existed.

I attended the following events last week:

July 07, 2018 12:00 PM

June 15, 2018

Edward Watson




Taken during a lunchtime walk; Garthdee along the river path, sometime over May 2018.

On yet another Dee walk, I was forced to skip over a cheeky alloy snek as he clackered across my arid path. This species's young has developed a bad name in industry as it is often found inside jammed shopping trolly wheels, being attracted to the maternal anti-theft magnets installed therein.    

June 15, 2018 02:21 PM

May 06, 2018

Edward Watson


Solar PV, Aberdeen and me - Update 3

I am currenly holidaying in sunny Gran Canaria and whilst dozing by the pool, I caught sight of a bank of solar panels. "Ah" I thought, I have those.

Its also been a while that I last blogged, so I thought I would kill two birds with one diamond laser and post an update. See graph.


It has been up and down with the generation rate but it has surpassed the conservative expectation that the installer gave me. Quite pleasing. The generation estimate was made against the avg Aberdeen sunlight hours, direction of panels, angles etc and the performance tracks  the avergate sunlight hours/day nicely - below, yellow line.


The sunlight hours highs are May followed by August, which appears odd at first (July being the hottest month) until you realise heat and sunlight are not the same thing. It does make me want to investigate all this talk on micro heat-powered Sterling engines I hear so much about. These are dinky blacker-than-black heat traps that expand and contract gasses to drive turbines that generate UnLiMeD POWER!!@%@45£!

At a 60% to 70% efficancy rating, Sterling engines kick solarpanels and their 20% max efficancy off the ballpark. They are more complicated I am given to understand, however the clever dicks are working on the problem and hope to have something investors can overspeculate on soon [1]. When done, we can call China and ask actual ballparks of them - hurrah.



May 06, 2018 02:03 PM

April 01, 2018

Robert McWilliam

Network effect

As Warren Ellis recently mentioned: once upon a time if your kink was one in a million you needed to be in a fairly big city to find a single digit number of people who might be on the same page.

Finding them was a non-trivial exercise.

Those days are past (but recent enough that I can remember them).

The internet now connects 3.something billion people. A one in a million kink lumps you in with a few thousand people. And they're only a Google search away.

You need to be a couple orders of magnitude further off the beaten track if you want to stand out these days.

April 01, 2018 11:00 PM

February 26, 2018

Iain R. Learmonth


A Virtual Machine Host

It’s very easy to set up libvirt with QEMU/KVM on a Linux box and have it work for a while. It’s more difficult to have a system set up that’s going to be somewhat robust and maintainable over a longer period of time.

I am using a dedicated server from Hetzner. I have not personally had any serious problems with Hetzner, and price is the main reason that I’ve chosen them over other providers.

Operating System Install

Hetzner provides a “rescue” system that you can boot your dedicated server into and this allows for a customisable installation of Debian. My server has two HDDs and by default the install script will set up software RAID for you. I’d like to use ZFS and give ZFS as much direct control over the disks as possible so the default installation wouldn’t work for me.

From the web interface, set your server to boot into the “rescue” system for Linux 64-bit and select your SSH key. You will need to reboot the server as a seperate step.

$ ssh root@YOURSERVER


  Welcome to the Hetzner Rescue System.

  This Rescue System is based on Debian 8.0 (jessie) with a newer
  kernel. You can install software as in a normal system.

  To install a new operating system from one of our prebuilt
  images, run 'installimage' and follow the instructions.

  More information at


Hardware data:

   CPU1: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz (Cores 8)
   Memory:  32044 MB
   Disk /dev/sda: 2000 GB (=> 1863 GiB) 
   Disk /dev/sdb: 2000 GB (=> 1863 GiB) 
   Total capacity 3726 GiB with 2 Disks

Network data:
   eth0  LINK: yes
         MAC:  xx:xx:xx:xx:xx:xx
         IPv6: xxxx:xxxx:xxxx:xxxx::2/64
         RealTek RTL-8169 Gigabit Ethernet driver

root@rescue ~ # 

You’ll see here information about your disks. In general this blog post is going to assume your server is pretty much identical to this one if you are following this.

Run installimage to start the installation script and you’ll see the following menu:

Select Debian (the first choice) from the menu and you’ll then be given a list of available Debian images. Choose Debian-93-stretch-64-minimal from this menu. You’ll then be shown a notice to tell you that an editor will now be opened to allow you to edit the parameters for the installation. Just press Enter to start the editor. (In case you were wondering, the editor is mcedit.)

Look for SWRAID 1 and change this to SWRAID 0 to disable the use of software RAID. You can also set the hostname for the installed system in the script by changing the HOSTNAME line. Finally set up the partitions for the new system. The installation script doesn’t know about ZFS so this will happen later, for now change the PART lines to look like:

PART /boot ext3 512M
PART / ext4 55G

These partitions will be created on /dev/sda and /dev/sdb will be left untouched by the installer. Press to exit the editor, and save your changes. The installation will then run:

                Hetzner Online GmbH - installimage

  Your server will be installed now, this will take some minutes
             You can abort at any time with CTRL+C ...

         :  Reading configuration                           done 
         :  Loading image file variables                    done 
         :  Loading debian specific functions               done 
   1/15  :  Deleting partitions                             done 
   2/15  :  Test partition size                             done 
   3/15  :  Creating partitions and /etc/fstab              done 
   4/15  :  Formatting partitions
         :    formatting /dev/sda1 with ext3                done 
         :    formatting /dev/sda2 with ext4                done 
   5/15  :  Mounting partitions                             done 
   6/15  :  Sync time via ntp                               done 
         :  Importing public key for image validation       done 
   7/15  :  Validating image before starting extraction     done 
   8/15  :  Extracting image (local)                        done 
   9/15  :  Setting up network config                       done 
  10/15  :  Executing additional commands
         :    Setting hostname                              done 
         :    Generating new SSH keys                       done 
         :    Generating ramdisk                            done 
         :    Generating ntp config                         done 
  11/15  :  Setting up miscellaneous files                  done 
  12/15  :  Configuring authentication
         :    Fetching SSH keys                             done 
         :    Disabling root password                       done 
         :    Disabling SSH root login without password     done 
         :    Copying SSH keys                              done 
  13/15  :  Installing bootloader grub                      done 
  14/15  :  Running some debian specific functions          done 
  15/15  :  Clearing log files                              done 

   You can now reboot and log in to your new system with
  the same password as you logged in to the rescue system.

root@rescue ~ # 

Now that you have installed the operating system, reboot. Your new system will have generated its own SSH host keys, so you’ll need to remove the old host keys from the rescue system from your known_hosts file:

ssh-keygen -R

Once logged in to the new system, you’ll need to create partitions on sda and sdb for ZFS to make use of. Use fdisk to create a third partition on sda to use all the remaining space and then again to create a partition on sdb using the entire disk. As sda is currently in use by the system you’ll see the following message when you write the changes to disk:

The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy

The kernel still uses the old table. The new table will be used at the next
reboot or after you run partprobe(8) or kpartx(8).

This is fine. Once all the changes are made, reboot.

ZFS Install

Now it’s time to install ZFS. ZFS is not in Debian main, but is available in Debian contrib. In the Hetzner Debian images, the contrib and non-free archive areas are enabled by default. A freshly installed Hetzner system will not have any package lists cached, so an update is needed first.

apt update
apt install linux-headers-$(uname -r) zfs-dkms

You’ll be shown a warning about the license conditions. They should not be a problem here. The ZFS kernel module and userspace services are not loaded unless there is a zpool to use them, so when they are first installed they won’t be loaded. You can load them with:

modprobe zfs
systemctl restart zfs-import-cache
systemctl restart zfs-import-scan
systemctl restart zfs-mount
systemctl restart zfs-share

You can then create a zpool. (You need to do this before you reboot or else the zfs module will again not be loaded!)

zpool create tank mirror sda3 sdb1
zpool status

Some space on sdb1 is wasted here, as sda3 is smaller than sdb1 but this is not something that particularly bothers me.

Open vSwitch Installation

Next up is Open vSwitch. I’d like to have some flexibility in how the networking for the virtual machines is configured and Open vSwitch is a nice way to provide that. Installation is simple:

apt install openvswitch-switch
ovs-vsctl show

libvirt and QEMU/KVM Installation

Next up is QEMU/KVM. Again, a simple installation:

apt install libvirt-daemon-system qemu-kvm
virsh nodeinfo

Next Steps

Now that all the software is installed, next up will be configuring the components to work together and turn it into a nice flexible and robust virtualisation system. My next steps will be:

  • Set up some basic networking with NAT
  • Create the first VM
  • Use ZFS snapshots and replication to backup the VMs automatically

February 26, 2018 06:00 PM

January 20, 2018

Tom Jones


FreeBSD on an Intel x5-z8350 tv box

There are a load of sort of generic tv boxes on ebay with an Intel x5-z8350 processor.

The x5 SOC (formerly Cherry Tree, formerly Cherry View) is the same family as the SOC in my beloved GPD Pocket. I was really having trouble with the i2c hardware in the GPD Pocket and wanted something I could take apart and poke with an oscilloscope. I looked first at the UP Board, an x5-z8350 in a raspberry pi form factor, but not only was it much more expensive than this tv box, but it has a CPLD between the SOC io and the pin header.

Installing FreeBSD

First I needed to get the board to boot from USB, the listing I bought came with both android and windows 10 (I guess that is what dual os means). In both android and windows 10 there was a handy reboot to other os application.

From installing on the GPD Pocket I suspected that the bios boot menu key would be F7 so I used that. Windows 10 also includes a handy reboot to uefi config option which makes it easy to get into a bios menu. I used it to disable quiet boot and set the boot delay to a more sensible number.

With those changes I rebooted and got a familiar AMI bios boot screen hit F7 and choose your usb stick from the menu. The FreeBSD loader menu came up and continued into a boot from the usb stick, but it hung probing ppc0.

I found a solution on the freebsd forum post about the upboard which suggested running:

OK unset

at the loader prompt. With that you I could boot and do an install.

Before you reboot make sure to make that change permanent, by removing this line in /boot/device.hints

..."0x100""isa"        # comment this line out

now reboot.


I setup the the drm-next-kmod driver, but the machine froze during boot. Next I tried using a frame buffer driver, which required the collowing config in /usr/local/etc/X11/xorg.conf.d/driver-scfb.conf :

Section "Device"
    Identifier "Generic FB"
    Driver "scfb"

 Section "Device"
     Identifier    "Card0"
     Driver        "scfb"


x5 ports

The box has:

  • Blue/Red LED
  • External Power button
  • External(ish) reset button
    • Pressing the reset button caused an instant power cycle.
  • 4 usb ports
    • 1 USB 3
    • 2 external USB 2
    • 1 internal USB 2
  • sd card reader
    • but it doesn't seem to be hotpluggable
  • ethernet
  • hdmi

The x5 box also has bluetooth and wifi, but neither currently have FreeBSD drivers.

x5 top without heatsink

Internally there are a whole bunch of unpopulated things that might be interesting.

x5 top top left

On the top left there is an unpopulated 2.54mm pin header slot next to the led, silkscreen on the board has 1 and a 7 on either end. Probing around with a multimeter suggested that P7 was ground.

I spent quite a while poking the board with a multimeter and osclloscope to see if any gpio or buses were exposed on the headers or the board. I did find that if you connect pin 1 to gnd (or pin 7) the red led comes on and the board goes off.

I did not find any useful or even really interesting signals.

x5 top bottom right

On the bottom right there is an unpopulate 15 pin header, all but two of these were connect to ground.


Some more gory insides:

x5 board with heatsink x5 bottom screw x5 bottom x5 case x5 ports naked x5 top without heatsink top right x5 without heatsink bottom right x5 without heatsink rtc

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r328126: Thu Jan 18 15:25:44 UTC 2018 amd64
FreeBSD clang version 6.0.0 (branches/release_60 321788) (based on LLVM 6.0.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(efifb): resolution 1920x1080
CPU: Intel(R) Atom(TM) x5-Z8350  CPU @ 1.44GHz (1440.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406c4  Family=0x6  Model=0x4c  Stepping=4
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 1946144768 (1855 MB)
Event timer "LAPIC" quality 600
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-114 on motherboard
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC" frequency 1440001458 Hz quality 1000
random: entropy device external interface
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff80ff8620, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
Firmware Error (ACPI): Failure creating [BDLI], AE_ALREADY_EXISTS (20180105/dswload-498)
ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20180105/psobject-371)
ACPI Error: AE_ALREADY_EXISTS, (SSDT: DptfTab) while loading table (20180105/tbxfload-355)
ACPI Error: 1 table load failures, 8 successful (20180105/tbxfload-378)
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0x90000000-0x90ffffff,0x80000000-0x8fffffff at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Braswell USB 3.0 controller> mem 0x91700000-0x9170ffff at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <serial bus, USB> at device 22.0 (no driver attached)
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0x91604000-0x91604fff,0x91600000-0x91603fff at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: turning off MSI enable bit.
re0: Chip rev. 0x4c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 84:39:be:65:0d:60
re0: netmap queues/slots: TX 1/256, RX 1/256
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
sdhci_acpi0: <Intel Bay Trail/Braswell eMMC 4.5/4.5.1 Controller> iomem 0x9173c000-0x9173cfff irq 45 on acpi0
mmc0: <MMC/SD bus> on sdhci_acpi0
sdhci_acpi1: <Intel Bay Trail/Braswell SDXC Controller> iomem 0x91738000-0x91738fff irq 47 on acpi0
mmc1: <MMC/SD bus> on sdhci_acpi1
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
mmcsd0: 31GB <MMCHC NCard  4.5 SN 6E7E9160 MFG 06/2017 by 136 0x0003> at mmc0 200.0MHz/8bit/8192-block
mmcsd0boot0: 4MB partion 1 at mmcsd0
mmcsd0boot1: 4MB partion 2 at mmcsd0
mmcsd0rpmb: 4MB partion 3 at mmcsd0
mmc1: No compatible cards found on bus
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/mmcsd0p2 [rw]...
uhub0: 13 ports with 13 removable, self powered
lock order reversal:
 1st 0xfffff8000417e240 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2607
 2nd 0xfffffe0000e46500 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:282
 3rd 0xfffff800042a09a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2607
stack backtrace:
#0 0xffffffff80b2bba3 at witness_debugger+0x73
#1 0xffffffff80b2ba24 at witness_checkorder+0xe34
#2 0xffffffff80a9cbeb at __lockmgr_args+0x88b
#3 0xffffffff80dc2565 at ffs_lock+0xa5
#4 0xffffffff810f7af9 at VOP_LOCK1_APV+0xd9
#5 0xffffffff80ba7006 at _vn_lock+0x66
#6 0xffffffff80b9599f at vget+0x7f
#7 0xffffffff80b87891 at vfs_hash_get+0xd1
#8 0xffffffff80dbe25f at ffs_vgetf+0x3f
#9 0xffffffff80db4886 at softdep_sync_buf+0xd16
#10 0xffffffff80dc3354 at ffs_syncvnode+0x294
#11 0xffffffff80d999ff at ffs_truncate+0x6df
#12 0xffffffff80dca7f1 at ufs_direnter+0x641
#13 0xffffffff80dd393c at ufs_makeinode+0x61c
#14 0xffffffff80dcf5b4 at ufs_create+0x34
#15 0xffffffff810f51d3 at VOP_CREATE_APV+0xd3
#16 0xffffffff80ba6908 at vn_open_cred+0x2a8
#17 0xffffffff80b9f14c at kern_openat+0x20c
ugen0.2: <Dell Dell USB Entry Keyboard> at usbus0
ukbd0 on uhub0
ukbd0: <Dell Dell USB Entry Keyboard, class 0/0, rev 1.10/1.15, addr 1> on usbus0
kbd2 at ukbd0
ugen0.3: <SanDisk Cruzer Fit> at usbus0
umass0 on uhub0
umass0: <SanDisk Cruzer Fit, class 0/0, rev 2.00/2.01, addr 2> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:0:0: Attached to scbus0
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <SanDisk Cruzer Fit 2.01> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 4C530302741216116074
da0: 40.000MB/s transfers
da0: 3819MB (7821312 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
re0: link state changed to DOWN
GEOM_PART: integrity check failed (da0s4, BSD)
GEOM_PART: integrity check failed (ufsid/5a1180062a826673, BSD)
GEOM_PART: integrity check failed (diskid/DISK-4C530302741216116074s4, BSD)

dmesg on

January 20, 2018 12:00 AM

January 02, 2018

Robert McWilliam

Save chrome eating battery life

I seem to collect chrome tabs. I usually have a few dozen open, often many more if I haven't cleaned them out for a while. Once upon a time reboots would clear them out but these days I reopen the previous session when I boot so tabs don't go away unless I actually tell them to. Lots of them do stuff in the background so chrome ends up using a decent chunk of CPU time. It is almost always at the top of the list in powertop.

Most of the time I don't care about this. If my computer is plugged in then it's fine. When I'm running on battery though I care a lot more. Then when I'm doing something that I don't need the web I put chrome to sleep with killall -STOP chrome and then when I want it back: killall -CONT chrome. Bind those two commands to keys and I can stop and start chrome as needed, and when I don't need it it doesn't get to eat my precious battery life.

January 02, 2018 12:00 AM

November 07, 2017

Iain R. Learmonth


Mastodon + XMPP + SIP

This is an idea. I haven’t looked at actually doing it yet, but I might.

Mastodon needs complimentary instant messaging and I’ve been thinking about how this might be achieved. XMPP and SIP are great federated protocols and it’s possible to use the same domain used for Mastodon for these through SRV records.

Authentication for XMPP and SIP is based on passwords. Mastodon could be extended to write out password hashes to a database to have one password for each service per device. You could then have those servers query the database and check password hashes. I’m not sure what the security properties of this would be compared to OAuth, but I think it’s similar just without the in-band setup.

I’m told there was previously an attempt to build out a social network on top of XMPP which sounded interesting, but OStatus seems to have won in this space. Still, if it’s possible for Mastodon to control authentication for the XMPP server, temporary passwords could be created for access from a web client embedded in the Mastodon frontend to provide federated real-time chat.

Adding users to your roster would likely need to be a seperate step (which would perhaps be beneficial to those that don’t want to chat with everyone they follow). The actual level of integration may vary between instances and that’s perfectly fine, implementations would need to consider this.

Presence information (online/offline status) would perhaps raise privacy concerns and so information would need to be conveyed to users about possible implications.

I’m not sure whether to start off with the backend work or the frontend work either. It would be possible for one to exist without the other entirely, you could link an XMPP account to your Mastodon account and use an external service, or you could only create the XMPP account from Mastodon and use only external clients.

If this is something that interests you, I’d be happy to have a discussion about it. On Mastodon I am

November 07, 2017 03:00 PM

November 06, 2017

Iain R. Learmonth


Talking to Hackers

Yesterday Ana finished setting up FreePBX for our house, and revived the Cisco SIP phones to make them useful again. Eventphone provides SIP, DECT, GSM and other telephony technologies at hacker events like CCC and also runs a long-running SIP service, known as EPVPN to connect hackerspaces and hackers between events.

We set up our extension as a trunk in FreePBX and could easily test outbound calls, but it was a little more difficult to test inbound calls. There was no system to queue up a test call (at least that I could find) so we needed to find someone to call back manually.

SpaceAPI provides a JSON document that lets you know if a hackerspace is open or not. It was 11pm in the UK so I wanted to make sure whoever I was calling wouldn’t mind the call. Once I found a hackerspace (Hackspace Marburg) that was open I placed a call and someone answered, but the hackerspace was in Germany and the call was answered in German. Luckily, they also spoke English.

After looking up the documentation of how to call Eventphone extensions using their hackerspace PBX, I got a call back and confirmed that I could receive incoming calls.

Some next steps:

  • Add extensions for the DECT phones which live downstairs
  • Set up a SIP phone for 57North Hacklab for which I have registered extension 1057

November 06, 2017 11:00 PM

November 03, 2017

Iain R. Learmonth


Free Software Efforts (2017W44)

Here’s my weekly report for week 44 of 2017. This has been a quiet week.

Tor Project

In this week I have looked at metrics-bot being compliant with the Tor metrics team’s Java style guide (#24080 ) and integrating it into a unified package naming scheme (#24036 ).

There has been a lot of refactoring in metrics-bot this week as components are starting to mature. I have started work on using metrics-base, the metrics team directory structure for Java projects and also formatting, but this still needs more work before it is completed.


I believe it is important to be clear not only about the work I have already completed but also about the sustainability of this work into the future. I plan to include a short report on the current sustainability of my work in each weekly report.

I have not had any free software related expenses this week. The current funds I have available for equipment, travel and other free software expenses remains £60.52. I do not believe that any hardware I rely on is looking at imminent failure.

November 03, 2017 10:00 PM

October 29, 2017

Iain R. Learmonth


Free Software Efforts (2017W43)

Here’s my weekly report for week 43 of 2017. This has been a quiet week.

Tor Project

In this week metrics-bot has had some issues where Onionoo was disappearing (#23928 ). I have been building in better error handling for requests to external services. I’ve also started to investigate other templates for status updates that might be useful (#23937 ), and have added a configuration framework to allow for credentials for services to be externally provided (#23933 ).


I believe it is important to be clear not only about the work I have already completed but also about the sustainability of this work into the future. I plan to include a short report on the current sustainability of my work in each weekly report.

I have not had any free software related expenses this week. The current funds I have available for equipment, travel and other free software expenses remains £60.52. I do not believe that any hardware I rely on is looking at imminent failure.

October 29, 2017 10:00 PM

October 26, 2017

Iain R. Learmonth



I joined Mastodon today. So did metrics-bot who is now simultooting to the fediverse.

metrics-bot on Mastodon

metrics-bot on Mastodon

Annoyingly, the API for Mastodon is different enough from the API for Twitter that I’ve needed to use a new library in metrics-bot. I’m using the mastodon4j library for now, but I’m really using so little of the API I do wonder if it would be easier to just reimplement the parts that are needed and drop the rest. The only part I’m really worried about is the OAuth authentication and I’m sure that won’t be anywhere near as bad as I think it will be.

October 26, 2017 10:00 PM

October 18, 2017

Robert McWilliam

I want a new camera

I think it's about time I got a new camera. I've thought this for a while, but keep baulking at the price of getting something that would be a decent improvement over what I currently have.

What I have

I have a Sony A55 at the moment. The internet tells me it was released in 2010, and I think I got mine between Christmas and New Year that year. On the lens side I've a Sigma 18-250 super zoom and a 50mm prime that cover most of what I do and I still have the 18-70 kit zoom from my A200 that I sometimes use as I have close focus filters that fit it and let it get really up close to stuff.

What I want

I'd like better performance - noise levels at higher ISOs look much better on more modern cameras, higher res video and not overheating when shooting video would be nice and, though 16 megapixels is usually plenty for what I do more would allow a bit more cropping.

But, the overriding consideration for me getting a new camera is to get something smaller.

My current setup is big enough and heavy enough that I mostly leave it at home - I want something small enough that it would live in my laptop bag (since that comes most places with me), and not be something that I have to think about if it comes or not when I'm travelling.

I definitely want a viewfinder on a camera. I much prefer that to only having a screen - I think it helps to have the picture you're taking be all that you see when composing it. This apparently makes me a bit weird as it seems to be only a small subset of cameras that still have viewfinders.

What I'm considering

Micro 4/3

For the most part the micro 4/3 bodies are a lot smaller than my current camera. The lenses are definitely smaller from targeting the smaller sensor - which has the nice effect of them also being quite a bit cheaper than equivalents for APS-C or 35mm. I really like the Panasonic rangefinder style GX cameras.

Canon EOS M5

APS-C 24 megapixel sensor in quite a compact body. But quite expensive, doesn't do 4K and a disappointing and expensive selection of native lenses.

Nikon 1 V3

Seems to be quite expensive for the specs. Again the native lenses have a disappointing selection and look expensive. And they're a pain to find as search results are swamped by "normal" Nikon lenses - "1" is a stupid name for a lens type.

Sony mirrorless

Maybe switch over to Sony E-mount? A6000 looks like it would be a decent step up from the A55. There's a decent amount of E-mount lenses available (though a bit more expensive than micro 4/3) and adaptors to let you use just about anything else. A bit higher res than the best of the micro 4/3 cameras. I was pretty tempted by the A6500 when it came out, but I don't think I want to spend that much and the jelly from the rolling shutter seems particularly bad with it.

A compact camera

If I really want to go all out on small: get something like a Sony Rx100 or Panasonic LX100. These are small enough to put in a pocket or clip onto my belt so could definitely come everywhere with me. I'm not really sold on giving up on interchangeable lenses though.


I think a small normal crossing zoom would be the best bet for usually living on the camera lens. That should give a nice small package for always carrying around and still be reasonably flexible.

I've seen a few reviews of the Olympus 40-150 that say it's quite good and very good for the price: from ~£50 on eBay. If I get a micro 4/3 camera I'll probably get one of these to have something for the longer end. Might not get that right away.

The Panasonic 12-35 f2.8, or Olympus 12-40 f2.8 "premium"/"professional" zooms look really nice, but they start at ~£500 on eBay so would eat pretty much all of my budget without getting a camera to put them on.

Micro 4/3 or Sony E-mount have a pretty small flange to sensor distance and with the sensors being quite small (especially micro 4/3) you can get adaptors to use most other types of lenses. There are a lot of really cheap and fast C-mount lenses out there, and some old M42 lenses look quite nice. C-mount extension tubes look fun for some macro stuff.

Speed Booster

Metabones Speed Booster gives a 0.71x focal length multiplier (on the ultra, 0.64 on XL variants) to allow full frame lenses to be used while kind of counteracting the crop from the small sensor and making the lens about a stop faster. The metabones version is quite expensive (and not widely available - I haven't yet found any for sale in Europe), there are a couple of cheaper alternatives, but reviews I've seen have been fairly negative for them. Since, to me, the whole point of this kind of adaptor would be to use some of the nice older lenses the adaptor adding issues would kind of spoil that. If I can live with distortion and vignetting C-mount lenses at about equivalent speeds are a couple of orders of magnitude cheaper. Some nice older lenses on a Speed Booster is fun to fantasise about but is probably still more expensive than I'm likely to spend in the near future.

A Speed Booster would let me get some fast older manual primes and make them even faster. Or there are versions that support modern AF lenses but those adaptors are more expensive and the lenses tend to be lots more expensive.

Crop factor

I was a bit worried about the crop factor taking away some of the wide end. My current widest lens is 18mm on the wide end of my kit and super zooms - adn I have often wanted something a bit wider than that. On my APS-C camera that comes in at about 29mm full frame equivalent. Need about 14mm to match that so the cheap standard zooms are about what I currently have and the premium/pro/bloody expensive standard zooms would be a bit wider. EBay has some cheap C-mount lenses out to 7mm without being listed as fisheye but they might have enough distortion that they actually would be fisheye in my book.


I'm still havering. I have bid on a couple of GX7s on eBay but other people seem to think they're worth more than I do, so haven't won them. I'm really surprised how much they go for, only £50-100 less than a GX80 (and second hand GX80s are surprising close to new ones - the higher end of the range are listing them above the new price on Amazon). I think the improved detail from the GX80 not having the low-pass filter makes it worth the extra over the GX7.

So, I'm currently leaning towards a Panasonic GX80 with the kit zoom and a couple of C-mount manual primes. Can maybe upgrade to native, or M42 primes later. I think I'll spend a while watching them on eBay but it is very tempting to push the button on Amazon to get one tomorrow...

October 18, 2017 11:00 PM

August 22, 2017

Tom Jones


FreeBSD on the GPD Pocket

In the distant past before smart phones became identical black rectangles there was a category of devices called palmtops. Palmtops were a class of PDA PC thing that fit in the palm of your hand. Today the Psion 5 series of devices most often capture peoples attention. Not only are they small and awesome, but they have something like a real keyboard.

This form factor is so popular that there are projects trying to update Psion 5 devices with new internals. The Psion 5 is the sort of device I have complained isn't made for a long time, at some point I picked one up on ebay with the intention of running the NetBSD port on it.

Earlier this year the world caught up and two big crowd funding projects appeared for modern Psion like palmtop devices. Neither the Gemini or the GPD Pocket campaigns convinced me that real hardware would ever appear. In May reviews of the GPD Pocket started to appear and I became aware of people that had backed and received their earlier campaign for the GPD WIN.

With a quirk in indiegogo allowing me to still back the campaign I jumped on board and ordered a tiny little laptop computer.

GPD Pocket vs Psion 5mx


FreeBSD is the only choice of OS for a pc computer. Support is good enough that I could boot and install without any real issues, but there was enough hardware support missing that I wanted to fix things before writing a blog post about it.

Somethings don't work out of the box others will need drivers before they will work:

  • Display rotation
  • WiFi (broadcom 4356)
  • Bluetooth (broadcom BCM2045A0)
  • Audio (cherry trail audio chrt54...)
  • Graphics
  • Nipple
  • USB C
  • Keyboard vanishes sometimes
  • Battery
  • Suspend
  • Touch Screen (goodix)
  • fan (there is some pwm hardware)
  • backlight
  • I2C
  • gpio


The most obvious issue is the display panel, the panel it self reports as being a high resolution portrait device. This problem exists in the bios menus and the windows boot splash is rotated for most of the time.

GPD Pocket FreeBSD bootsplash

Of course the FreeBSD bootsplash and framebuffer are also rotated, but a little neck turning makes the installer usable. Once installed we can address the rotated panel in X, accelerated graphics are probably in the future for this device, but the X framebuffer drive is good enough for FreeBSD hacking.

With X we can sort of the rotation problem. xf86-video-scfb is required to use the framebuffer.

# pkg install xf86-video-scfb

And the following lines have to be added to /usr/local/etc/X11/xorg.conf.d/driver-scfb.conf

Section "Device"
    Identifier "Generic FB"
    Driver "scfb"
    Option "Rotate" "CW"

 Section "Device"
     Identifier    "Card0"
     Driver        "scfb"

GPD Pocket FreeBSD

The screen resolution is still super high, there doesn't seem to be anyway to do DPI hinting with the framebuffer driver (or in i3 at all), but I can make terminals usable by cranking up the font size.

Keyboard and touchpoint

A Keyboard is vital for a usable computer, out of the box the keyboard works, but the touch point does not. Worse, touching the touch point caused the built in USB keyboard to die.

Some faffing trying to debug the problem with gavin@ at BSDCam and we got both keyboard and mouse working. For some reason my planck keyboard presents as a mouse among other things, pluggin in a mouse and power cycling the USB device caused ums(4) to correctly probe and attach.

Manually loading ums(4) at boot got the touch point working correctly. In fact, ig4(4) also attaches when manually loaded.

Add these lines to /boot/loader.conf


The dmesg shows some problems with ACPI probing, this is probably the source of some of the device problems.

Other devices

Wifi, bluetooth and graphics are bigger problems that will hopefully be caught up in others work and made to work soon. The touchscreen controller is adding a driver and support for Cherry View GPIO, there are datasheets for these and I am working on them.

No battery level indicator makes it annoying to use the GPD Pocket out and about. Without a driver the charge controller is using a really low current to recharge the battery. Datasheets are quite readily available for these devices and I am writing drivers now.

GPD Pocket

The Pocket is a great little device, I think its 'cuteness' makes everyone fall in love with it on first sight. I am really looking forward to getting the final things working and using this as a daily device.

August 22, 2017 12:00 AM

August 21, 2017

Tom Jones


Gherkin 30% keyboard


I like keyboards, I have been using an OLKB Planck as my daily driver for 18 months now. I saw a really nice ortholinear 30% keyboard go by on mastodon and I had to have one.

The keyboard I saw was actually the excellent gherkin by di0ib. di0ib has worked in the true spirit of open source and provided all of the design files and firmware for the gherkin. Beyond that they have included child proof instructions to order pcds.

Gherkin PCB

I tricked some friends into agreeing to build boards if I got a run of PCBS and set off. Amazingly was offering 5 more boards (10 vs 5) for just $2 extra. I managed to get 10 sets (board, key plate and base) of the PCBs for about £80.


The build was really easy to do, there is some advice for the socket on 40 percent club, but if you test fit everything as you go it should be straight forward. A build is probably around 2 hours depending on proficiency.

Parts Per Keyboard:

1  Keyplate PCB
1  Bottom PCB
1  Main PCB

16 M2 Spacers (14mm length)
32 M2 screws

30 key switches
30 key caps

1  Arduino Pro micro
1  machine pin socket (wide 24 pin (2x12))

30 3mm leds (your choice of colour)
30 1N4148 diodes
1  100 ohm resistors
1  100k ohm resitors
30 470 ohm reistors
1  mosfet (probs A04406A 4406A)

Key caps are a harder thing to buy (so many awesome choices) so I ended up using some spares I found in a desk drawer.


Flashing the firmware to the keyboard was a little harder to figure out. Eventually I found some instructions that included the correct avrdude flags on, you also need to use a switch pulling RST down to GND to put the micro controller in programming mode.

Pro Micro flash switch

Most of the work is done by the TMK make file, but you must manually specify a target for the program command. The command I used looks like:

# programming directive
MCU = atmega32u4
PROGRAM_CMD = avrdude -p $(MCU) -P /dev/tty.usbmodem1411 -c avr109 -U flash:w:$(TARGET).hex


With the board built and programmed (first try) it is time to figure out how to use it. It took a couple of months of daily use to get used to using the planck, it will be the same with the gherkin. To help learn I have printed out the keyboard layout and the combination of layers.

Gherkin Layout

I modified the default layout a little to make it more similar to how I normally type. I moved space bar to my left hand, made 'X' a repeatable key(gotta be able to delete chars in vim) and added a 'CMD' key. I have a fork of the repo with my layout and Makefile changes.

The layer system is easy to use, if you hold any of the keys on the base layer it will enable the alternate function for a meta key or it will switch to another layer for a layer key.

August 21, 2017 12:00 AM

May 23, 2017

Bob Taylor

Dementia Tax

So it is well known that the Tories have been nibbling at the NHS to privatise services but to actually privatise social care is shocking.

The idea behind the NHS was that everyone would be covered by a country wide insurance scheme so why now try to make everyone pay for their own long term care whether that be in the house or a care home.

The tempter for Tory politicians and their right wing friends is the amount of money currently swilling about in houses and bank accounts for a section of the elderly.  This is a big prize. So these financial corporations have already got equity release products that could, I presume, easily be adapted to suck money out of your assets making these financial companies very happy.

So in effect this was a staggeringly stupid attempt to give the financial corporations a huge boost to profits on the back of the grandparents.

Then there is the notion that this is intergenerational fairness.  It is nothing of the sort.  If the money is taken out of your house value it then does not go to your children and grandchildren as an inheritance.

Make no mistake this was an attempt at "mugging" all generations on the back of a long term sickness.

There are plenty of other ways of raising money to cover social care that are fair and do not penalise sick old people while at the same time penalising the young people's inheritance.

by BobT ( at May 23, 2017 07:21 AM

April 06, 2017

Edward Watson


Project Intro: Research and design of self-organising, algae farming, soft robots

An amateur home-based project to design and evaluate self-organising, algae farming, soft robots.

This is a project started inpersuit of a personal interest in home-grown complete food sources. The notes below for my project log and project substages will be added ad-hoc.

The project itself will take place in 3 phases: (1) Stable Spirulina cultivation and remote monitoring via a conventional self-built monitoring platform, (2) evaluation and improvement of a self-built remote sensing and actuator 'nodes', tethered and water-bound to distribute aeration, inspection and monitoring; and (3) Migration from central controller-based coordination to autonomous, untethered agent-based methods. Succinct project updates will be made on the Research Gate platform whereas informal project notes will be made on the Author's personal blog (


Phase 1: Stable Spirulina cultivation and remote monitoring via a conventional self-built monitoring platform.

1.1 Preparing the culture medium

Chemical ingredients for the cost-efficient RM6 culture medium [1] (Roof, Kaushik and Prasanna 2006) were obtained via various Amazon and eBay supplier sources. Super single phosphate (SSP) was substituted with an alternative inorganic phosphate source Potassium Phosphate due to SSP sourcing problems. Measures of all fertiliser ingredients were made for a 4l liquid volume within a 4.5l demijohn growth vessel, marked GV1:

  • Potassium phosphate 5g,
  • Sodium nitrate 10g,
  • potassium Chloride 3.92g,
  • magnesium sulphate 0.6g,
  • calcium chloride 1.6g,
  • sodium chloride 2g,
  • sodium bicarbonate 32g.

The stated measures of each ingredient were made within 0.5l of dechlorinated local tap water and added to 3.5l within GV1. Any crystallised ingredients were ground to powder form.


April 06, 2017 02:19 PM