Rein van 't Veer

A Generative Life Drawing Model

Sat, 26 Mar 2022 00:00:00 +0000

On how AI can be of use to artists

The word “model” has to be one of the most overloaded terms in the history of human language. An abstraction or physical thing, tangible or ephemeral, something that stands for something else but can be something of itself at the same time. In this article I will combine “models” of both kinds, from the artistic domain and the “artificial intelligence” domain, although I try to shy away from this presumptious term, I’d rather opt for “machine learning” in most cases.

An artist's "model"? Or the real thing? Source: Pixabay, by Victory Borodinova

The Ford "model" T. Yep, a model too. Source: CC-BY-SA 4 WikiMedia Commons, by Acabashi

Art models

The artistic context: a model (in the stricter sense) is usually a person. Someone offering to pose for artists such a sculptors or painters. The model acts as a reference for the artist to draw or sculpt a work of art. There is, of course, something weird about this. The “model” in this case, the actual thing. The artwork could be more aptly described as a model for the real thing, rather than the other way round!

Machine learning models

Machine learning models usually are proxies for a real (or an imagined) world domain. “Classic” domain examples, in as far as anything in a field so young can be called as such, are models for the “MNIST” handwritten digit recognition or the “ILSVRC” image recognition challenge. These datasets have played a significant role in shaping machine learning history. The “models” in the machine learning case, are pieces of software that are able to perform some task or other: recognizing digits, or classifying images.

Models generating models

Machine learning and art share a great deal of shared interest. There are numerous examples of cases where machine learning has been used to generate or emulate art, often with great success.

In my previous post I used a generative model to generate images from text to include in my article. This was more something of a gimmick, to spice up the article a bit and to play around. But it sparked my interest again in using generative models for artistic purposes. Especially since I started sharing an atelier with seven other art painters.

FAFA-VAE

Troubleshooting

The first iteration of my model created logs like this:

 1/16 [>.............................] - ETA: 51s - loss: 283867.6875 - reconstruction_loss: 283867.4375 - kl_loss: 0.2540
 2/16 [==>...........................] - ETA: 22s - loss: inf - reconstruction_loss: 70986436616009023488.0000 - kl_loss: inf
 3/16 [====>.........................] - ETA: 20s - loss: nan - reconstruction_loss: nan - kl_loss: nan
 4/16 [======>.......................] - ETA: 22s - loss: nan - reconstruction_loss: nan - kl_loss: nan
 5/16 [========>.....................] - ETA: 19s - loss: nan - reconstruction_loss: nan - kl_loss: nan
 6/16 [==========>...................] - ETA: 16s - loss: nan - reconstruction_loss: nan - kl_loss: nan
 7/16 [============>.................] - ETA: 14s - loss: nan - reconstruction_loss: nan - kl_loss: nan
 ... (etc)

When this happens, you know there is something seriously wrong with the training. It could happen randomly, but in my case, it was consistent. Deep learning is a much more a matter of “negotiation” with a model, data and a loss function than “normal” programming is. At times it really feels like you have to “coach” your model into training properly.

So what do these nans mean? nan is, of course “not a number”, which happens whenever your model tries to:

divide by zero
tries to compute log(0)
tries to take the root of a negative number
all the things they taught you that you shouldn’t do with numbers

When we look closely, we see that the model doesn’t start out with nan loss. It actually computes something sensible, but right after epoch 1, it spirals out of control. The reconstruction loss explodes and the kl_loss, which is the Kullback-Leibler divergence loss, is so large that it overflows the max value and becomes inf. This is generally caused by a phenomenon known as “exploding gradients”. The trouble is that there are many known causes, but fortunately also many known fixes for it. It is also an inescapable part of machine learning model development.

Things that may help:

Normalizing/whitening of your input data. Deep neural nets generally need data with mean 0 and unit variance. In my case, I already used either feature-wise normalization or sample-wise normalisation, but no whitening.
Lowering the learning rate. In my case, this helped to debug things. At the very least it allows you to see how the model losses develop.
Fix your loss function. Maybe there’s a mathematical error or a bug in there somewhere. Look closely.

In my case, I switched from a binary cross-entropy to a mean squared error to calculate the image reconstruction loss. Why this helps in the first place is still beyond me. The reasons to choose either one of these loss functions is quite complex.

The easier way to K8s Operator development using Argo Events

Sat, 29 Jan 2022 00:00:00 +0000

On why you don’t need special operator SDKs to develop K8s Operators

"Kubernetes Operator using Argo Events". An AI impression. Source: Allen Institute for AI Computer Vision Explorer

TL;DR (or the “management summary” if you like)

If your organisation uses Kubernetes, chances are that you’re using operators. Once you get to the point that you want to develop your own operators, there comes the choice in how to develop them. Most people go with one of the major operator frameworks, but this isn’t necessarily the right track if you just start out: many frameworks are bulky and hard to set up. What I found out, is that using Argo Events and Argo Workflows, you can

have fully functional operators using just a handful of files (manifests);
have your operator without the need for publishing custom-built Docker images;
have much, much improved inspection into your operators’ handling of resources thanks to the Argo UI.
develop and test the operator logic in the Argo workflow template independently from the operator itself, using the Argo UI;
use any programming language you like, not just the ones supported by the “official” frameworks!

However

The example given below is set up as a tutorial. Make sure you don’t simply copy-paste it into production: let your dev team set things up properly and have your infrastructure security advisor go through it. The tutorial below is just for instruction purposes. It is a serious proposal though. You can have fully functional operators without pushing a single Docker image, or without using a single Operator SDK and have a scalable solution. Bear in mind that:

The setup in this tutorial uses quite a few containers, mostly for debugging. This can be compressed into a single container however, with a bit of refactoring. This, I didn’t put into the tutorial.
This method isn’t the fastest. It uses an Argo Workflow, which takes a few seconds to instantiate, for each time you create, update or delete a custom resource. For the vast majority of purposes, this is fine. If you really must scale your operator to deploy hundreds of applications per hour, then this method can still work, but it might not be the most efficient solution. Then again, few companies need to build an operator from scratch that needs to scale to this level immediately. For starting out, this method is excellent because it gives you so much more transparency.
Operator development is never easy. And with this approach, you will probably need to learn quite a bit of Argo domain-specific language. Like most k8s frameworks it’s huge, but it’s useful for learning it in its own right, as Argo is useful for a lot more than Operator development. And: it’s very well documented.

Intro: Kubernetes Operators

In the past few weeks or so, I followed up on an idea I had that had been lingering in my mind for a while. Something that showed particular promise, but I hadn’t found the time to dig into, until now. I had been involved in Kubernetes Operator development for a few months and had the distinct feeling that things could be improved significantly. But before we go into detail, let’s look at operators for a moment. Kubernetes Operators are central to doing anything in Kubernetes. Operators are controllers that make sure that particular kinds of Kubernetes “things”, be it an application or something else, end up the way you specified them to be.

In kubernetes speak: operators are responsible for reconciliation of the desired state with the actual state for a particular custom resource type that extend the Kubenetes API. Kubernetes comes with its own controllers for Deployments, Ingresses, Pods and what have you - these you don’t need to build yourself. But once you want to extend the Kubernetes system with your own types, you get to Operators.

Why Operators exist

"Extending the Kubernetes API", an AI-generated impression. Source: DeepAI.org Text To Image API

In practice it means that an operator is responsible for the life cycle of a resource type of your own design. For example a particular database type, say PostgreSQL, or something else entirely. Of course there already are perfectly fine operators available for PostgreSQL , so you don’t need to design your own. The reason this operator exists, is because deploying a production-grade PostgreSQL database cluster including a backup strategy, is far from trivial and you need many, many manifests to explain to Kubernetes how you want your infrastructure set up. Operators like these help you declare the layout and settings of your PostgreSQL cluster, set up complex infrastructure, taking most cognitive load off your plate. There are operators that deploy applications, manage storage, or even interface with something outside the Kubernetes cluster.

Two articles of vital importance if you really want to sink your teeth in:

The “Operator pattern” article
The “CNCF Operator White Paper”. CNCF stands for the “Cloud Native Computing Foundation” and is the main organisation backing the design and development of Kubernetes.

Important to know is that:

Operators listen to your self-designed “custom resource definitions”, instances of which are “custom resources”, so you can extend the Kubernetes API itself.
Operators act on create, update and delete events, so they are responsible for the entire life cycle of the resources they create from your custom resource.
Operators should not do any heavy lifting, as they are expected to easily and stably scale. If any resource-heavy work is involved, this is to be left to other resource definitions, such as Jobs.
Operators should be highly available: they’re not supposed to drop the ball and should reliably catch each and every event that they should act on.

Your own operator

So, why would you want to develop your own operator? Well, maybe your organisation offers a particular product, let’s say some kind of communications service. Once your organisation chooses to use Kubernetes to deploy this application, you can define your own resource type that describes the deployment of this application. You can define security parameters, the way it is exposed to the outside world, the data storage it uses to persist information, the amount of resources like memory and CPU it is allowed to take up, etcetera.

"Curly braces all over the place". An AI-generated impression. Source: DeepAI.org Text To Image API

If your application is relatively simple, you may use Helm so you don’t need a custom resource to describe your application to Kubernetes resources. If you can, you probably should. But if your self-defined application definition requires more complicated logic, Helm is not such a good idea. Trying to put lots of “business logic” inside Helm templates results in very hard to maintain Helm charts with curly braces all over the place. Helm is without a doubt a very useful tool, and it will stay being a very useful tool even when you decide to write your own operator: you can use Helm to deploy the operator! We won’t go into Helm in this tutorial, there are plenty of those around. Instead, we’re going to take a look at when you want to implement your own operator.

Operator development and operator frameworks

The current state of operator development is complex to say the least. The framework we used was the exact opposite of everything you would want from an operator framework. The framework was complex, clunky, buggy, enormous, the operator images we made were huge, the operator itself slow and heavy on resources, and hard to maintain. Which framework we used doesn’t matter here, this is not a framework bashing post. All I will say is that all of the options listed under the operator frameworks to use for operator development are more complex than the one I’m about to show you. Furthermore, all of them suffer from visibility problems that we’ll discuss in some detail below. So, let’s get to it!

Argo Events as an operator framework

For about a year now, I had been using Argo Workflows to much satisfaction. We used it to ingest data from external and internal sources, mostly targeting PostGIS databases to pipe the data to. It has been working very nicely, and we created an operator and accompanying “custom resource definition” that allowed us to describe a dataset ingestion operation, which worked very well for us.

Then, we started looking into Argo Events. It’s a great system for letting other infrastructure components know that some state changes occurred, such as completing the ingestion of a data set. Argo Events has a fantastic set of sources that it can listen to, and if that isn’t enough you can even define your own. Argo Events splits event handling into nicely and cleanly separated parts with each its own responsibilities:

an EventBus, which will keep event messages until handled,
an EventSource, which will put a message on an event bus for a particular state change,
a Sensor, which will listen on the event bus for particular messages and translate these into actions

This system alone is a very nice message bus that you may compare with (but is quite distinct from) products like RabbitMQ (with which I had favourable experience) and Kafka (with which I have no experience). The big difference is that Argo Events is, like all Argo ecosystem products, built to work with Kubernetes. This means that it is probably more “cloud native” than other event handling systems.

One thing that struck me is that EventSources can put a message on the bus on the creation, change or deletion of any resource kind in the cluster. Sounds familiar? It’s exactly the same role as the main responsibility of Operators: to listen to particular custom resources and act on them. So, what if you could use Argo Events to define your Operator, using just that or maybe a few scripts to handle the “business logic” part of the custom resource handling?

It took me a little while to work this out - like any framework on Kubernetes or even Kubernetes itself, it has its own learning curve. But once I saw on how to combine Argo Events and Argo Workflows, it started to dawn on me: this is better than any current special-purpose operator framework and at the same time, it is useful for a lot more as well!

Let’s dig in

So to see this experiment in action, we are going to replicate a tutorial from the Operator SDK and create an operator that handles Memcached deployments. Since I have not used Memcached for anything myself, I can offer little information on its purpose other than that you can use it to take load of your web services by caching http requests, for example. The idea is that, if your http gateway has handled a particular http request, Memcached can keep a copy of the response and serve it much faster because the web service or database the response originated from does not have to go through the operation of assembling the response. This does leave you with the classic hard question on when to invalidate the cache - one of the purportedly “two hard things in Computer Science”.

1. Install Argo Events

The simple solution: follow these steps. To keep things very simple, we’ll go with the cluster-wide installation for now.

NOTE: make sure you only follow the “Cluster-wide Installation” steps, there are three of them. Skip the “Namespace Installation”.

2. Install Argo Workflows

Why do we need Argo Workflows? Because the operator “business logic” is going to be handled in a Workflow. Think of Workflows as much, much better versions of Jobs.

Let’s start with installing Argo server and Workflows cluster-wide:

kubectl create namespace argo
kubectl -n argo apply -k github.com/argoproj/argo-workflows/manifests/cluster-install?ref=master

By default, Argo uses the Docker runtime executor. If you’re on Azure, or you use Podman or some other runtime execution
engine, be sure to patch the Argo Workflow executor:

# Use "pns" executor instead of Docker:
# See https://github.com/argoproj/argo-workflows/issues/826#issuecomment-872426083
kubectl patch -n argo configmap workflow-controller-configmap --patch '{"data":{"containerRuntimeExecutor":"pns"}}'

If your workflow pods fail to start or Argo gives you errors reaching the pods, then switching to a different runtime executor will probably help. Also, for our tutorial we want to disable having to log in to access the Argo UI:

kubectl patch -n argo deployment argo-server --patch \
  '{"spec":{"template":{"spec":{"containers":[{"name":"argo-server","args":["server","--auth-mode=server"]}]}}}}'

Once you have this set up, you can tunnel to the Argo UI using:

kubectl -n argo port-forward svc/argo-server 2746:2746

You can access the UI on https://localhost:2746. The default for the server is to run over https, so you will get a self-signed certificate warning. As the port-forward option claims the terminal (the tunnel is active until you disconnect using ctrl-c), you will have to open a new terminal for the commands following down below.

The most important part here is that the installation comes with Argo Server, the UI that allows you to inspect both EventSources, Sensors and, if you generate them from the Sensor triggers, Workflows. Personally, I’d skip the step to install the argo command line tool, I found it to be of little use. It is, however, very useful in CI test setups, such as the one I made for this tutorial .

3. Install the Memcached custom resource definition

Installing the Argo components can take like a minute to start up. Once it’s up and running, we can start deploying our custom resources. This is not intended to be a full Operator tutorial, so I’ll try to keep things as simple as possible.

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # This must be a combination of the spec.names.plural + spec.group
  name: memcacheds.cache.example.com
spec:
  # This is the api "namespace" you define for yourself.
  # Usually some subdomain of your organisation site url.
  group: cache.example.com
  # This defines the names under which you can list created resources of this kind,
  # using kubectl for example
  names:
    kind: Memcached
    plural: memcacheds
  # Either Namespaced or Cluster-scoped.
  scope: Namespaced
  # `versions` is mandatory:
  # there should be at least one version that is served and stored.
  versions:
    # You usually start out with some kind of alpha version.
    # Be sure to create a roadmap on how to get to a stable "v1".
    - name: v1alpha1
      served: true
      # See also: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#create-a-customresourcedefinition
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            # Like the `spec` key this key resides under, any custom resource
            # created as kind `Memcached` will need a `spec` key.
            spec:
              type: object
              properties:
                # So the only spec key will be a `size` parameter as integer
                # All tutorial business, of course
                size:
                  type: integer
                  description: The number of replicas for the Memcached deployment

You can simply install this with

kubectl apply -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/crd.yaml

4. Create the service account and access rights

By default, the Kubernetes API doesn’t allow access to our newly devised custom resource. By basically anything - we need to tell the Kubernetes API that it’s OK, provided a special service account is used to access it. Like the custom resource definition, this is basic Operator engineering 101. This will be our memcached-sa service account, that comes with the associated rights:

---
# This is just a tutorial setup - adapt
# to your access control requirements
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: memcached-role
kind: ClusterRole
rules:
  - verbs:
      - get
      - list
      - create
      - watch
      - patch
      - update
      - post
      - put
      - delete
    resources:
      - memcacheds
      - workflows
      - workflowtemplates
      - deployments
      - pods
      - pods/log
      - pods/exec
    apiGroups:
      - cache.example.com
      - argoproj.io
      - apps
      # Core API group: for pods
      - ""
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: memcached-rb
roleRef:
  name: memcached-role
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
subjects:
  - kind: ServiceAccount
    name: memcached-sa
    # In case you installed Argo Events in a different namespace:
    # adapt namespace here
    namespace: argo-events
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: memcached-sa
  namespace: argo-events

Install with:

kubectl apply -n argo-events -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/rbac.yaml

4. Install the EventSource

The EventSource specifies what to listen to - in this case our custom “Memcached” resources. We specify that we listen to all life cycle events. In event-speak, these are ADD, UPDATE and DELETE.

---
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: memcached-source
spec:
  template:
    # Assign a service account with `get`, `list` and `watch` permissions
    # on the resource being listened to.
    serviceAccountName: memcached-sa
  resource:
    memcached:
      group: cache.example.com
      version: v1alpha1
      resource: memcacheds
      eventTypes:
        # Types: https://github.com/argoproj/argo-events/blob/00e2ae801addcd362a22613a745ae424932efa40/pkg/apis/eventsource/v1alpha1/types.go#L280
        # Listen to all lifecycle events of the specified custom resource
        - ADD
        - UPDATE
        - DELETE
      filter:
        afterStart: true

You can install this with

kubectl -n argo-events apply -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/event-source.yaml

And we can see it appearing in the UI:

5. Install the Sensor

The sensor specifies what to do with the event messages generated by our EventSource. In our case, we offload the work to an Argo Workflow that calls a WorkflowTemplate (see below) with a few parameters from the event message:

---
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: memcached-sensor
spec:
  template:
    serviceAccountName: memcached-sa
  dependencies:
    - name: source-dependency
      # References the metadata.name in the event source
      eventSourceName: memcached-source
      # References the key directly under "resource:" in the event source
      eventName: memcached
  triggers:
    - template:
        name: argo-workflow-trigger
        argoWorkflow:
          operation: submit
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: deploy-memcached-workflow-
              spec:
                serviceAccountName: memcached-sa
                entrypoint: deploy-memcached-template
                arguments:
                  parameters:
                    # These values get overridden, see `parameters` below
                    - name: memcached-manifest
                      value: ""
                    - name: operation
                      value: ""
                    # These values are required for the Sensor to deploy
                    - name: script
                      valueFrom:
                        configMapKeyRef:
                          name: memcached-cm
                          key: main.py
                    - name: deployment-manifest
                      valueFrom:
                        configMapKeyRef:
                          name: memcached-cm
                          key: deployment-manifest
                templates:
                  - name: deploy-memcached-template
                    steps:
                      # `steps` may look a little funky, as lists of lists:
                      # the outer lists are run sequentially, the inner ones
                      # in parallel
                      - - name: call-template
                          # We call the template reference listed under `name`,
                          # from the template onwards listed under `template`
                          templateRef:
                            name: memcached-deploy-workflow-template
                            template: main

          # These are the parameters of what data gets selected from the event bus
          # message into the resource generated by the sensor
          parameters:
            - src:
                dependencyName: source-dependency
                # The message body is the Memcached custom resource
                dataKey: body
              dest: spec.arguments.parameters.0.value
            - src:
                dependencyName: source-dependency
                # This is the `operation` parameter to the workflow,
                # one of ADD, UPDATE, or DELETE, covering the entire life cycle
                dataKey: type
              dest: spec.arguments.parameters.1.value

You can install this with

kubectl apply -n argo-events -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/sensor.yaml

And it appears in the UI:

6. Install the configmap containing the operator implementation details

Instead of putting all implementation details for the operator into the Workflow or Workflow Template (see next step), we use a common practice in Kubernetes: a ConfigMap. ConfigMaps hold configuration, in our case the script we use to configure the Memcached deployment and the deployment template for our Memcached deployment.

The main reason why this is a good idea is that operators tend to grow in complexity and with it the lines of code required to configure the target resources to apply to the cluster. Putting all this into the WorkflowTemplate will result in larger and larger WorkflowTemplates that become harder to maintain. With ConfigMaps we can “modularize” our operator logic into separate files that are stored in separate ConfigMap keys or even into separate ConfigMaps if we like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: memcached-cm
  labels:
    # Note that this label is required for the informer to detect this ConfigMap.
    # See also: https://github.com/argoproj/argo-workflows/blob/master/examples/configmaps/simple-parameters-configmap.yaml
    # https://github.com/argoproj/argo-workflows/blob/master/examples/arguments-parameters-from-configmap.yaml
    workflows.argoproj.io/configmap-type: Parameter
data:
  main.py: |
    import argparse
    import json
    
    parser = argparse.ArgumentParser(description='Memcached to deployment')
    parser.add_argument('--memcached-manifest', required=True)
    parser.add_argument('--deployment-manifest', required=True)
    args = parser.parse_args()

    memcached = json.loads(args.memcached_manifest)
    deployment = json.loads(args.deployment_manifest)
    
    deployment['metadata']['namespace'] = memcached['metadata']['namespace']
    deployment['spec']['replicas'] = int(memcached['spec']['size'])
    result = json.dumps(deployment, indent=2)
    
    # Printed output goes into outputs.parameters.result
    with open('/output-manifest.json', 'wt') as f:
      f.write(result)

  # Storing the manifest in JSON allows us to use the Python standard library
  deployment-manifest: |
    {
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
        "name": "memcached-deployment",
        "labels": {"app": "memcached"}
      },
      "spec": {
        "replicas": 1,
        "selector": {"matchLabels": {"app": "memcached"}},
        "template": {
          "metadata": {"labels": {"app": "memcached"}},
          "spec": {
            "containers": [{
              "name": "memcached",
              "command": ["memcached", "-m=64", "-o", "modern", "-v"],
              "image": "docker.io/memcached:1.4.36-alpine",
              "ports": [{"containerPort": 11211}]
            }]
          }
        }
      }
    }

As you can see, this ConfigMap also holds the “template” we use for our Memcached deployment. Instead of YAML, we express this manifest in JSON because this is much easier to load. Many programming languages come with a JSON parser in the standard library (think Python, JavaScript, Ruby, Go, PHP, etc), but I know of not a single language with a YAML parser in the standard lib. Since any JSON is valid YAML, we can pass the result as JSON and be a valid output manifest.

Install with

kubectl apply -n argo-events -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/configmap.yaml

7. Install the workflow template to deploy Memcached resources

Finally, we’re deploying the Argo WorkflowTemplate that takes care of the deployment for us - the “heavy lifting” that the Sensor delegated to the Workflow:

---
#  The workflow template that does the "heavy lifting"
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: memcached-deploy-workflow-template
spec:
  # See the `main` template below: this gets executed
  entrypoint: main
  # Use the service account with sufficient rights to deploy
  serviceAccountName: memcached-sa
  ttlStrategy:
    secondsAfterCompletion: 300
  # Garbage collection settings for clearing up old pods
  podGC:
    strategy: OnPodSuccess

  # Global Workflow parameters, passed/overridden from the Sensor
  arguments:
    parameters:
      # Injected by Sensor workflow
      - name: memcached-manifest
        # Sets a compact default which we can test-run in Argo UI
        default: |
          { "apiVersion": "cache.example.com/v1alpha1", "kind": "Memcached",
            "metadata": { "name": "memcached", "namespace": "argo-events" },
            "spec": { "size": "3" } }
      # Injected by Sensor workflow
      - name: operation
        default: ADD
      - name: script
        valueFrom:
          configMapKeyRef:
            name: memcached-cm
            key: main.py
      - name: deployment-manifest
        valueFrom:
          configMapKeyRef:
            name: memcached-cm
            key: deployment-manifest

  templates:
    # A simple logging container template - we'll use this to debug
    - name: log
      inputs:
        parameters:
          - name: content
      container:
        name: simple-logger
        image: ubuntu:latest
        command:
          - echo
        args:
          - "{{inputs.parameters.content}}"

    # The actual "business logic" for our operator.
    # As tutorials go, it's a bit contrived - there's very little
    # action going on
    - name: operator-logic
      container:
        name: python-operator-logic
        image: python:3.10-slim
        command:
          - python3
          - -c
          - "{{workflow.parameters.script}}"
          - --deployment-manifest={{workflow.parameters.deployment-manifest}}
          - --memcached-manifest={{workflow.parameters.memcached-manifest}}
      outputs:
        parameters:
          - name: result
            valueFrom:
              path: /output-manifest.json

    - name: operation-mapping
      script:
        image: python:3.10-slim
        command:
          - python3
        # The "operations" from our event source differ from the ones used in the
        # deployment syntax - we need to map from one to the other
        source: |
          mapping = {
            "ADD": "apply",
            "UPDATE": "apply",
            "DELETE": "delete",
          }
          print(mapping["{{workflow.parameters.operation}}"])

    - name: deploy
      inputs:
        parameters:
          - name: action
          - name: manifest
      resource:
        action: "{{inputs.parameters.action}}"
        manifest: "{{inputs.parameters.manifest}}"

    # This is the actual entrypoint for our workflow
    # It takes care of the execution flow
    - name: main
      steps:
        - - name: log-memcached-manifest
            # Uses the "log" template
            template: log
            arguments:
              parameters:
                - name: content
                  value: "{{workflow.parameters.memcached-manifest}}"

        - - name: log-operation
            # Again: uses the "log" template below, so we only need to define once
            template: log
            arguments:
              parameters:
                - name: content
                  value: "{{workflow.parameters.operation}}"

        - - name: parameterize-replicas
            template: operator-logic

        - - name: log-result
            template: log
            arguments:
              parameters:
                - name: content
                  value: "{{steps.parameterize-replicas.outputs.parameters.result}}"

        - - name: map-operation-to-action
            # Calls the actual operator "business logic" part
            template: operation-mapping

        - - name: log-action
            template: log
            arguments:
              parameters:
                - name: content
                  value: "{{steps.map-operation-to-action.outputs.result}}"

        - - name: deploy-memcached
            template: deploy
            arguments:
              parameters:
                - name: action
                  value: "{{steps.map-operation-to-action.outputs.result}}"
                - name: manifest
                  value: "{{steps.parameterize-replicas.outputs.parameters.result}}"
# Done! 

Install with

kubectl apply -n argo-events -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/workflow-template.yaml

Voilà, the workflow template is present in the Argo UI:

This is a fully functional workflow template. As most workflow templates, we can issue workflows from it, using the Argo UI using the “submit” button. You can test your operator logic in this way. Note that this does not test the correctness of the Sensor, though!

8. Let’s go!

Now we can deploy our Memcached instance:

---
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached
spec:
  # Let's try it out!
  size: 3

which we should be able to, in good operator-fashion, apply in any namespace we want in our cluster. Let’s say that we want our Memcached deployment to go into the “services” namespace:

# Create fresh namespace
kubectl create namespace services
# This is where we want our Memcached deployment to live:
kubectl apply -n services -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/memcached.yaml

And the workflow completes, deploying our application:

How can we tell it’s running?

And we can watch as our operator cleans it up again:

kubectl delete -n services -f https://raw.githubusercontent.com/reinvantveer/reinvantveer.github.io/master/_includes/memcached/memcached.yaml

A Quality Definition of Done for Python projects

Tue, 20 Jul 2021 00:00:00 +0000

Setting an internal standard for code quality for your Python software

I love Python because it’s concise, readable and its speed is sufficient for most of the work I do. What I particularly like about the language is the fact that it really encourages collaboration between team members. In Python, it’s just a little harder to make a mess of things than in some other programming languages. However, this does not come without issues.

This article is about how to manage code quality around team-maintainable Python software projects. It’s based on quite a bit of hard-won personal experience. My recommendations certainly are not meant as a pedantic dictate. However, I intend to clear up any questions on why each recommendation for a quality DoD is of particular value.

Development and Definitions of Done: why code quality matters

It’s not uncommon for teams to have an unclear picture of what constitutes the definition of done for doing work, maybe even more so for Python projects. Often team members are just starting out and quite possibly are even new to Python itself. Also, more often than not, Product Owners or other team leads often fail to see the importance of code quality when they realize that more time is required on a product that already “just works”. If the customer is satisfied, why do anything anymore?

Does this sound like a familiar scenario already?

Many code repositories start to rot when they are hard to maintain. This isn’t difficult to see: someone wrote a bunch of Python code, but no-one really grasps what has been written and why. Often a complex intertwining of Python, SQL, some RegEx sprinkled on top, maybe some JavaScript, Bash or R thrown in, and suddenly:

The “someone” who wrote all the code moves along to another position in the company, or leaves for another.
The product that once “just worked” suddenly “just doesn’t” after a few months and nobody seems to know how to fix it.

So what just happened? Probably the team didn’t set a clear definition of done for this project to be maintained by a team. This team-maintainability is a big thing. Only if the team understands the code base, then the code base can be maintained by it. This sounds gratuitous, but often shared knowledge is really hard for teams to maintain. Probably someone (“the SQL guy”) got type-cast into a role that nobody really understood. Probably the team ran out of time at some point at which the code wasn’t reviewed.

This is why having a solid agreement within the team of what constitutes good quality code is important. Even if human or financial resources are constrained at some point, at least team members can agree on that at some other point in time, this technical debt has to be fixed.

So what follows here is what I consider a good set of quality standards to agree upon, and why they matter.

A set of DoD quality standards

Document your project and agree on what it means to document

This may be harder than it sounds, but you need to document your code in order for future maintainers of your code. This isn’t hard to grasp or accept for anyone I met in IT. I recently found out that there’s some best practices for this, but you probably need to write different kinds of documentation. Write inline comments and docstrings. Document how your documentation is organized, write tutorials and topic guides. If your project is itself a library, then compile reference guides from your docstrings using Sphinx, and write how-to guides. Agree with your teammates on what should and what shouldn’t be included in the documentation, agree on some kind of documentation template that will make things easier for everyone.

Review each other’s code in a source code management system

There really is no way around this one. If you want your team members to be in on what you write and why, have your code reviewed by someone. Probably by someone with less experience. Why? Make sure they understand what you wrote and why, even if it means admitting for that person that they don’t understand everything and that that is quite OK. Share your knowledge and allow others to grow. If you are uncertain on reviewing some issues, just ask another team member to pitch in. It almost goes without saying that you should use some kind of source control management system like Git, Fossil or perhaps even Mercurial in order to keep track of changes but there is generally no disagreement on this. Almost everyone uses Git.

Write automated tests for your code

If you write code for your team, help them out maintaining your code by writing tests. Tests are there to make sure that the product meets the requirements. There’s a pretty big chance that the only place where the formal requirements for your product are documented, is in your tests! Writing tests means that, once the code needs to be changed, these requirements are still met. Have the requirements changed? Change the tests! There’s simply no good excuse not to write tests and I’m pretty sure I’ve heard all of them:

“There’s not enough time”, which is probably favourite. Is there time a couple of months down the road to spend days on figuring out what the code does and why? Is there time a couple of years down the road to have to re-write everything because the requirements are unknown? If there is no time to write tests, don’t do the project.
“We don’t write tests here”. You should have these people read the horrifying case of the THERAC-25 and ask them again if they still think it unnecessary to write tests. Writing tests is an ethical obligation for software developers. It’s simply irresponsible not to do so.
“I don’t know how to write tests”. Yes, this is a tricky one. Writing tests can be difficult, in fact often it’s more difficult than writing the production code itself. Fortunately, there’s plenty of useful resources out there, just do a web search for them. If you need advice: I’d start with pytest, it’s probably the most sensible framework to use these days. But most importantly: collaborate with teammates on learning to write good tests. Practice, and you’ll get better in it.

Use a style checker

Style checkers can catch an incredibly large set of “code smells” or just stuff you forgot to put right. Unused imports, unused variables, unreachable code, indentation errors, you name it. All these things can confuse your fellow team members considerably (“why is this here???”), reduce the cognitive load and probably make your code run smoother as well. Often, there’s little time involved in type-checking your code. If you need advice: use flake8 with this configuration in your setup.cfg (adapted from here):

[flake8]
# The .git folder only holds version control information
# Cached directories only hold compiled Python files that can't be checked
# Build and dist directories contain derived artifacts
exclude = .git, __pycache__, build, dist
max-complexity = 10
# Use modern IDE line length settings as default. We're not in teletype-age anymore
max-line-length = 121

Type-check your Python code

In case you’re not familiar with type hints in Python: start here. Dynamic typing is a big source of both runtime failures and confusion in Python code. Use mypy or another type checker to make sure that you got your types right. Writing typed python helps for no less than three quality aspects:

To make sure that you pass the right arguments to functions, and handle its return types appropriately
It clarifies to your teammates what a function or method expects. Is users a list? A dictionary? An object? Specify the type of it!
It will probably allow your code to run faster in the not-too-distant future. If the compiler has knowledge of what type your variable is, it can optimize for it.

Use mypy and add to your setup.cfg:

[mypy]
# Ignore dependencies that do not have typings - otherwise this renders many libraries useless
ignore_missing_imports = True
# Don't allow the `Any` type
disallow_any_explicit = True
# Skip checking site packages for typing
no_site_packages = True
# Require type hints on all functions and methods
disallow_untyped_defs = True
# Disallows defining functions with incomplete type annotations.
disallow_incomplete_defs = True
# Type-checks the interior of functions without type annotations.
check_untyped_defs = True

Use a dependency manager for your installed libraries

Pip as a package installer has worked great for a while, but there are a few ingredients missing in pip as a package manager that allows you to create reproducible builds. For example: the default way of specifying which packages your project requires used to be in a requirements.txt file. However, there were a few shortcomings in this strategy. It was kinda hard to specify separate sources for your libraries (say, pypi.org or your private company-hosted python libraries), you had to keep track of what you installed along the way, and splitting dev-dependencies such as type and style checkers from production dependencies was a bit of a hassle. And then, you had to manage your virtual environments. Nowadays, pipenv can take care of all this stuff for you. I’d recommend pipenv for this, but you could also go for poetry.

Run your tests and checks in a continuous integration environment

When you have all the above parts in place, use some kind of continuous integration tool in order to make sure that all is OK once you check your code into a repository. If you don’t have anything in place, start with GitHub Workflows, it’s fantastic!

The advantages of having CI are many, but regarding team collaboration: it’s the transparency that it offers. Your code may run or check out on your machine, but it may not on your teammate’s. A “neutral ground” of CI allows you to view the result your tests and checks on a clean working environment. If it exits in error, you have a hyperlink to point at and ask your colleague if they can help resolving the issue. If your teammate is having trouble getting his checks to pass but the CI pipeline doesn’t, maybe it’s time to clean some stuff out on their end. In the end, it should be your CI pipeline that should give the final formal check on whether a revision of the product can get the go-ahead.

A Motorola 68000 (m68k) retro computing project

Mon, 10 May 2021 00:00:00 +0000

Last week I finished my first actually working retro computing project: a 1980’s Motorola 68000 educational board remake by Jeff Tranter. The original design is called a TS2, Jeff did a great job modernizing it, swapping hard to find chips with more easily obtainable ones and creating an open PCB design for it in KiCad.

I can connect to it using PuTTY:

It’s hard to actually see, but in the top right of the picture the terminal says:

TUTOR 1.3 >

I’m pretty proud that I have it up and running that far, it isn’t my first attempt at building retro computer hardware and it can be both awesome and pretty difficult at times. Debugging hardware with my scope is a skill that comes with its own learning curve.

I had a couple of earlier attempts. One was on a hybrid 68008/CPLD board by Steve “Big Mess O’Wires” Chamberlin. The design is pretty cool because it uses an easier to use Motorola 68008 which is code-compatible with the 68000 but has a smaller address and data bus, so less things to mess up. Also, it has enough RAM and ROM to run an early version 2 of the Linux kernel! But it proved too difficult for my humble soldering skills to get the CPLD SMD flat package soldered on the board. So now I just stick to through-hole soldering :)

This brought me to Jeff Tranter’s 68000 project. He created the PCB and component placement based on Alan Clements’ “Computer Systems Design”, on of the few books of which I own two editions (one for the included CD-ROM).

The TS2 educational board design is described in detail by Jeff Tranter here, but it’s peculiar in several aspects:

It uses no programmable chips anywhere except for the ROMs of course. That means that all the address decoding logic, power-on reset circuitry and data acknowledgement circuitry is implemented in TTL logic. The chip count is, expectedly, on the large size with 36 (!) chips on a pretty sizeable board. The upside is, of course, the cost: the TS2 board was supposedly a lot cheaper to build than Motorola’s own Educational Computer Board that went for a rather pricey $495
Despite the chip count, only 32kb of ROM and 32kb of RAM are available. This makes sense from an educational point of view: there’s no requirement for large software packages to run on this kind of hardware.
It has a “single-stepping” (or more aptly a single-bus-cycle-stepping) circuit that can be operated using physical switches.
There’s data acknowledgement speed selection options using jumpers 1 thru 4 to allow for slower, cheaper RAM or ROM chips. Jeff swapped ultraviolet-erasable 2764 ROM chips to electrically-erasable 28C64s, which are of course much easier to use, for example with ubiquitous and low-cost TL866II-type programmers.
The current board design has no keyboard input, no computer screen output and no board connectors other than two serial ports and the power connector. This means it’s pretty much impossible to attach board peripherals at the moment.

All in all, it is a very cool build, I contributed back some minor updates to the board and the software and I hope to do some more cool stuff with it, write some programs and flash it to the board. I don’t know whether it will fit in ROM, but porting a version 2 of minix to the TS2 would be awesome.

Jeff’s design featured on Hackaday.io in February 2017.

QGIS plugin development

Sat, 10 Apr 2021 00:00:00 +0000

TL;DR:

This isn’t a beginners’ guide to QGIS plugin development, there are other excellent resources out there (although some of them may be somewhat dated, as this post will some day be as well). This post discusses how to design integration tests for python plugin scripts that target QGIS version something like 3.18. They were tested on 3.18. If you’re targeting a newer version, things may not work as explained here.

My take on the approach is all open here

My Dockerfile is here, and the integration test is here. The test is run using this GitHub Actions spec.

Use this QGIS Docker image to run your integration tests
Derive your own Dockerfile from this image, copy your plugin and tests into the image
Install the plugin to /QGIS/build/output/python/plugins (possibly subject to future change)
Add the path /QGIS/build/output/python/plugins to the PYTHONPATH
Add an integration test script containing a def run_all() function. This function is expected to be present.
Inside the run_all() function, instantiate your plugin using the classFactory function. This is what QGIS would normally call as well to instantiate your plugin and it should be part of the integration test to verify it’s working correctly.
Put your (calls to) integration tests in this function.

If you use the standard lib unittest module and define a `class MyTestCase(unittest.TestCase), and execute it using something like

 def run_all() -> None:
     suite = unittest.TestLoader().loadTestsFromTestCase(MyTestCase)
     assert unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful()

If you want to write pytest tests, add a print('Ran OK') line in your run_all function, otherwise the integration script will not recognize the test as successful.
COPY over the integration test to the image
RUN the test using RUN xvfb-run qgis_testrunner.sh {insert_your_test_script_filename} without the .py file extension!
Sit back and observe all the black magic unfold.

QGIS is awesome

QGIS, as a (the?) premier GIS client software - is one of my favorite open source projects. Not just because I’m a geospatial developer/researcher but also because I have seen the fantastic development it has gone through over the years. It gained a ton of features and a lot more stability along the way. Not that there’s no rough edges, but rough edges are in any piece of software if you dig deep enough.

Although I have been a QGIS user for quite some time, although I never acquainted myself with all the nooks and crannies of the package. However I recently followed an (internal) masterclass on using the Atlas feature that lets you auto-generate templated fancy cartographic layouts of your data. Nowadays I’m more of a “what did I actually produce using this {script,query,export}?” user of QGIS, but I did extensive mapping and editing previously and I found it an extremely useful tool for the job. Particularly helpful for example is the setting where new polygon parts overlapping with already present objects can be automatically clipped (see tutorial part 15-17) , then select “Avoid overlap on active layer”:

For heavy digitisers, this option will save you hours of work and delivers clean maps with non-overlapping polygons.

But I digress.

QGIS plugin development

My Python skills have come to fruition only relatively recently over the past couple of years, and pure out of curiosity I started doing some research on what the current state of Python plugin development for QGIS is at the moment. I started with a simple goal: just to learn some stuff about QGIS plugins in Python and have some fun along the way ( mission: accomplished), but I also decided not to back down on the more complicated stuff. I created a plugin called Namari that allows you to do anomaly detection on geospatial data.

So, I took to the docs which recommends using the plugin builder to get started. I must say that this plugin builder is phenomenal to get started with creating your own plugins. It bootstraps an absolute ton of resources for you, but perhaps a little on the heavy side - it’s easy to get overwhelmed by the amount. I also created some level of discussion on the current state of this plugin builder .

Now, I’ve become something of a testing enthusiast over the years. I think it has to do a lot with both research reproducibility and product stability. Tests allow you to formally verify that something does what it is expected to do, which often makes me wonder why so little unit/integration testing is done in research.

It probably has a lot to do with the fact that writing good tests is very hard. I can tell, because I do a lot of it. But if it were only hard, I wouldn’t be doing it. Writing tests offers me a level of satisfaction that I cannot reach from writing working software alone. Testing doesn’t offer every guarantee: your software is stable only to the level of tests can verify and then it still doesn’t protect from misuse. But even more than some level of proven correctness, I truly believe testing can be fun, because of the main goals I set for myself: just to learn stuff.

And boy does testing stuff get you to learn stuff. I’d almost say that writing tests is maybe the single most effective way of learning about library APIs and the inner workings of your own code doodles. None more so than for writing integration tests for QGIS plugins. So here we go.

QGIS plugin integration testing

Doing unit testing on parts of the plugin is all fine and dandy, but I wanted to do a little bit more than just that. After all, a QGIS plugin often has some kind of interaction with the graphical user interface of QGIS. So, how do we test this interaction?

The answer is through integration testing with a working QGIS instance. Now this is where things get quite complex. QGIS offers a qgis python package installed in the site-packages of the Python version used by QGIS. But importing this in a stand-alone Python script doesn’t get you a fully working QGIS instance, it’s just a reference to a module that’s intended to run as an embedded Python process. So, this process needs to be tied to a running QGIS instance if you want to make use of its full range of capabilities.

The `iface` object

Central in the understanding of the connection between the Python qgis namespace and the QGIS app is the notion of the iface. This iface object can be accessed from a running QGIS Python console (just press Ctrl-Alt-P to start one):

This iface object gives you an instance of a qgis.gui.QgisInterface, which kind of is the bridge between a plugin and events in QGIS. This is why practically all QGIS Python plugins are instantiated by passing (or you could say dependency-injecting) this iface object into the plugin constructor.

But you’re getting none of this iface goodness just by unit testing. The plugin builder side-steps this problem by offering a stubbed QGIS interface object , but of course this doesn’t do much unless you re-create a lot of the functionality here yourself. But that is both tedious and unhelpful: we want to validate the workings of our plugin in a real QGIS environment, not in a stub. How are we going to check whether it will work in the wild otherwise?

The hunt for this elusive iface object is documented on several places: here is my own , but others have hit this as well here and here. After a lot of searching and trying, I managed to come up with a solution that will work almost everywhere.

Ingredient number one: a Dockerfile

For quite some time now, I have been using Docker as a very helpful development tool. Docker isn’t just handy for packaging and shipping, it’s also very versatile as part of a development toolchain. The reason is that it:

allows you to build on your own or others’ previous work, the whole standing on shoulders of giants (including your own :) thing
allows you to mess up you build environment inside an isolated environment without having to deal with the fallout of a broken development system. If your development environment inside Docker made a mess, then just rinse and repeat without any sweat
allows you to make reproducible builds much more easily. If you can get it to work on your local Docker env, whether you built on Linux, Mac or Windows, it will probably work on something like GitHub actions/Travis/Jenkins/TeamCity/Gitlab CI/CircleCI or whatever CI system you’re using. There’s no self-respecting CI system that doesn’t support Docker.
allows you to just remove the test image you built from your local machine and get rid of every last bit of toolchain you needed for your project and leave with a clean local system. I cannot tell you how many times I screwed up my machine setting up all kinds of weird build toolchains that started to get into version conflicts and having no clue on how to untangle things other than just to uninstall the whole lot (or worse, just flush my machine OS).

In this case I think it’s definitely worth using the official QGIS latest Docker image for the integration testing. It has all the assets you need to get started designing integration tests. You could use your own local QGIS installation but the trouble is of course that when you copy or remove the wrong files or to the wrong location, you end up with a broken QGIS (or worse). So, for development and experimentation, I find a Docker image to be ideal. So: start your own Dockerfile with

FROM qgis/qgis:release-3_18  
# Or whatever version you're targeting

You can even parameterize the release version, so you can test your plugin against multiple QGIS versions without having to install them:

ARG VERSION
FROM qgis/qgis:release-$VERSION  
# Or whatever version you're targeting

and then build using docker build -t my_plugin:newer --build-arg VERSION=3_18

Ingredient number two: a special integration test script

Second aspect of our integration test toolchain is a special python script that can be executed by a qgis_testrunner.sh script that is present in the QGIS Docker image by default. You can call your python script anything you want, but the minimal requirement is that:

it can be found by the qgis_testrunner.sh shell script,
that it has a run_all() function (having no parameters, returning nothing) and
that at some point it emits a Ran and OK through standard out.

The most simple test script that can be ran by the test runner inside the docker image is thus:

def run_all():
    print('Ran OK')

However, this test script is capable to do a lot more: it is able to instantiate our plugin with an actual iface interface to the QGIS application!

from qgis import utils
from namari import classFactory


def run_all() -> None:
    print('Hello QGIS!')

    plugins = utils.plugins
    assert utils.iface is not None, 'This test needs an actual iface interface to QGIS'
    plugins['namari'] = classFactory(utils.iface)
    print(plugins)

    print('Ran OK')

So, this script will not only verify that we have a real iface object instead of a stubbed or empty one, but it will also instantiate our plugin (called namari) to hook directly into the QGIS application. For comparison: you can import iface from qgis.utils at any time from any python script, but the iface object will be None unless you import it inside a running QGIS application - or from this qgis_testrunner.sh.

Now that we have the beginnings of our Python integration test script, we can call it using qgis_testrunner.sh.

We expand our Dockerfile by:

copying our plugin,
using pb_tool to deploy the plugin files to the QGIS plugins dir,
add the plugins dir to the PYTHONPATH and
call the test script

FROM qgis/qgis:release-3_18
# You should consider using pipenv for dependency management
RUN pip install pb_tool

WORKDIR plugin
# Copy all your plugin files to the plugin working directory
COPY . ./

# Put the rather long path to the standard plugins dir in a PLUGIN_DIR environment variable for re-use
ENV PLUGIN_DIR=/QGIS/build/output/python/plugins

# Deploy plugin to standard plugins dir using pb_tool
RUN pb_tool deploy --config_file pb_tool_docker.cfg --plugin_path $PLUGIN_DIR --no-confirm

# Run the integration tests by using the test script
ENV PYTHONPATH=$PYTHONPATH:/namari/test:$PLUGIN_DIR
RUN xvfb-run qgis_testrunner.sh integration_test

And qgis_testrunner.sh will run your integration tests!

SQL as programming language? Better not.

Sun, 19 Jul 2020 00:00:00 +0000

The past few months I have been given the opportunity to work with a prototype built in and around PostGreSQL. In this post, I intend to share some of the experiences with application programming in SQL and the accompanying procedural languages for use inside a database (such as pl/pgsql). I’d like to stress that the opinion expressed here in no way is pointed toward any individual. Instead, it’s about technology and how I feel towards its implementations.

TL;DR:

In short: use SQL for persistence: fetching data from, or writing to a database, but don’t use it for application logic.

It’s perfectly fine to prototype your application in any kind of programming language, including SQL. SQL has some extremely concise expressivity that can make a data transformation pipeline very compact. However, be wise and:

Provide every bit of information on why each SQL statement or clause is doing what it’s doing
Drop the prototype as soon as you start thinking about going to production. It will not be worth the effort to put a prototype into production. Don’t ever be tempted into thinking it will perform, be stable or even work in the first place: it is unlikely to work out in your (financial) benefit.
Use a generic programming language, such as Python, Java, JavaScript, C#, for your application logic and don’t stick any of it in SQL. Use any programming language that lets you:
- Write code in a concise and readable way. Long SQL queries tend to become utterly incomprehensible for anyone except the writer of the code. There’s no generally agreed upon code style and no linter for SQL that will help you format the code in a way that makes it compliant to a standard or easier to understand for fellow colleagues.
- Most importantly of all: write tests for your code. Testing application logic in SQL is hard and expensive because it has no built-in testing framework. Use a generic programming language instead for any application logic you need.
- Do proper dependency/package management to specify what your application logic needs to work in production. Install dependencies and packages that you trust and that are thoroughly tested. For many open source dependencies there are built-in testing pipelines in place where you can see how your dependencies are tested.

A very short introduction into the purposes of SQL

SQL is a language for interacting with a database. It is a set-based language that operates primarily on table-level. In this sense, the set that a SQL query operates on, is an entire set of records in one or more tables over which the SQL ‘loops’ automatically. Especially in querying different data tables, SQL is very good at integrating, filtering and summarizing data. It is then also very good at writing the results of this operation back into a new table. This is often referred to as an ETL process: an Extract (load data from tables), Transform (filter the data, summarize), and Load (write results into a new table).

The database here acts as a strategy for “persistence”: a way to store data that is not lost when a machine restarts. It is a popular method, because the way a database organizes lots of data is much more performant than the default persistence method, namely a file system. A file system generally is a tree-like system of folders that contain either subfolders and/or file. You may have a “Documents” folder, containing “Projects”. This “Projects” folder may have a “Git” folder, to designate project documents that are tracked in a version control system called Git. This “Git” folder then may then contain the subfolder “web_app_for_incredible_solutions” or some other project, etcetera. The file system therefore often expresses a tree of “things” that go from generic to specific, but this tends to gets swamped if you have a lot of fine-grained data.

Databases have a lot fewer levels than paths in your filesystem do. There is (such as for PostGreSQL databases) the Database level, the Schema level and then there are Tables which have Rows and Columns. This is the order of significance: you have a database for your department, with perhaps a schema for each customer (or you may have a database for each customer) or perhaps for each application or for a particular data set. Each schema will probably contain tables. Often, you name the table after a plural of the thing type it contains. A record in a Table often represents a real-world or imagined “thing”: an invoice, a car, a car brand, a building, a customer, etcetera. Each field will express some kind of property or attribute of the object in the record. These are the basis building blocks of a database.

This is why simple SQL queries often make a lot of sense, just by looking at them. SELECT customer_name FROM customers; will evidently get you all names of your customers from a set containing all customers for a given schema (the name of which is the default schema here).

Where the trouble starts

There’s trouble with SQL on many fronts that hamper production application development.

Readability

SQL as a human-readable language

The simplicity of SELECT {attribute} FROM {thing_type}; has earned SQL its reputation as an easy to read language. This is more than just a nice idea. Programming is primarily of use as a method for communication: from humans to other humans, that is. It gets read by programmers after you write it, so people after you need to know what and why your program does what it does. This is no different for SQL.

However, the readability of SQL quickly evaporates when the SQL query gets longer and more syntax is used. There is very little readable left in queries that

Use complicated and multiple join types such as FULL OUTER JOIN, or CROSS JOIN. Even after reading the tutorials, I don’t know whether a FULL JOIN will return something different than a FULL OUTER JOIN. If it’s optional and if it does return the same result set, then why is it even there? If the CROSS JOIN doesn’t do anything than just the cartesian product of tables, then why does it exist? What exactly is “crossy” about a cross join?
Mimic control flow, such as COALESCE. To “coalesce” means to merge or grow into one another. But that is not what this function does: it takes the first non-null value from a list of values passed to the function. What is wrong about having called this function just FIRST_NON_NULL()?
Use many, many nested functions. This is often the case, since you can’t assign to an intermediate or temporary variable. You can use WITH clauses to work around this, but this will generally reward the reader with an endless stream of WITH blocks that only vaguely resemble a procedural flow.

Instead, use a programming language to write out the application logic in an easy to understand way. I find Python to be ideal for this purpose. Yes, it’s possible to make an unreadable mess in any programming language, but Python enables you to write readable code unlike any other programming language does. It doesn’t COALESCE or CROSS JOIN and so shouldn’t you. Write readable code and douse with plenty of comments why you do it that particular way. Don’t us endless nesting of functions that SQL pushes you towards, but break up meaningful pieces of procedural code into functions and execute them one after another. Use classes if they improve the readability of your code: SQL can’t.

So, the supposed readability benefit for SQL is mostly voided once you get to more advanced SQL queries. It is a language with a lot of syntax, just as most other programming languages. Readability automatically brings us to perhaps the most difficult part of programming in general: complexity and testing.

Code style

Unlike other commonly used programming languages, SQL does not have a generally agreed upon code style. Generally, programming languages make heavy use of style guides to keep code legible. There is some level of debate on whether code style is really necessary, but most coders seem to agree that it is a good idea, specifically for situations where code is shared between people. JavaScript coders often use something like the Google style standard or the AirBnB standard, There’s PEP 8 for Python, Java also has a Google style guide and the list goes on. Every single one of these style guides comes with some kind of tool that helps you check for style errors: a linter. Usage of these linters is essential to keep track of all style errors in your code, because it’s generally impossible to keep track of all the different kinds of style rules.

There’s a few websites out there that propose SQL formatting, but it’s also clear that there is no SQL style formatter or checker - a linter. I worked with someone who used this site to format his SQL, but having to copy-paste your code between tools is no more than an ad-hock workaround, not a solution. The GitLab SQL style guide stresses this lack of linters, to where it becomes a “collective responsibility” to enforce it. The lack of SQL style linters is a little surprising, seeing the vast amount of programming being done in SQL. Generally, most programming languages are used in IDEs that provide style hints that prevent you from typing errors and typos, but I have little evidence so far that these are in any frequent use. This means that a SQL programmer can apply basically any kind of formatting or lack thereof to sql. For example, there’s no agreed upon standard for using CAPITALIZATION (a practice that seems quite old-fashioned to me, and rather shouty). But also, since SQL often makes extensive use of function call nesting, this is detrimental to the readability of longer SQL code blocks. This, to me, very much points to SQL as being a pseudo-programming language.

Complexity and testing

Once SQL queries start to get longer, an uncanny feeling starts to creep up on me. Why is there so much code needed to get some stuff from the database? Is the structure of the database that complex?

Often, however, I see that the SQL code is not only fetching things from the backend or writing data to it. Often, it applies some kind of transformation. It performs part of the application logic that the client code is supposed to do. I hear my colleagues explain: “It’s much easier this way.” Yes, it’s easier for the one writing the query because he/she designed it. There was no complicated testing procedure involved. There is a simple reason for this: SQL does not have a testing framework.

Please allow me let this sink in. SQL does not have a testing framework.

Now, let me explain why this is such a shocking statement to make. It means that SQL has no way of checking the outcome of a SQL statement in a basic validation strategy that comes with every single self-respecting programming language around. Even COBOL has a unit testing framework, for crying out loud. What makes SQL script writers think they don’t need it?

One of the most fundamental properties that separate production code from hacking, is tests. Without a testing framework, SQL encourages hacking around, rather than writing tested production code. Putting SQL application code into production without tests is simply nuts.

Imagine telling Java, or C#, or Python developers that all the testing frameworks they were used to working with, were removed and that they were no longer allowed to write tests in their beloved and preferred method. Imagine telling them that they would have to resort to a different language to express their tests in. That they would have to build their application logic, and then inject the functions into that testing language to see whether or not it produced the right outcome. Really.

But this is the reality for SQL. It’s not SQL in itself, of course, it shares this unfortunate trait with some other languages: there’s

bash (Bourne again shell) scripting: a language without a testing framework.
regex (regular expression) doesn’t come with a unit testing framework. This sounds quite obvious, but it also explains why I start frowning in the presence of any regular expression containing more than four characters. Much more than that, and they’re only readable for those who wrote them. Also: if you use them, they need to be tested in any language other than regex that does have a testing framework.

From the comments in the links it’s clear that it’s not impossible to test bash scripts, regular expressions and SQL. The point is, that considering that these languages do not have testing possibilities makes it much more likely that they don’t. There may be testing frameworks for bash, but the chance that you will come across anyone ever having used the proposed frameworks is extremely unlikely. The same goes for SQL. How many people have you met that employ a testing framework for SQL scripts?

Instead, for SQL, bash and strictly speaking regex, you need something called “integration tests” to validate their correct workings. Regex is often built into the programming language used for testing, so you’ll find few regular expressions ‘out in the wild’ without some kind of programming language to use them. But for bash and SQL, this is different. You need an external dependency to test them. This makes them integration tests: the tests need to integrate separate components or applications into a requirements or behavior test. For bash this isn’t much of a dependency: you need a Linux system. For SQL, you need an entire database, running. Probably on a different machine than the one you’re running your tests on. It’s a systems integration test!

It gets worse for SQL. Many SQL scripts are action queries that alter the state of the database, and often requiring schemas and tables to pull data from. This makes these queries expensive to test: not only do you need to test to validate the requirements of the output, you also have to insert test data as well. Since the actions in the SQL script affect the state of the database, you don’t want your test to contaminate the production databas. So, you need some kind of test database. If you don’t have any such thing, it is improbable that you’re going to test your queries. Once you’re done testing, you need to remove the test assets from the test database as well to restore it back to a clean state. This means you’re going to have to write functions to inject and restore database state, that you will probably have to test. Still feeling the desire to test queries, are you?

In short: please do test your queries in a test database, but keep application logic from SQL. Testing SELECT * FROM customers; is hardly necessary or complicated. This is no longer the case once you start moving application logic inside of your queries, because now you will have to write tests for them. Make sure you do. Saying “last month my query was fine,” is useless because last month doesn’t matter if your application crashes now. Having test means making sure that under any circumstance and at any time, you are able to validate the correct workings of your code. Doing some manual testing now and again and calling “But it worked on my machine!” is a recipe for disaster. Therefore: put as little application or ‘business’ logic in your queries as possible, because testing for meeting all the requirements is costly for SQL. Instead, write you application logic in a proper programming language: it’s much easier and therefore cheaper to write unit tests for ‘generic’ programming languages.

Dependency management

Every self-respecting programming language nowadays has a dependency management system. JavaScript has NPM, Java has Maven, C# has NuGet, and Python has Pipenv. No programming language is complete without some kind of package management system. SQL has none. You cannot express in a suite of SQL scripts, a SQL project if you like, what dependencies it should or shouldn’t require. That means that the code is possibly going to break in unknown places once (and at some time you will) upgrade your database.

Version-dependent code

The more usage of database extensions and procedural logic there is in your SQL, the higher the chance that your SQL will break. There are at least 3 current versions of PostGIS, the geospatial extension for PostGIS, with one major version difference, from major version 2 to 3. Multiply this by about 4 major PostGreSQL database versions and you have a combination of backend components for which your query may or may not work. There is a special legacy FAQ for the trouble that upgrading from major version 1 to 2 gave. There are many dependency pitfalls in PostGIS - consider the SFCGAL functions that require CGAL to be present at build-time. That means that if you upgrade your database but forget to install CGAL, part of your code is going to break.

Dependency management is of course a problem in any system, but the point is that programming languages tend to work with them very diligently. All major programming languages have options to specify what packages you want, versus the (sub)packages versions that your application requires to work. This requirement is often captured in a “lockfile” of some kind. For Python with Pipenv, this is Pipfile.lock. For JavaScript and NPM, this is package-lock.json. For Ruby and Gem, it’s Gemfile.lock. For C# and NuGet, it’s project.lock.json. The list goes on, but it’s clear that the pseudo-programming languages such as SQL, bash and regex don’t have these. They don’t have package managers at all, and you can argue that bash and regex don’t need them. This is clearly not the case for database extensions, so you’re on your own, especially when using database extensions.

The upshot of this is that you have no guarantees on whether code in SQL will keep working in production if you start upgrading databases or other database dependencies such as PostGreSQL extensions. Again, if you simply SELECT * FROM customers this is highly unlikely to pose a problem. But the more business and application logic you put into your SQL, and the more extensions you use, the greater the risk is going to be. In a worst-case scenario, you won’t be able to upgrade your database until you’ve rewritten every application you have in production.

Having said this, I have yet to encounter a SQL script that states in which version of what database it is supposed to work, and which extensions are required. Please move your application logic to a generic programming language ASAP.

Battle-tested dependencies

A second reason why most programming languages make heavy use of package management, is because it allows you to re-use code that has been tried and tested by many users before. Most package management systems gather some form or other of popularity statistics, allowing you to see how many users have successfully used a dependency.

Since SQL doesn’t have a package manager, it’s up to you to pick code examples from the web, wherever you may find them. This propagates a cut-and-paste mentality, for example from sources as Stack Overflow. Now let me be precise: my work wouldn’t be possible without Stack Overflow, it’s essential for finding clues on how to find solutions for coding problems that I find hard to solve. However, Stack Overflow isn’t a dependency management system. When you start copy-pasting accepting answers as accepted solutions for coding problems, it doesn’t allow you to skip writing tests for the part you copied. The code may have been for a problem like yours, but it’s probably contextual. To make sure that this context applies to your situation as well, you normally write a test to verify. Without a testing framework, SQL lures you into bad coding practices of copying accepted answers from forums at face value.

Bottom line: copy all you want from forums, but then write a test in language different from SQL to verify that the SQL solution applies for your specific problem and every requirement in encompasses.

Elliptical Fourier analysis

Fri, 12 Jul 2019 00:00:00 +0000

Recently, I’ve become quite interested in Fourier series for the purposes of geospatial shape analysis. In particular, I’ve spent quite some time getting to grips with an elliptical variant of the well-known Fourier transform family, a variant based on calculating the coefficients of ellipses. Using a predefined number of ellipses, any closed contour can be described using the parameters of these ellipses. Once created, the parameters can be used to reconstruct an approximation of the original shape.

I created a web demo to demonstrate its workings. The demo is primarily based on leaflet for the geospatial part, and tensorflow.js for the coefficient operations. That these libraries exist in free, open source form is truly marvellous. The demo is best viewed on a desktop environment.

How did we get here?

Fourier transformation is the deconstruction of a signal into its main constituent frequencies. Or in other words: mapping from (often) a time or space domain to a frequency domain, as in the following animation:

Animation of signal decomposition into main frequencies

Let’s start in the early 19th century, with Jean-Baptiste Joseph Fourier (1768-1830), the man who gave his name to the Fourier transform. Fourier contributed in his 1820’s work Théorie analytique de la chaleur the notion that any mapping function of a variable could be expressed as a Fourier series (this might be an infinite ): a series of waves of multiples (harmonics) of the variable itself.

Seven harmonics of a signal

Fourier series definition

This series was defined as N pairs of coefficients (a and b) for each ‘harmonic order’ n, using the trigonomical formulation:

\[\begin{aligned} a_{n} &=\frac{2}{P} \int_{P} f(x) \cdot \cos \left(2 \pi x \frac{n}{P}\right) d x \\ b_{n} &=\frac{2}{P} \int_{P} s(x) \cdot \sin \left(2 \pi x \frac{n}{P}\right) d x \end{aligned} \tag{1}\]

Where:

the harmonic order n is the number of sine/cosine pair integrals to fit to the signal;
$f(x)$ is some known function producing the signal, mapped to the y-axis. In practice we map this to the y-axis.
The variable P represents the interval (or P for period) of the signal - the length of the signal we have sampled.

There are a few other interesting things here to note:

Each harmonic order n adds one more level of detail to the analysis.
Each harmonic order n represents n passes over the period P
The Fourier series decomposition isn’t simply looking for all the frequencies in the signal: in practice it’s looking for frequencies that fit a combination of multiples of period P. That means that, for some signals, you have to fit a very large number of harmonics to the signal to get good signal characteristics.

Complex-valued formula

There is another, more common method for expressing the Fourier transform, using complex number notation. According to the definition from Wolfram Alpha, the Fourier transform is a generalized form of complex Fourier series. Euler’s formula states that:

\[e^{i x}=\cos x+i \sin x, \tag{2}\]

expressing the relationship between trigonometry and complex numbers. Thus, the sum of sine and cosine values can be expressed as a complex number with real part $x$ and an imaginary part $i$. Note the similarity of the cosine and sine parts of the formula with Fourier’s trigonometrical approach. This means that the Fourier transform for frequency $k$ can be expressed in the parts of a complex number:

\[F(k)=\int_{-\infty}^{\infty} f(x) e^{-2 \pi i k x} d x \tag{3}\]

source: Wolfram Alpha

which is also commonly written as

\[\mathcal{F}(\omega)=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} f(t) e^{-i \omega t} \mathrm{d} t \tag{4}\]

which is substituting x-axis value $x$ with time variable $t$, substituting and factoring out the $\frac{1}{\sqrt{2 \pi}}$. Also, whe have the angular frequency $\omega$ as $\omega=\frac{2 \pi}{T}=2 \pi f$.

Reconstruction from coefficients

The brilliant thing is that this decomposed signal allows inverse transormation as well, to a reconstruction of the signal itself:

\[s_{N}(x)=\frac{a_{0}}{2}+\sum_{n=1}^{N}\left(a_{n} \cos \left(\frac{2 \pi n x}{P}\right)+b_{n} \sin \left(\frac{2 \pi n x}{P}\right)\right), \tag{5}\]

using the trigonometrical notation. Which boils down to that the reconstruction is the sum of all the harmonic orders reprojected from circular parameters back into cartesian space.

Discrete Time Fourier Transforms

Fitting Fourier series to perfect mathematical functions is all fine, but in practice we need to fit this series to data that is sampled from an unknown signal function. Remember, from the previous section, that for the standard Fourier series computation, we need access to the full continuous signal mapping function $f(x)$. However, in many practical cases, we do not have access to the original function - it is usually a sampled signal produced in nature or through some unknown mechanism that we happen to sample.

This is where the Discrete Time Fourier transform comes in:

\[X_{2 \pi}(\omega)=\sum_{n=-\infty}^{\infty} x[n] e^{-i \omega n} \tag{6}\]

for frequency variable $\omega$ as the radians per sample, for each integer n. The advantage here is that we do not need a full integrand of the mapping function f and the other terms, but we can sum over the individual values of x over n harmonics.

Discrete Fourier Transform

The Discrete Time Fourier Transform is similar the Discrete Fourier Transform (DFT). The DFT “converts a finite sequence of equally-spaced samples of a function into a same-length sequence of equally-spaced samples of the discrete-time Fourier transform” (wikipedia):

\[\begin{aligned} X_{k} &=\sum_{n=0}^{N-1} x_{n} \cdot e^{-\frac{i 2 \pi}{N} k n} \\ &=\sum_{n=0}^{N-1} x_{n} \cdot[\cos (2 \pi k n / N)-i \cdot \sin (2 \pi k n / N)] \end{aligned} \tag{7}\]

This produces the k-th pair for N complex numbers, the values n are sampled at equal intervals. Here, we show both the complex-valued form (top) and trigonomical form (bottom) from wikipedia: Discrete Fourier Transform for clarity. So, the Discrete Time Fourier Transform is equal to the Discrete Fourier transform only if our samples are equally spaced.

Fast Fourier Transform

There was however one problem with the standard Fourier transform method: it was computationally rather expensive. It scaled quadratically with the number of harmonics. So, we jump forward in time, to the point where extensive machine computation of Fourier series was becoming quite desirable. So suppose you would want to monitor some nation’s nuclear detonation experiments using sensors outside of their geographical domain, you would have to deal with signals that are probably rife with background noise. There would be a lot of data, and in the 1960’s computer hardware and computation was expensive. Ideally, you would like on-site, real-time processing and possibly other compound words that require dashes.

So, in 1965 just after the cold-war pinnacle cuban missile crisis, James Cooley and John Tukey together published their groundbreaking and much-improved variant on Fourier transformation. They came up with a computationally more efficient means to arrive at the same coefficients as the standard Fourier transform:

$$ X_{k}=\sum_{n=0}^{N-1} x_{n} e^{-i 2 \pi k n / N} \tag{8} $$

for which they only needed four and a half pages of perfectly legible article to explain their innovation. Their work was so profound that it has been called (one of ) the most important numerical formula(s) (see top of wikipedia page).

From Fast to Elliptic

From Cooley and Tukey’s work sprung a great many variants of Fourier decomposition algorithms. One of them, the Discrete Cosine Transformation was used in the MP3, JPEG and MPEG standards, being used by millions upon millions of users. A big difference with the standard Fourier series analysis is that it uses only the cosine term ($b_n$) as seen in eq. 1.

From variants came other variants, amongst which the elliptical variant. The Elliptic Fourier series was published in 1982 by Frank Kuhl and Charles Giardina. They extended the principle of time domain to a point series domain in two-dimensional space. So, rather than having N samples in time, you have a sequence of N points of a polygon in two spatial directions. The advantage here is that these N points are guaranteed to form a continuous, stable signal since together these points form a “closed contour”, as Kuhl and Giardina put it. Now, things get a little complicated. The algorithms uses four coefficients, which makes sense because we have a signal in two dimensions, so there will be two coefficients for each of the two dimensions. So for the x-coordinates we have:

\[a_{n}=\frac{T}{2 n^{2} \pi^{2}} \sum_{\rho=1}^{K} \frac{\Delta x_{p}}{\Delta t_{p}}\left[\cos \frac{2 n \pi t_{p}}{T}-\cos \frac{2 n \pi t_{p-1}}{T}\right]\] \[b_{n}=\frac{T}{2 n^{2} \pi^{2}} \sum_{p=1}^{K} \frac{\Delta x_{p}}{\Delta t_{\rho}}\left[\sin \frac{2 n \pi t_{p}}{T}-\sin \frac{2 n \pi t_{p-1}}{T}\right] \tag{9}\]

and for the y-axis we have

\[c_{n}=\frac{T}{2 n^{2} \pi^{2}} \sum_{p=1}^{K} \frac{\Delta y_{p}}{\Delta t_{\rho}}\left[\cos \frac{2 n \pi t_{p}}{T}-\cos \frac{2 n \pi t_{p-1}}{T}\right]\] \[d_{n}=\frac{T}{2 n^{2} \pi^{2}} \sum_{p=1}^{K} \frac{\Delta y_{p}}{\Delta t_{p}}\left[\sin \frac{2 n \pi t_{p}}{T}-\sin \frac{2 n \pi t_{p-1}}{T}\right] \tag{10}\]

This extracts the set of ${a, b, c, d}$ coefficients (Kuhl & Giardina 1982, 239-240) for a series of N ellipses from the x- and y-deltas ($\Delta x_{p}$ and $\Delta y_{p}$) between each consecutive point p in the K points in the polygon.

Then, we also need the center-point (Kuhl & Giardina 1982, 240) for the main ellipse as $A_0$ and $C_0$:

\[\begin{array}{l}{A_{0}=\frac{1}{T} \sum_{p=1}^{K} \frac{\Delta x_{p}}{2 \Delta t_{p}}\left(t_{p}^{2}-t_{p-1}^{2}\right)+\xi_{p}\left(t_{p}-t_{p-1}\right)} \\ {C_{0}=\frac{1}{T} \sum_{p=1}^{K} \frac{\Delta y_{p}}{2 \Delta t_{p}}\left(t_{p}^{2}-t_{p-1}^{2}\right)+\delta_{p}\left(t_{p}-t_{p-1}\right)}\end{array} \tag{11}\]

where

\[\begin{array}{l}{\xi_{p}=\sum_{j=1}^{p-1} \Delta x_{j}-\frac{\Delta x_{p}}{\Delta t_{p}} \sum_{j=1}^{p-1} \Delta t_{j}} \\ {\delta_{p}=\sum_{j=1}^{p-1} \Delta y_{j}-\frac{\Delta y_{p}}{\Delta t_{p}} \sum_{j=1}^{p-1} \Delta t_{j}}\end{array} \tag{12}\]

and $\xi_{1}=\delta_{1}=0$

Like other Fourier series transforms, the method is invertible to create a reconstruction from the coefficients, as is shown in the demo at the top of the page.

The algorithm was implemented in Matlab at least as early as 1996 by Trier, Jain and Taxt, again in 2006 and again in 2016 before it came to Python in 2016 as well. From there, it was re-implemented in Wolfram and I re-implemented it in Python using the Pytorch framework, and in Javascript using Tensorflow.js in 2019. But I find it unlikely that there was no imlementation in either Fortran or C in the fourteen years between 1982 and 1996; it simply hasn’t persisted on the web (yet).

Myoware Muscle Sensor Project

Wed, 08 May 2019 00:00:00 +0000

Tizen web app development using node.js, NPM and WebPack

Sun, 05 May 2019 00:00:00 +0000

A lot of people at my work are showing up with smartwatches. I don’t expect the smartwatch having such an impact on daily life as the smartphone did for mobile phones, it has much more the feel of a gadget. To fidget with.

However, I took the step of buying one myself as well. I didn’t want to invest a lot, so I purchased a ‘second chance’ model (basically a returned item) under € 200, just to see what all the fuss is about. Are my colleagues just being silly, showing off oversized, clunky watches, or is there actually something useful to the smartwatch? I decided to find out for myself.

One feature I definitely required, is that the software platform on the device must be hackable. Preferably in an open source-y kind of manner. This ruled out any stuff from any manufacturer with an ecosystem that is too self-oriented or closed. This got rid of quite a few products. I mostly compared Android and Tizen platform stuff, since these appeared to gravitate to the more open ecosystem side. I read up on some ‘versus’ posts and reviews such as this one and settled for a Samsung Wear S3 Frontier, also because it was very affordable.

However

To get the development environment set up and getting a first application to run, is quite, quite the hassle. Beware, for example, that Tizen Studio (the version I’m running: v3.2) does not run with the latest version of Java, which is v.11 at the moment. You need Java 8. I’m not going to run through all the motions for getting Java 8 installed and available to your system at this point, but I will suffice to say that the %JAVA_HOME% environment variable needs to be set and that the %JAVA_HOME%/bin directory of your Java install needs to be in your path. Same probably goes for Linux environments: you will need a fully-set up Java installation, which on Linux is perhaps a little easier. Trick is that when you have different Java versions installed, you can use update-alternatives to select Java 8 on Linux. Prepare for a few days to get everything up and running.

Connecting the watch

The S3 Frontier doesn’t have WiFi, so you connect it to Tizen Studio through a truly convoluted way: through your phone. The Samsung Wear app on Android is allowed to connect over bluetooth to the watch, and the btconnect sub-app somehow tunnels the watch to the pc, but only if you have developer mode enabled on your phone. Really? Really. However, in trying to connect, I did not have any success. The watch wouldn’t show up. Until I read this pivotal piece of information: have you tried turning it off and on again?. That seemed to do the trick. Unless you lose the connection, and then you need to restart your watch again.

Also, I found that connecting the watch through Galaxy Watch Designer was the most stable way. You click the ‘Run on device’ button on the top right (also reached through shortcut F9) which allows you to scan for the S3 Frontier, which will take some more patience. All in all, it’s a pretty meagre development experience, but it works. That’s just getting it connected, though. You’ll have to persevere some more, if you have a non-WiFi watch. The workflow that seems to work in a repeatable fashion is like:

Connect the Android phone to your dev laptop. Make sure the watch is paired with the phone in use and that the phone is in developer mode.
Have the Samsung Wear app running on your phone
Reset the watch by powering down and up again
Open the Galaxy Watch Designer application on the dev laptop. Hit F9, which will open up a menu box.
Select the Android phone listed there by clicking on it.
This will open up the sdboverbt app on your phone. Connect the watch over bluetooth by hitting the ‘SELECT DEVICE FOR DEBUGGING’ button and selecting the watch item in the list.
The Galaxy Watch Designer application will appear to do nothing. Click the phone item in the Run on Device menu again. Click on “Scan devices”.
Develop a vague sense of “this isn’t working” and select the phone item again. Click “Scan devices again”.
The Galaxy Watch Designer application seems to get stuck for a split second, then shows a popup informing you that having the watch connected over bluetooth over a prolonged period can drain the battery on the watch.
If the above steps fail to produce a working connection, reiterate from 1.

You’re in!

You can now add the watch device in Tizen Studio using the ‘Remote devices’ in the Device manager (Alt-Shift-V). It should be easy to find now. The cool thing is that you can

Permission hurdles: certificates, features and priviledges

The second hurdle to overcome is that you need special Samsung-issued certificates to install your home-made apps on a Samsung watch. This step is a little less fickle than getting your watch connected through a phone, but it does require a few steps. The easiest method is probably through Tizen Studio. You need to install the Samsung certificate plugin through the package manager (Alt-Shift-P). Under “Extension IDE”, find the Samsung Wearable and Samsung Certificate plugins and install these. Then, follow the instructions to install the certificates. If you don’t, you will get a security error on trying to deploy your pretty app to your watch.

Once you have the certificates installed, you need to declare the features and privileges your app needs. For example, if your app needs to access stuff from around the web (and who doesn’t?), you need to edit the config.xml file of your project in Tizen Studio. Add the http://tizen.org/feature/network.internet feature and the http://tizen.org/privilege/internet privilege.

Setting these features and privileges isn’t enough by itself, though. The user needs to grant permissions to these privileges through JavaScript tizen.ppm.requestPermission() calls. Fortunately through this method, the application itself can ask the user for these privileges itself, allowing the user to opt in for a default setting/’remember this choice’ that prevents the application to ask permission every time the application starts. However, the watch will ask permission each time you re-deploy a newer version of your app, though.

Setting up the app and permissions

In order to ask the user for the required privileges, you need some code, which we will cover in some more detail. Since we are making a Tizen web app, this is done in JavaScript. The Tizen web apps a fairly recent JavaScript engine available. The command navigator.userAgent gives the following useful information: "Mozilla/5.0 (Linux; Tizen 4.0; SAMSUNG SM-R760) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.0 Mobile Safari/537.36" So although it isn’t exactly the most recent version (I have Chrome v.74 running at the moment, 56 was issued January 2017 so over 2 years ago), it has support for ES6 language features, including classes, Symbols and fat arrow functions. In a nutshell: every language feature that makes JavaScript into a great modern-age functional programming language.

Not so pretty is the fact that Tizen Studio has not yet figured out the existence of ES6. It still assumes ES5 syntax. I would recommend, therefore, that at this point, we drop Tizen Studio as a development tool, and to use it just as a build tool. Instead, we are building our app in Visual Studio Code and the node.js/npm/WebPack toolchain.

To be continued…

Auto-starting ten Amazon machine learning instances

Wed, 18 Apr 2018 00:00:00 +0000

UPDATE

This is part three on automating deep learning training sessions as software builds.

Disclaimer

Use any of this at your own risk. Automating machine bootup and shutdowns can be a tricky thing and I will not guarantee that the configuration below will always work.

TLDR;

Rent a cheap machine as a master and deploy this listening script to fire up a couple of Amazon machines to train your model.

Part one is here. Part two is here.

Firing up ten Amazon machines

Previously, I wrote on the wonders of using a build server to ‘build’ deep learning models as part of a continuous delivery pipeline of sorts. You could even go as far as publish the pre-trained and saved model as a build artifact, a ‘release’ of your exertions. We will look into that another time, but having a full build trail of a pre-trained model could be part of the answer to the machine learning reproducibility crisis!

Then, I wrote on why it is important to repeat the training results for a model to get a sense of the spread of the performance of your model. Many experiments in exact sciences hinge on repeated results. The Higgs boson was only confirmed after repeated measurements with a 5-sigma or one in 3.5 million chance the hypothesis would be false.

We’re not going to be that stringent, but I did go as far as to say that if you tweak your model to produce a 0.02 point increase in performance (say, accuracy) you can only claim victory if you can reproduce this gain a couple of times, because deep learning most often includes a level of randomness: in the selection of the batch and validation samples, the initialisation of the network. This is not a bad thing: we like our model to generalize as best as possible. We can’t select the data we’re going to throw at it in production either: it’s very probably going to be random.

In embracing this randomness and the spread that our model performance is going to exhibit, I plead to do repeated experiments on the same model setup using (at least) ten machines. These I rent from Amazon and, as these machines are costly and not needed all of the time, I’m going to automate the crap out of it. Here is what I did.

Listening

First, we need a machine that is always on. It is our master machine and it has the authority to fire up other machines if some change in our repository is detected. For this, we keep things as simple as possible, as simplicity is our friend. The more complexity we introduce, the more likely it is to break at some time.

I used a python script to set up a service. For this script, you will need a couple of things. Firstly, it is going to read a yaml file called instances.yml from the same directory the script is run. It is going to look like this:

ec2:
  - i-11111111
  - i-222222222
  - i-333333333
  - etcetera

It will contain the ec2 instance ids you are going to use to do your training sessions. These instance ids you can find in your Amazon EC2 console. You want to start with one machine learning instance to test the correct workings of your automation setup. Then, if everything is working to satisfaction, you can create a private image of your machine and create as many ‘clones’ from the first test machine image.

Secondly, the script is going to read a couple of environment variables:

SLACK_API_TOKEN: you can generate this here. For more information, see here. Although this part is not mandatory, I cannot recommend enough the importance of some kind of notification service to follow around the startup and shutdown of your machines. If something goes wrong, you should know. Slack does this brilliantly.
SLACK_CHANNEL: the channel name that you designate for messages on starting and stopping machines. I suggest something like #monitoring or something. You should create the channel first.
SECRET_TOKEN: this is the canonical environment variable name as recommended by GitHub for the webhook system. You need to set this webhook for the repository you want monitored. You do this by going to https://github.com/{your_username}/{repo_name}/settings/hooks and by clicking Add webhook: You generate the token using ruby -rsecurerandom -e 'puts SecureRandom.hex(20)' as suggested here

Docker compose

I made sure these environment variables were set, by Dockerizing the service with a Docker compose configuration in a docker-compose.yml file:

  ml_manager:
    build:
      .
    restart: unless-stopped
    volumes:
      - ./:/ml_manager
    ports:
      - "80:4000"
    environment:
      - SLACK_API_TOKEN=my_slack_api_token
      - SLACK_CHANNEL=#some_channel
      - SECRET_TOKEN=secret_github_commit_hook_token
      - AWS_REGION_NAME=eu-west-1
      - AWS_ACCESS_KEY_ID=aws_secret_key_id
      - AWS_SECRET_ACCESS_KEY=aws_secret_access_key

Which you can run on the listening or master server using docker-compose up -d Of course you need to set proper tokens for the placeholder text here. Also, you need a web server with a reverse proxy with the same route as set in your webhook configuration, pointing to your listening service, a system which I described earlier here. Make sure NOT to commit any of this in an open repository. The information will certainly be found and someone will start mining bitcoins, hacking or spambotting using YOUR account on YOUR expenses!

Dockerfile

You now of course need a simple Dockerfile:

FROM python:latest

COPY requirements.txt /requirements.txt
RUN pip3 install -r /requirements.txt
WORKDIR /ml_manager
CMD ["python3", "manager.py"]

Which comes with a requirements.txt file for the python dependencies:

boto3
slackClient
github_webhook
pyyaml

The python script

import http
import os
from datetime import datetime

import boto3
import yaml
from flask import Flask
from github_webhook import Webhook
from slackclient import SlackClient

SCRIPT_NAME = os.path.basename(__file__)
TIMESTAMP = str(datetime.now()).replace(':', '.')

# Get environment variables
slack_token = os.environ.get('SLACK_API_TOKEN')  # Slack is not required
if slack_token:
    channel = os.environ['SLACK_CHANNEL']  # but it requires a channel if set

secret = os.environ['SECRET_TOKEN']  # You really need to set a webhook secret!
region_name = os.environ['AWS_REGION_NAME']

# Initialize frameworks
ec2_client = boto3.client('ec2', region_name=region_name)
ec2_res = boto3.resource('ec2', region_name=region_name)
app = Flask(__name__)  # Standard Flask app
webhook = Webhook(app, endpoint='/postreceive', secret=secret)


# Slack notification function
def notify(signature, message):
    if slack_token:
        sc = SlackClient(slack_token)
        sc.api_call("chat.postMessage", channel=channel,
                    text="Script " + signature + " notification: " + str(message))


def start_instance(instance):
    running_instances = ec2_res.instances.filter(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    running_instances = [ri.id for ri in running_instances]

    if instance in running_instances:
        print(instance, 'already running')
    else:
        try:
            response = ec2_client.start_instances(InstanceIds=[instance])
            http_status = http.HTTPStatus(
                response['ResponseMetadata']['HTTPStatusCode']).name
            print('Start instance {}: {}'.format(instance, http_status))
            notify(SCRIPT_NAME, 
                   'started {} with response {}'.format(
                    instance, http_status))
        except Exception as e:
            this_instance = [i for i in ec2_res.instances.filter(
                Filters=[{'Name': 'instance-id', 'Values': [instance]}])][0]
            notify(SCRIPT_NAME, 
                   'Error starting instance {}, status {} with {}'.format(
                        instance, this_instance.state, e))


# Read list of instances
with open('instances.yml') as f:
    instances = yaml.load(f)


@app.route("/pushservice")
def status():
    return "The push service is running"


@webhook.hook(event_type='push')  # Defines a handler for the 'push' event
def on_push(data):
    for instance in instances['ec2']:
        start_instance(instance)


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=4000)

Now, we can automatically fire up as much Amazon machine learning instances as we like! Of course, these will run forever until our credit card runs out. So, next, we’re going to see how we can automatically shut these instances down once they start to run idle.

Rein van 't Veer

A Generative Life Drawing Model

On how AI can be of use to artists

Art models

Machine learning models

Models generating models

FAFA-VAE

Troubleshooting

The easier way to K8s Operator development using Argo Events

On why you don’t need special operator SDKs to develop K8s Operators

TL;DR (or the “management summary” if you like)

However

Intro: Kubernetes Operators

Why Operators exist

Your own operator

Operator development and operator frameworks

Argo Events as an operator framework

Let’s dig in

1. Install Argo Events

2. Install Argo Workflows

3. Install the Memcached custom resource definition

4. Create the service account and access rights

4. Install the EventSource

5. Install the Sensor

6. Install the configmap containing the operator implementation details

7. Install the workflow template to deploy Memcached resources

8. Let’s go!

A Quality Definition of Done for Python projects

Setting an internal standard for code quality for your Python software

Development and Definitions of Done: why code quality matters

Does this sound like a familiar scenario already?

A set of DoD quality standards

Document your project and agree on what it means to document

Review each other’s code in a source code management system

Write automated tests for your code

Use a style checker

Type-check your Python code

Use a dependency manager for your installed libraries

Run your tests and checks in a continuous integration environment

A Motorola 68000 (m68k) retro computing project

QGIS plugin development

TL;DR:

QGIS is awesome

QGIS plugin development

QGIS plugin integration testing

The iface object

Ingredient number one: a Dockerfile

Ingredient number two: a special integration test script

SQL as programming language? Better not.

TL;DR:

A very short introduction into the purposes of SQL

Where the trouble starts

Readability

SQL as a human-readable language

Code style

Complexity and testing

Dependency management

Version-dependent code

Battle-tested dependencies

Elliptical Fourier analysis

How did we get here?

Fourier series definition

Complex-valued formula

Reconstruction from coefficients

Discrete Time Fourier Transforms

Discrete Fourier Transform

Fast Fourier Transform

From Fast to Elliptic

Myoware Muscle Sensor Project

Tizen web app development using node.js, NPM and WebPack

However

Connecting the watch

Permission hurdles: certificates, features and priviledges

Setting up the app and permissions

Auto-starting ten Amazon machine learning instances

UPDATE

Disclaimer

TLDR;

Firing up ten Amazon machines

Listening

The `iface` object