Tag Archives: Software Development

Jekyll plugin to bundle zip archives

The website for my recently published C++ book, cpp.hasper.info (German), was made with the static site generator Jekyll. It contains additional information, such as the code of all projects inside the book, as well as the sample solution for the exercises at the end of each chapter.

The reason why I used a static site generator in the first place was that I had all the code files organized in a folder, equipped with a CMake file which made sure the projects compile and are statically analyzed (I used cppcheck and cpplint, albeit with a very reduced set of checks due to the nature of the code examples). In order to not destroy this automation by copy-pasting code into a CMS, the site had to be generated around the code files.

I also wanted to enable the download of all C++ files as zip archive on a per-chapter basis. Again – I did not want to manually create this archive, in case I had to change some code in the future. So I wrote a Jekyll plugin which bundles given files into a zip archive which then can be placed behind a download link.

How it is used:

Filenames as multiple parameters:

{% zip archiveToCreate.zip file1.txt file2.txt %}

Spaces in filenames:

{% zip archiveToCreate.zip file1.txt folder/file2.txt 'file with spaces.txt' %}

A variable to contain a list of files is also possible:

{% zip ziparchiveToCreate.zip {{ chapter_code_files }} %}

The plugin code:

The plugin can be found here: https://github.com/PhilLab/jekyll-zip-bundler

# frozen_string_literal: true

# Copyright 2021 by Philipp Hasper
# MIT License
# https://github.com/PhilLab/jekyll-zip-bundler

require 'jekyll'
require 'zip'
# ~ gem 'rubyzip', '~>2.3.0'

module Jekyll
  # Valid syntax:
  # {% zip archiveToCreate.zip file1.txt file2.txt %}
  # {% zip archiveToCreate.zip file1.txt folder/file2.txt 'file with spaces.txt' %}
  # {% zip {{ variableName }} file1.txt 'folder/file with spaces.txt' {{ otherVariableName }} %}
  # {% zip {{ variableName }} {{ VariableContainingAList }} %}
  class ZipBundlerTag < Liquid::Tag
    VARIABLE_SYNTAX = /[^{]*(\{\{\s*[\w\-.]+\s*(\|.*)?\}\}[^\s{}]*)/mx.freeze
    CACHE_FOLDER = '.jekyll-cache/zip_bundler/'

    def initialize(tag_name, markup, tokens)
      super
      # Split by spaces but only if the text following contains an even number of '
      # Based on https://stackoverflow.com/a/11566264
      # Extended to also not split between the curly brackets of Liquid
      # In addition, make sure the strings are stripped and not empty
      @files = markup.strip.split(/\s(?=(?:[^'}]|'[^']*'|{{[^}]*}})*$)/)
                     .map(&:strip)
                     .reject(&:empty?)
    end

    def render(context)
      # First file is the target zip archive path
      target, files = resolve_parameters(context)
      abort 'zip tag must be called with at least two files' if files.empty?

      zipfile_path = CACHE_FOLDER + target
      FileUtils.makedirs(File.dirname(zipfile_path))

      # Create the archive. Delete file, if it already exists
      File.delete(zipfile_path) if File.exist?(zipfile_path)
      Zip::File.open(zipfile_path, Zip::File::CREATE) do |zipfile|
        files.each do |file|
          # Two arguments:
          # - The name of the file as it will appear in the archive
          # - The original file, including the path to find it
          zipfile.add(File.basename(file), file)
        end
      end
      puts "Created archive #{zipfile_path}"

      # Add the archive to the site's static files
      site = context.registers[:site]
      site.static_files << Jekyll::StaticFile.new(site, "#{site.source}/#{CACHE_FOLDER}",
                                                  File.dirname(target),
                                                  File.basename(zipfile_path))
      # No rendered output
      ''
    end

    def resolve_parameters(context)
      # Resolve the given parameters to a file list
      target, files = @files.map do |file|
        next file unless file.match(VARIABLE_SYNTAX)

        # This is a variable. Look it up.
        context[file]
      end

      [target, files]
    end
  end
end

Liquid::Template.register_tag('zip', Jekyll::ZipBundlerTag)

C++ Schnelleinstieg

English summary: I have published a C++ beginners book. As it is in German, so is this post.

Auf 304 kompakten Seiten bekommen Sie einen fundierten Einstieg in die C++ Programmierung. Besonders hervorzuheben und für ein Einsteigerbuch ungewöhnlich ist die Einbindung von diversen Open-Source-Bibliotheken – professionell gemanaged mittels des Paketmanagers vcpkg. Im Rahmen des Buches erlernen Sie auch fortgeschrittene Fähigkeiten wie die Programmierung graphischer Oberflächen und das Arbeiten mit Web-APIs, wie der von Wikipedia. Das Buch legt wert auf einen verständlichen aber dennoch professionellen Programmierstil und gibt einen Ausblick über das Programmieren im professionellen Umfeld, sowie Anküpfungspunkte zum selbstständigen Erweitern der eigenen Kenntnisse.

Das Begleitmaterial sowie das Inhaltsverzeichnis und eine Leseprobe finden Sie auf https://cpp.hasper.info/

Beim Verlag kaufen: C++ Schnelleinstieg

Bei Amazon kaufen: C++ Schnelleinstieg

Natürlich freue ich mich sehr über eine Bewertung des Buches.

Buchcover C++ Schnelleinstieg, Philipp Hasper
ISBN: 9783747503225
  • Alle Grundlagen einfach erläutert
  • Objektorientierte Programmierung
  • Einsatz von Open-Source-Bibliotheken
  • Grafische Benutzungsoberflächen (GUI)
  • Internetanfragen und JSON-Parsing
  • Zeiger und virtuelle Methoden
  • Fehlersuche und Debugging
  • Moderner Programmierstil
  • Programmcode, Lösungen und Glossar zum Download

»Während viele andere C++-Lehrbücher ihr Heil in der Erklärung jedes noch so obskuren Details suchen, fokussiert sich Hasper auf alles Wichtige, das Entwicklerinnen und Entwickler bei der alltäglichen Arbeit brauchen. Haben sie bisher mit Sprachen wie JavaScript oder C# Erfahrung gesammelt, findet sich hier ein hilfreicher Wegweiser durch den Dschungel des modernen C++.«

(Heise Online, 09/2021)

»Didaktisch gut aufgebaut.«

(ekz Bibliotheksservice, 10/2021)

How to remove Unity Services from your project

I recently found myself in the situation that a Unity project which changed ownership, still linked to now non-existing Unity Services. It constantly gave me warning messages like:

Unable to access Unity services. Please log in or request membership to this project to use these services

These were only warnings, so nothing particularly dangerous, but still I wanted to get rid of them. As it took me multiple iterations to solve it and there are multiple unsolved or incomplete post about it, I wanted to share here how I finally managed to solve this problem:

  1. Navigate to the file <Your project folder>/ProjectSettings/ProjectSettings.asset and open it with a text editor
  2. In the section cloudServicesEnabled, set all to 0
  3. Set the cloudProjectId empty (i.e. a space after the colon)

Especially the space part took me a while to figure it out. The respective sections of the Unity project’s settings file now look similar to this:

...  
  vrEditorSettings:
    daydream:
      daydreamIconForeground: {fileID: 0}
      daydreamIconBackground: {fileID: 0}
  cloudServicesEnabled:
    UNet: 0
  luminIcon:
    m_Name: 
    m_ModelFolderPath: 
    m_PortalFolderPath: 
  luminCert:
    m_CertPath: 
    m_SignPackage: 1
  luminIsChannelApp: 0
  luminVersion:
    m_VersionCode: 1
    m_VersionName: 
  apiCompatibilityLevel: 6
  cloudProjectId: 
  framebufferDepthMemorylessMode: 0
...

Having these modifications in the project settings made the warning disappear, as the unused cloud services are now properly removed from the project. However, given that Unity sometimes changes their framework and configurations, this might get out-dated some day in the near future. If this doesn’t work anymore when you are trying it, please drop me a message, so I can retry and update the article. But so far, the warning has not popped up again in any of my Unity projects.

How we organize a small development team with minimal overhead in Gitlab

At ioxp we develop a novel way of sharing knowledge using Augmented Reality, AI and Computer Vision to change the way how industries operate. Our ecosystem consists of many different tools and technologies – the AI backend, Hololens apps , Android apps, a web-based editor and cloud services – so we are very dependent on an efficient way to organize our developers and their different, often overlapping sub-teams.

And after all we are a startup and engineers by heart – we have to keep the balance between a thorough designed process and the creative freedom needed to explore and create new things.

This post will concentrate on how we facilitate developer communication and team work using features of the DevOps tool Gitlab, factoring out all other important stages like product management, planning or sprint planning. It is an excerpt from our internal Handbook (naturally hosted as Wiki in Gitlab) and describes the process I have designed and implemented together with my team.

The development process is primarily guided by issues and their three most important attributes: labels, assignees and related merge requests. It uses as little as 10 different issue labels and is divided into six stages:

Continue reading

How we keep an eye on our internal infrastructure with Gitlab – automated!

At ioxp, Gitlab is our tool of choice to organize our development. We also use their Wiki tool for our employee handbook and this is where we also have a list of all our internal services to maintain an overview about what is deployed, why and where. It looks like this:

To a) check that every service listed there is reachable and thus b) checking that the list is up-to-date (at least in one direction), we added a special Gitlab CI job which curl’s them all. This is its definition inside our main project’s .gitlab-ci.yml:

The script

Internal IT Check:
  only:
    variables:
      - $CI_CUSTOM_ITCHECK
  needs: []
  variables:
    GIT_STRATEGY: none
    WIKI_PAGE: IT-Address-List.md
  before_script:
    - apk add --update --no-cache git curl
    - git --version
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/wiki/wiki.wiki.git
  script:
    # Scrape the URLs from the IT wiki page
    - cd wiki.wiki
    - |
      urls="$(cat $WIKI_PAGE | grep -Ev '~~.+~~' | grep -Eo '(http|https)://([-a-zA-Z0-9@:%_+.~#?&amp;/=]+)' | sort -u)"
      if [ -z "$urls" ]; then echo "No URLs found in Wiki"; exit 1; fi
    - echo "Testing URLs from the Wiki:" &amp;amp;&amp;amp; echo "$urls"
    # Call each URL and see if it is available. set +e is needed to not exit on subshell errors. set -e is to revert to old behavior.
    # Special case: As one of our services does not like curls, we have to set a user agent
    - set +e
    - failures=""
    - for i in $urls ; do result="$(curl -k -sSf -H 'User-Agent:Mozilla' $i 2>&amp;amp;1 | cut -d$'\n' -f 1)"; if [ $? -ne 0 ]; then failures="$failures\n$i - $result"; fi; done
    - set -e
    - if [ -z "$failures" ] ; then exit 0; fi
    # Errors happened
    - |
      errorText="ERROR: The following services listed in the Wiki are unreachable. Either they are down or the wiki page $WIKI_PAGE is out-dated:"
      echo -e "$errorText\033[0;31m$failures\033[0m"
      curl -X POST -H "Content-Type: application/json" --data "{ 'pretext': '${CI_JOB_NAME} failed: ${CI_JOB_URL}', 'text': '$errorText$failures', 'color': 'danger' }" ${CI_CUSTOM_SLACK_WEBHOOK_IT} 2>&amp;amp;1
      exit 1

Script Breakup

  1. Gitlab wikis are a git repository under the hood. You can clone them via <projectUrl>.wiki.git (The URL can also be retrieved by clicking on “Clone repository” at the upper right on the wiki main page
  2. The URLs are extracted by grepping the respective markdown page for http/https links: grep -Eo '(http|https)://([-a-zA-Z0-9@:%_+.~#?&/=]+)'
  3. To ignore deprecated links (strikethrough in the wiki), we exclude them before with grep -Ev '~~.+~~' and to filter out duplicates we use sort -u afterwards
  4. Each found URL is then called via curl:
    • -k disables the TLS certificate verification if you use self-signed certificates and haven’t installed your own root CA.
    • -H 'User-Agent:Mozilla' was needed because one of our services didn’t like to be curl’ed by a non-browser so we dress up as one.
  5. With set +e we prevent that a subshell error (i.e. the curl) makes the Gitlab job exit with a failure prematurely as we want a list of all errors, not just the first one.
  6. If any error is found, a post containing the list of unavailable services is sent to our IT Slack channel using their webhooks.

Gitlab Preconditions

To run this job in a fixed interval, we set up a scheduled pipeline, running every day at two in the afternoon. The schedule has the variable CI_CUSTOM_ITCHECK defined so the job’s only: selector fires (see in the script above). All other jobs of our pipeline are either defined to not run under schedules or not when the variable is defined (using the except: selector).

Gitlab pipelines only/except for WIP merge requests

At ioxp we automate our continuous integration using the awesome Gitlab project. However, their model of when testing jobs run on commits induced quite a heavy load on our testing servers. So we were very delighted when pipelines for merge requests were introduced in Gitlab 11.6.

We then configured our testing to run with

only:merge_requests

However, our development process (more on that in a later post) recommends:

If a developer starts working on an issue, they have to

1. (if not already done) assign them
2. Create a branch starting with their two-letter initials, like ph/purposeOfTheBranch,
3. Open a new WIP merge request mentioning the issue to mark that it is currently being worked on. As the Merge Request window even offers a small button to automatically append this prefix, the workflow is quite easy here.
4. Assign themselves to the merge request because they are currently responsible for it

We do this to have a nice overview about what is currently going on and to discuss about unfinished work. However, our only:merge_requests became useless now – as every implementation starts with a merge request, the pipelines are immediately started wasting CI time.

Luckily, variables expressions came to our rescue:

integrationtest:
  stage: integrationtest
  tags:
    - withBackend
  only:
    - merge_requests
  except:
    variables:
      - $CI_MERGE_REQUEST_TITLE =~ /^WIP:.*/

It is as easy as that. The expression filters the merge requests by their title and the job only runs when it does not start with the WIP prefix.

Update: Newer GitLab versions use the “Draft:” prefix instead of “WIP:” by default, so to support both notations, the script needs to make use of the OR operator like this:

  stage: integrationtest
  tags:
    - withBackend
  only:
    - merge_requests
  except:
    variables:
      - $CI_MERGE_REQUEST_TITLE =~ /^WIP:.*/ || $CI_MERGE_REQUEST_TITLE =~ /^Draft:.*/