Tag Archives: programming

Transliterating arbitrary text into Latin script

This post explores one of the capabilities of the PyICU library, namely its text transformation module. Specifically, we’ll look at the simplest use case: transliterating text into Latin script.

Say you are given a list of phrases, names, titles, whatever, in a script that you can’t read. You want to be able to differentiate them, but how, when they all look like random lines and curves? Well, let’s turn them into Latin characters!

>>> import icu
>>> tr = icu.Transliterator.createInstance('Any-Latin; Title').transliterate
>>> tr('Ἀριστοτέλης, Πλάτων, Σωκράτης')
'Aristotélēs, Plátōn, Sōkrátēs'

There we go. Even if you still can’t pronounce these names correctly, at least they’re hopefully easier to recognise because they are now in a script that you are more used to reading (unless you’re Greek, of course).

'Any-Latin; Title' means we want to transliterate from any script to Latin, then convert it to title case. If that’s too simple, the ICU documentation has the gory details of all the supported transforms.

Easy, no?


Do not rely on the output as pronunciation guide unless you know what you’re doing. For example, the Korean character 꽃 is transliterated by ICU as kkoch to keep it reversible, even though the word certainly does not sound like the gunmaker’s nor Kochie’s last names, and definitely not like the synonym for rooster (the modern romanisation, which matches closer to the correct pronunciation, is kkot).

The transliteration of Han characters (shared between Chinese, Japanese, and Korean) uses Chinese Pinyin, and thus may not resemble the Japanese and Korean romanisations at all. This makes the transliteration of many Japanese texts particularly awful.

>>> tr('日本国')  # "Nippon-koku" = Japan
'Rì Běn Guó'

Oops, that could start an Internet war. Use a different library if you are primarily dealing with Japanese text.

Another unfortunate thing with ICU is that there are still scripts that it doesn’t support at all. For example, it can’t transliterate to/from Javanese.

>>> tr('ꦫꦩꦏꦮꦸꦭꦲꦶꦁꦱ꧀ꦮꦂꦒ')

Maybe one day.


Using GitLab’s CI server

GitLab provides a continuous integration server, which is pretty nice for building, testing, and packaging your software and having all the UI integrated in GitLab. If you’re just using the free GitLab.com hosting, you get to utilise their Docker-based runner. (If your build process requires a non-Linux OS you’ll have to provide your own runner.)

Getting a basic build up and running is pretty simple. For example, here’s one job named test that only runs make check:

    - make check

If your test suite can measure code coverage, GitLab can also show it in the UI. At the moment this feature is rather rudimentary and requires you to go to the project settings and enter a regular expression to find the coverage amount in the build output.

The following is an example that works with coverage.py when you only have a single Python file. I haven’t tried it with multiple files; it may require a wrapper script that calculates the total coverage amount.

  image: python:3-alpine
    - pip install coverage
    - coverage run foo.py
    - coverage report -m

# Regex that matches the coverage amount:
# ^\S+\.py\s+\d+\s+\d+\s+(\d+\%)

A few lessons learnt from setting up test hooks for a small Python app:

  • There is no way to test changes to your build script without pushing a commit. And then the build results will stay in your project page forever with no way to clean up the irrelevant ones. You can run builds locally with gitlab-runner, e.g. gitlab-runner exec shell test to run the test job on the local shell (replace shell with docker to use Docker). (Thanks Evan Felix for the info about gitlab-runner.)
  • GitLab.com’s default Docker image (ruby:2.1 at the time of writing) is really fast to spin up, possibly because it’s cached. However, you should still explicitly name a Docker image in case the default changes.
  • Installing packages is slower than downloading a Docker image. It’s not worth going out of your way to use the default image if you then have to call apt-get. Try to find a pre-built Docker image that has all the packages you need instead. However, we’re talking about differences of about ten seconds, so just choose the image that is most convenient.
  • The ruby:2.1 image has Python 2 but not Python 3.
  • Docker’s Python repository lists a number of Alpine-based images. As you would expect, these are smaller and slightly faster to download than the other (Debian-based) images.
  • The coverage regex requires the percent sign to be escaped (\%).

CMake’s ugly programming language

I’ve just discovered Rosetta Code not long ago, and found it quite fun to browse around. It shows you the code for various programming tasks in different programming languages. While looking at the Quicksort page, I noticed that it didn’t have a CMake version, so I decided to try writing one.

function (quicksort array_var)
    set (array ${${array_var}})
    if ("${array}" STREQUAL "")
        return ()
    endif ()

    set (less)
    set (equal)
    set (greater)
    list (GET array 0 pivot)

    foreach (x ${array})
        if (x LESS pivot)
            list (APPEND less "${x}")
        elseif (x EQUAL pivot)
            list (APPEND equal "${x}")
        else ()
            list (APPEND greater "${x}")
        endif ()
    endforeach ()

    set (array)
    if (NOT less STREQUAL "")
        quicksort (less)
        list (APPEND array ${less})
    endif ()
    list (APPEND array ${equal})
    if (NOT greater STREQUAL "")
        quicksort (greater)
        list (APPEND array ${greater})
    endif ()
    set ("${array_var}" ${array} PARENT_SCOPE)
endfunction ()

set (a 4 65 2 -31 0 99 83 782 1)
quicksort (a)
message ("${a}")

I’ve worked with CMake for years, and I think it’s a good build system, but I really wish it had switched to a saner language. The CMake language is actually pretty simple and consistent at the syntax level: everything is in the form command(string), where the string syntax is slightly confusing but still rather understandable once you’ve figured out the quoting and variable expansion mechanisms. It’s how that string argument is used that can be messy, inconsistent, and ambiguous. Effectively, it’s as if every command had its own syntax.

Around 2008, there was an experiment to allow writing CMake scripts in Lua. The project never caught on and was abandoned. I think part of the reason was that the thread discussing it in Lua’s mailing list was single-handedly derailed into pointless bickering (which reminds me of the “poisonous people” talk).

CMake is stuck with a mediocre programming language for the foreseeable future. It’s not as bad as it sounds, though. The simplicity of the syntax has its advantages, and writing CMake buildfiles rarely gets frustrating. It would make a terrible general programming language, but as a build system scripting language it’s workable.

Win32 Python: Getting all window titles

This post shows how you can retrieve all window titles in Microsoft Windows using Python’s ctypes module. Moreover, it also acts as a ctypes tutorial, showing how to create and use callback functions.

The following is the full code. Keep reading if you want to understand how it works. (Note: If you are reading this as a ctypes tutorial and are having trouble following the explanation, you may want to go through my previous tutorial first.)

import ctypes

EnumWindows = ctypes.windll.user32.EnumWindows
EnumWindowsProc = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int))
GetWindowText = ctypes.windll.user32.GetWindowTextW
GetWindowTextLength = ctypes.windll.user32.GetWindowTextLengthW
IsWindowVisible = ctypes.windll.user32.IsWindowVisible

titles = []
def foreach_window(hwnd, lParam):
	if IsWindowVisible(hwnd):
		length = GetWindowTextLength(hwnd)
		buff = ctypes.create_unicode_buffer(length + 1)
		GetWindowText(hwnd, buff, length + 1)
	return True
EnumWindows(EnumWindowsProc(foreach_window), 0)


Continue reading

If you get a PUSHL-related error…

If you’re compiling a piece of code and getting an error message saying something about PUSHL or “invalid suffix for PUSH”, it means you’re feeding x86 assembly code to an x86_64 assembler.

Possible causes:

  • The code you’re trying to compile really does contain x86 assembly, perhaps inlined from C, but you’re targeting x86_64.
  • Your toolchain somehow contains a mix of x86 compiler and x86_64 assembler. Try cleaning up your PATH.

strxfrm in Vala

Here, let me save you some time: string.collate_key.

I spent hours trying to get Posix.strxfrm to work and still failed. It was actually so much easier to write a simple strxfrm wrapper in C and use it as an external function. While admiring my work on this wrapper, I found GLib’s g_utf8_collate_key, which in Vala translates to the aforementioned string.collate_key.

By the way, I’ve just realised that strxfrm stands for string transform.

Pango: Determine if a font is monospaced

If you have a GtkFontButton, finding out whether the chosen font is monospaced is quite a complicated process. Here is a complete walk-through.

(By the way, I will be using PyGTK’s Pango documentation because the C version is a mess.)

FontButton.get_font_name returns the font family (a.k.a. “font name”), style, and size; for example, “Liberation Serif Italic 14”. The first thing we need to do is pick just the family name. We do this by going through a PangoFontDescription.

desc_str = font_button.get_font_name()  # Liberation Serif Italic 14
desc = pango.FontDescription(desc_str)
family_name = desc.get_family()  # Liberation Serif

Next, check whether the font family describes a monospaced font. Here is where it gets dodgy. We need an arbitrary PangoContext, which can be obtained from a GtkWidget using Widget.get_pango_context. We then list all available font families and find the one with the appropriate name. Call FontFamily.is_monospace to finish the job.

(By the way, this is also a good place to show off Python’s for-else construct.)

context = widget.get_pango_context()  # widget can be any GtkWidget.
for family in context.list_families():
	if family.get_name() == family_name:
else:  # Should not happen.
	assert False
family.is_monospace()  # False -- Liberation Serif is proportional.