Lachlan Phillips

Grengle #1: generating code-switching audio with Facebook MMS

Lachlan Phillips — Sat, 12 Aug 2023 07:30:53 GMT

I previously mentioned my attempts to learn Greek, and I haven't yet talked about my experiments with Grozer (and I won't right now, but they involve flash cards, clozes, and code-switching text). The big problem I've had with flash cards and other techniques is that they require my full attention, and take time out of the day. The result is that I eventually just drop them when something more important (or interesting) comes along.

So I've been wondering for a while whether there was a way to generate audio and make it comprehensible. I have a long commute to work every day, and I tend to use that for listening to the Bible in English. Is there a way to instead listen to the Bible in English+Greek; a code-switching audiobook that over time becomes more and more Greek?

Grengle is my set of experiments to attempt that.

The idea is to have a list of words I "know", and a handful of words that I'm "learning". Then as I listen to a passage, the words I "know" or am "learning" will be played in Greek, and the rest in English. Over time, it will automatically move the "learning" words into the "known" category, and pick new words to learn.

The first thing I tried was to get espeak to read Greek and English for me. I've been using espeak to read chapter headings for the Bible for a while now, and though it sounds like a robot, I can understand it. Greek, on the other hand, was terrible. Robotic Greek was beyond me, and so this experiment reached a dead end.

Recently, Facebook AI research put out a set of models that could (amongst other things) do text-to-speech for over a thousand languages. And Greek was one on the list. I finally got around to giving it a go, and once I got it installed, it generated some audio that at first seemed promising.

Then I plugged it into the Grengle interleaver engine, which slices a text into short phrases like the following:

GRK Καί λέγει αυτοῖς. ουκ οίδατε τήν
ENG parable
GRK ταύτην, καί πῶς πάσας τάς
ENG parables will you understand?
GRK ο
ENG sowing
...

The engine then used the model to render a bunch of audio files meant to be glued together.

The result was disappointing. There were a few dealbreaking issues:

Most importantly, the Greek speech was way too fast for me.
The Greek audio seemed to miss words occasionally, eg. a lone "ό" like the one above (which, yes, wouldn't normally occur in speech).
The English audio was rather muffled.

Thus, this second experiment failed.

I have a few ideas of how else I could get an interleaved audio system working:

Generate the text and record it myself. Which, since the words that should be played in Greek vs English is constantly changing, defeats the purpose of getting a model to read it for me.
Attempt to train a model myself (possibly a model that can handle both languages at once). I've never done this before.
Give up on the code-switching idea. Chop up some existing recordings into verse snippets, and play each one twice as English then Greek (or vice versa). However I suspect this will be too large a gap to get good Comprehensible Input at the word level.

Learning to read the Greek Bible

Lachlan Phillips — Sat, 04 Mar 2023 12:05:58 GMT

I want to read the Bible in the original Greek. To be specific, I'd like to be able to sit down and read the New Testament (or LXX) as fluently as I can English. For that, I need to learn Koine Greek.

The typical skill people are taught in ancient language classes is the ability to parse and "decode" those languages (but this is slowly changing). You learn a bunch of stuff about the grammar, and memorise a bunch of vocabulary, and then you're able to pick your way through a text.

This is reasonable given the constraints of those courses. It takes a long time and a lot of effort to really learn a language to fluency — something people eg in seminary don't always want/need to invest. In those cases having the skills to pick through a text and judge between more skilled scholars is enough, and useful.

But I'm a guy with free time, and I'd like to get beyond that stage, working my way asymptotically towards the point of fluency. Also I don't feel like memorising a bunch of vocab and conjugation tables!

So what I need is to internalise the language. Learn it in a similar way people learn living languages today.

There are multiple different paths you can take. Some people go to immersion colleges like Polis, or take online immersion courses like Seumas Macdonald's. While I might eventually do one of these, I want to start off on my own first, and at minimal cost.

The heart of language is Sound associated with Meaning. Even when we read silently, we usually subvocalise the written words into a stream of sound that we interpret as speech. And so the skills of knowing a language are to listen and speak: be able to interpret Sound into Meaning, and then (optionally in my case) be able to encode Meaning in Sound.

Apparently the best way to train you brain to unconsciouly do this is to get Comprehensible Input. This is any kind of input (words written or spoken) that you can understand "somehow", whether directly because you know them, or by some means of a side-channel — you can infer the meaning using something else. Getting lots of this kind of input — and it really needs to be a LOT of it — will over time will train your brain to automatically interpret the Sound.

So that's what I'm trying to do. I've been at this for a few years on and off, using various approaches. None of them have stuck very long, though some for longer than others. My current stab at this led me to build the application Grozer, and take it through multiple generations as my learning theories changed and new ideas came to me.

Next up, I'll talk more about my experiments with Grozer.

An afternoon build: Unravelled

Lachlan Phillips — Sat, 31 Dec 2022 04:28:14 GMT

This morning's digest email

I use Raindrop.io to collect links at work to read later. Every now and then I'd go in to the Inbox and read some of those things. But it's outside my normal routine, and for a while now the links have been accumulating.

What I wanted was a service that'd email me a few of the links every week, as if it was a newsletter.

A few days ago, in my holidays, I finally got around to building it, and came up with Unravelled — a script that takes the latest 8 links from a Raindrop.io collection, sends me a newsletter email, and removes them from the collection.

This was the fastest project I've ever built from start to finish. I made a point to start with all the boring deployment bits:

writing a Dockerfile
building the Docker container in a GitLab CI script
getting Podman to run it via a SystemD timer

I hadn't used SystemD to actually run a container before (previously only to call start/stop), and it worked really well.

Once I had my script running, getting it to actually do what it needed to do was straightforward: calling out to the Raindrop api a few times and generating an email body. Sending emails isn't intuitive in Python, but at least it's short.

And now it's running. After a couple hiccoughs with podman, it's now reliably emailing me every week.

Memoir of Rev. Robert Murray M'Cheyne

Lachlan Phillips — Thu, 05 Nov 2020 09:42:11 GMT

I've been privately recording audiobooks for myself for a year or so now, and I just finished a new one: the Memoir section from Andrew Bonar's Memoir and Remains of Robert Murray M'Cheyne.

Since the text is public domain, this was a book I could upload.

It's not perfect; I'm an amateur, read too fast, and couldn't edit out all the mistakes. But it's good enough for me to listen to, and hopefully someone else will find it useful.

Programming language experiments

Lachlan Phillips — Wed, 18 Jan 2017 10:43:14 GMT

This details a project called language_design that I started September 2015 and worked on sporadically ever since. It was written in Scala.

The initial aim was to build a language that mixed aspects of Scala and Erlang. It would be strongly & statically typed, have actors, and be roughly functional. Functional in the sense that immutability was encouraged and functions were used everywhere, but purity wasn't a big concern.

It had a feature where, given a function fun lines(reader: Reader): List[String] in scope and an r: Reader, you could call r.lines(). The point was to feel like objects (like Java), even the language didn't have any. That was cool.

Actors would be typed, and could implement interfaces. Actors could only be sent messages it could handle. Example: an actor could implement the Sink[T] interface, handle message Next(item: Int), then be used wherever a actor Sink[T] was needed.

Here's an example of desired syntax from the first commit to the project:

module me.lachlanap.file_average {
  process Main {
    handle Run(args: List[String]) = {
      val out = \Sink {
        handle Next(item: Int) = {
          println("Average: " + item)
        }
      };

      val accumulator = new Accumulator(out);
      val filename = args.get(0);

      val reader = new Reader(filename, accumulator);
      reader.watch(error =>
        println("Error reading file: " + error);
        crash(error);
      );

      reader ! Execute;
      // desugared to: reader.!(reader.T.Execute);
      //         then: !(reader, reader.T.Execute);

      // TODO: Garbage collect dropped processes?
    }
  }

  process Reader(filename: String, out: CloseableSink) {
    handle Execute = {
      use flat lang.io;
      val reader = openReader(filename, charset.UTF_8)

      with(reader, reader => {
        reader.lines()
              .map(line => line.parseInt())
              .foreach(parse => parsed match {
          case None => crash("Failed to parse " + line);
          case Some(v) => out ! Next(v);
        });
      });

      out ! Close;
      finish();
    }
  }

  process Accumulator(out: Sink[Int]): CloseableSink[Int] {
    var total = 0;
    var count = 0;

    handle Next(item: Int) = {
      total += item;
      count++;
    }

    handle Close = {
      out ! Next(total / count);
      finish();
    }
  }
}

I built my own set of combinator parsers to parse this... and quickly replaced them with FastParse. I managed to parse most of the language (although operator precendence is not a thing).

I didn't get very far with that language. I never got to generics. Or actors. Or data types beyond Strings and Ints.

The language today is very different.

module lang {
  data Bool = True | False
  data Unit = Unit

  crash :: String => String
   = message => message
}

module test.i2s {
  main :: () => String
   = () => "value: " + intToString(factorial(input))

  input :: Int = 10

  fib :: Int => Int
   = in =>
    ifeq_int(in, 0,
      () => 1,
      () => {
        ifeq_int(in, 1,
          () => 1,
          () => {
            fib(in - 1) + fib(in - 2)
          })
      })

  intToString :: Int => String
   = in =>
    if0_str(in,
      () => "0",
      () => {
        div10 = in / 10
        digit = in - (div10 * 10)
        digitStr = digitToString(digit)

        if0_str(div10,
          () => digitStr,
          () => intToString(div10) + digitStr)
      })


  digitToString :: Int => String
   = in => match(in) {
    case 0 = "0"
    case 1 = "1"
    case 2 = "2"
    case 3 = "3"
    case 4 = "4"
    case 5 = "5"
    case 6 = "6"
    case 7 = "7"
    case 8 = "8"
    case 9 = "9"
  }


  if0_str :: (Int, () => String, () => String) => String
   = (test, true, false) => ifeq_str(test, 0, true, false)

  ifeq_str :: (Int, Int, () => String, () => String) => String
   = (test, expected, true, false) => {
    match(test - expected) {
      case 0 = true()
      case _ = false()
    }
  }

  if0_int :: (Int, () => Int, () => Int) => Int
   = (test, true, false) => ifeq_int(test, 0, true, false)

  ifeq_int :: (Int, Int, () => Int, () => Int) => Int
   = (test, expected, true, false) => {
    match(test - expected) {
      case 0 = true()
      case _ = false()
    }
  }

  factorial :: Int => Int
   = {
       go = in =>
              ifeq_int(in, 1,
                 () => 1,
                 () => {
                   in * go(in - 1)
                 })
       go
     }
}

This is the language today. It's purely functional. Functions are defined exclusively with lambdas. It has no generics, hence ifeq_str and ifeq_int. There is no integer-to-string function, so I implement my own --- and the only comparison operator is the match expression --- booleans aren't a thing yet. The only way to get output is to return the result from main.

But it actually works. The example above prints:

result :: lang.String = "value: 3628800"

How it works

The source code above is parsed into:

module lang {
  data Bool = True() | False()
  data Unit = Unit()

  crash :: (String) => String
   = (message) => message
}
module test.i2s {
  main :: () => String
   = () => + "value: " intToString(factorial(input))

  input :: Int
   = 10

  fib :: (Int) => Int
   = (in) => ifeq_int(in, 0, () => 1, () => {
    ifeq_int(in, 1, () => 1, () => {
      + fib(- in 1) fib(- in 2)
    })
  })

  intToString :: (Int) => String
   = (in) => if0_str(in, () => "0", () => {
    div10 = / in 10
    digit = - in * div10 10
    digitStr = digitToString(digit)
    if0_str(div10, () => digitStr, () => + intToString(div10) digitStr)
  })

  digitToString :: (Int) => String
   = (in) => match(in) {
    case 0 = "0"
    case 1 = "1"
    case 2 = "2"
    case 3 = "3"
    case 4 = "4"
    case 5 = "5"
    case 6 = "6"
    case 7 = "7"
    case 8 = "8"
    case 9 = "9"
  }

  if0_str :: (Int, () => String, () => String) => String
   = (test, true, false) => ifeq_str(test, 0, true, false)

  ifeq_str :: (Int, Int, () => String, () => String) => String
   = (test, expected, true, false) => {
    match(- test expected) {
      case 0 = true()
      case _ = false()
    }
  }

  if0_int :: (Int, () => Int, () => Int) => Int
   = (test, true, false) => ifeq_int(test, 0, true, false)

  ifeq_int :: (Int, Int, () => Int, () => Int) => Int
   = (test, expected, true, false) => {
    match(- test expected) {
      case 0 = true()
      case _ = false()
    }
  }

  factorial :: (Int) => Int
   = {
    go = (in) => ifeq_int(in, 1, () => 1, () => {
      * in go(- in 1)
    })

    go
  }
}

Not much difference. Whitespace is canonicalised, and binary operations are printed in prefix notation.

The parsed AST is translated into an Intermediate Representation.

{
  lang.crash :: (lang.String) -> lang.String
   = (message_$1 :: lang.String) => message_$1

  test.i2s.main :: () -> lang.String
   = () => "value: " + test.i2s.intToString(test.i2s.factorial(test.i2s.input))

  test.i2s.input :: lang.Int
   = 10

  test.i2s.fib :: (lang.Int) -> lang.Int
   = (in_$2 :: lang.Int) => test.i2s.ifeq_int(in_$2, 0, () => 1, () => test.i2s.ifeq_int(in_$2, 1, () => 1, () => test.i2s.fib(in_$2 - 1) + test.i2s.fib(in_$2 - 2)))

  test.i2s.intToString :: (lang.Int) -> lang.String
   = (in_$5 :: lang.Int) =>
    test.i2s.if0_str(in_$5, () => "0", () =>
      let
        div10_$6 = in_$5 / 10
        digit_$7 = in_$5 - div10_$6 * 10
        digitStr_$8 = test.i2s.digitToString(digit_$7)
      in
        test.i2s.if0_str(div10_$6, () => digitStr_$8, () => test.i2s.intToString(div10_$6) + digitStr_$8))

  test.i2s.digitToString :: (lang.Int) -> lang.String
   = (in_$10 :: lang.Int) =>
    match(in_$10) {
      case 0 = "0"
      case 1 = "1"
      case 2 = "2"
      case 3 = "3"
      case 4 = "4"
      case 5 = "5"
      case 6 = "6"
      case 7 = "7"
      case 8 = "8"
      case 9 = "9"
    }

  test.i2s.if0_str :: (lang.Int, () -> lang.String, () -> lang.String) -> lang.String
   = (test_$11 :: lang.Int, true_$12 :: () -> lang.String, false_$13 :: () -> lang.String) => test.i2s.ifeq_str(test_$11, 0, true_$12, false_$13)

  test.i2s.ifeq_str :: (lang.Int, lang.Int, () -> lang.String, () -> lang.String) -> lang.String
   = (test_$14 :: lang.Int, expected_$15 :: lang.Int, true_$16 :: () -> lang.String, false_$17 :: () -> lang.String) =>
    match(test_$14 - expected_$15) {
      case 0 = true_$16()
      case _ = false_$17()
    }

  test.i2s.if0_int :: (lang.Int, () -> lang.Int, () -> lang.Int) -> lang.Int
   = (test_$19 :: lang.Int, true_$20 :: () -> lang.Int, false_$21 :: () -> lang.Int) => test.i2s.ifeq_int(test_$19, 0, true_$20, false_$21)

  test.i2s.ifeq_int :: (lang.Int, lang.Int, () -> lang.Int, () -> lang.Int) -> lang.Int
   = (test_$22 :: lang.Int, expected_$23 :: lang.Int, true_$24 :: () -> lang.Int, false_$25 :: () -> lang.Int) =>
    match(test_$22 - expected_$23) {
      case 0 = true_$24()
      case _ = false_$25()
    }

  test.i2s.factorial :: (?) -> ?
   = letrec
    go_$27 = (in_$28 :: lang.Int) => test.i2s.ifeq_int(in_$28, 1, () => 1, () => in_$28 * go_$27(in_$28 - 1))
  in
    go_$27
}

Now there's some difference. The translator qualifies all identifiers and assigns types to each expression. All the local bindings are addressed by a numeric id, though they still have a label for readability's sake. All the code blocks {x = ...} have been replaced with let/letrec expressions. They're the same thing, except a let expression is guaranteed to return something.

Next comes the optimiser. The optimiser was my favourite part, and I spent hours writing passes that tried to inline and simplify. It's the most magical and least understood part of the system.

This is the list of passes today:

val RepeatedPasses: List[OptimiserPass] = List(
  new InlineGlobalBindingsPass,
  new RemoveDeadGlobalBindingsPass,
  new OptimiseLocalAliasesPass,
  new InlineImmediateFunctionsPass,
  new InlineLocalBindingsPass,
  new SimplifyUselessArithmeticPass,
  new InlineImmediateFunctionsPass,
  new ApplyKnownMatchesPass,
  new RemoveDeadLocalCodePass,
  new ExplodeComplexOperationsPass,
  new FlattenBadlyNestedLetsPass
)

InlineGlobalBindings: for a given top-level binding, inline all referenced global bindings (except itself) as local declarations. Several iterations of this results in everything ending up in main.
RemoveDeadGlobalBindings: removes any top-level binding not referenced by main.
OptimiseLocalAliases: replaces things like let x = y in x + x (where y is a variable, not an arbitrary expression) with let x = y in y + y (ie x is now unused). It's important for function inlining (actually it's probably redundant considering InlineLocalBindings).
InlineImmediateFunctions: replaces things like (\x => x + x)(y) with let x = y in x + x. Depends on InlineLocalBindings to finish the job.
InlineLocalBindings: replaces things like let x = 4 + 5 in x + x with (4 + 5) + (4 + 5).
SimplifyUselessArithmetic: replaces things like x + 0 with x.
ApplyKnownMatches: replaces match(2) { case 2 => x } with x. Tries to simplify a match expression when the input is known at compile-time.
RemoveDeadLocalCode: removes local bindings that aren't referenced.
ExplodeComplexOperations: here for the awesomeness of generating heaps of temporary variables. It takes a complicated binary, function-call, or match expression and explodes its arguments into a let block.
FlattenBadlyNestedLetsPass: here for the aesthetics of debug output. Replaces let x = let y = a in b in c

with

let y = a in let x = b in c

which gets pretty-printed as

let y = a x = b in c

In effect, flattening the lets.

The list RepeatedPasses gets run over the IR repeatedly until the IR stops changing or we've hit 5 iterations. In particular, I don't have any limits on inlining, so it hits the maximum frequently.

The program, optimised:

{
  test.i2s.main :: () -> lang.String
   = () =>
    letrec
      test.i2s.intToString_$42 =
        (in_$43 :: lang.Int) =>
          match(in_$43) {
            case 0 = "0"

            case _ = let
              _520_$520 = in_$43 / 10
            in
              match(_520_$520) {
                case 0 = match(in_$43 - in_$43 / 10 * 10) {
                  case 0 = "0"
                  case 1 = "1"
                  case 2 = "2"
                  case 3 = "3"
                  case 4 = "4"
                  case 5 = "5"
                  case 6 = "6"
                  case 7 = "7"
                  case 8 = "8"
                  case 9 = "9"
                }

                case _ = test.i2s.intToString_$42(in_$43 / 10) + match(in_$43 - in_$43 / 10 * 10) {
                  case 0 = "0"
                  case 1 = "1"
                  case 2 = "2"
                  case 3 = "3"
                  case 4 = "4"
                  case 5 = "5"
                  case 6 = "6"
                  case 7 = "7"
                  case 8 = "8"
                  case 9 = "9"
                }
              }
          }
    in
      letrec
        go_$327 =
          (in_$328 :: lang.Int) =>
            let
              _523_$523 = in_$328 - 1
            in
              match(_523_$523) {
                case 0 = 1
                case _ = in_$328 * go_$327(in_$328 - 1)
              }
      in
        let
          _524_$524 = go_$327(10)
          _525_$525 = test.i2s.intToString_$42(_524_$524)
        in
          "value: " + _525_$525
}

The program is somewhat shorter, and it's all been inlined into main. The fib function and other unused code is gone. We're left with three main parts: the recursive intToString, the recursive go function from factorial, and the original main function.

After optimising, the IR is translated in a highlevel bytecode format. The VM is fully register-based.

{
  main = @G0

  ===== DATA TYPES =====

  ===== STATIC BINDINGS =====
  @G0 = @F0

  ===== FUNCTIONS =====
  @F0 () => (() => lang.String) {
    LVT {
      . #0 :: () => lang.String
    }
    CODE {
      0000 #0 = @F1<>
      0001 return #0 :: () => lang.String
    }
  }
  @F1 () => lang.String {
    LVT {
      . #test.i2s.intToString :: (lang.Int) => lang.String,
      . #go :: (lang.Int) => lang.Int,
      . #_524 :: lang.Int,
      . #3 :: lang.Int,
      . #_525 :: lang.String,
      . #5 :: lang.String,
      . #6 :: lang.String
    }
    CODE {
      0000 #test.i2s.intToString = @F2<#test.i2s.intToString>
      0001 #go = @F3<#go>
      0002 #3 = 10
      0003 #_524 = #go(#3)
      0004 #_525 = #test.i2s.intToString(#_524)
      0005 #5 = "value: "
      0006 #6 = #5 + #_525 :: String
      0007 return #6 :: lang.String
    }
  }
  @F2 (lang.Int) => lang.String {
    LVT {
      c #test.i2s.intToString :: (lang.Int) => lang.String,
      a #in :: lang.Int,
      . #2 :: lang.String,
      . #3 :: lang.Int,
      . #_520 :: lang.Int,
      . #5 :: lang.Int,
      . #6 :: lang.Int,
      . #7 :: lang.Int,
      . #8 :: lang.Int,
      . #9 :: lang.Int,
      . #10 :: lang.Int,
      . #11 :: lang.Int,
      . #12 :: lang.Int,
      . #13 :: lang.Int,
      . #14 :: lang.Int,
      . #15 :: lang.Int,
      . #16 :: lang.Int,
      . #17 :: lang.Int,
      . #18 :: lang.Int,
      . #19 :: lang.Int,
      . #20 :: lang.Int,
      . #21 :: lang.Int,
      . #22 :: lang.String,
      . #23 :: lang.Int,
      . #24 :: lang.Int,
      . #25 :: lang.Int,
      . #26 :: lang.Int,
      . #27 :: lang.Int,
      . #28 :: lang.Int,
      . #29 :: lang.Int,
      . #30 :: lang.String,
      . #31 :: lang.Int,
      . #32 :: lang.Int,
      . #33 :: lang.Int,
      . #34 :: lang.Int,
      . #35 :: lang.Int,
      . #36 :: lang.Int,
      . #37 :: lang.Int,
      . #38 :: lang.Int,
      . #39 :: lang.Int,
      . #40 :: lang.Int
    }
    CODE {
      0000 mg37v_case0    #3 = 0
      0001                if #in != #3: jump @mg37v_case1
      0002                #2 = "0"
      0003                jump @mg37v_end
      0004 mg37v_case1    #5 = 10
      0005                #_520 = #in / #5
      0006 mqcdg_case0    #6 = 0
      0007                if #_520 != #6: jump @mqcdg_case1
      0008                #7 = 10
      0009                #8 = #in / #7
      0010                #9 = 10
      0011                #10 = #8 * #9
      0012                #11 = #in - #10
      0013 m2p9w_case0    #12 = 0
      0014                if #11 != #12: jump @m2p9w_case1
      0015                #2 = "0"
      0016                jump @m2p9w_end
      0017 m2p9w_case1    #13 = 1
      0018                if #11 != #13: jump @m2p9w_case2
      0019                #2 = "1"
      0020                jump @m2p9w_end
      0021 m2p9w_case2    #14 = 2
      0022                if #11 != #14: jump @m2p9w_case3
      0023                #2 = "2"
      0024                jump @m2p9w_end
      0025 m2p9w_case3    #15 = 3
      0026                if #11 != #15: jump @m2p9w_case4
      0027                #2 = "3"
      0028                jump @m2p9w_end
      0029 m2p9w_case4    #16 = 4
      0030                if #11 != #16: jump @m2p9w_case5
      0031                #2 = "4"
      0032                jump @m2p9w_end
      0033 m2p9w_case5    #17 = 5
      0034                if #11 != #17: jump @m2p9w_case6
      0035                #2 = "5"
      0036                jump @m2p9w_end
      0037 m2p9w_case6    #18 = 6
      0038                if #11 != #18: jump @m2p9w_case7
      0039                #2 = "6"
      0040                jump @m2p9w_end
      0041 m2p9w_case7    #19 = 7
      0042                if #11 != #19: jump @m2p9w_case8
      0043                #2 = "7"
      0044                jump @m2p9w_end
      0045 m2p9w_case8    #20 = 8
      0046                if #11 != #20: jump @m2p9w_case9
      0047                #2 = "8"
      0048                jump @m2p9w_end
      0049 m2p9w_case9    #21 = 9
      0050                if #11 != #21: jump @m2p9w_nomatch
      0051                #2 = "9"
      0052                jump @m2p9w_end
      0053 m2p9w_nomatch  crash "Match failed"
      0054 m2p9w_end      jump @mg37v_end
      0055 mqcdg_case1    #23 = 10
      0056                #24 = #in / #23
      0057                #22 = #test.i2s.intToString(#24)
      0058                #25 = 10
      0059                #26 = #in / #25
      0060                #27 = 10
      0061                #28 = #26 * #27
      0062                #29 = #in - #28
      0063 mysu8_case0    #31 = 0
      0064                if #29 != #31: jump @mysu8_case1
      0065                #30 = "0"
      0066                jump @mysu8_end
      0067 mysu8_case1    #32 = 1
      0068                if #29 != #32: jump @mysu8_case2
      0069                #30 = "1"
      0070                jump @mysu8_end
      0071 mysu8_case2    #33 = 2
      0072                if #29 != #33: jump @mysu8_case3
      0073                #30 = "2"
      0074                jump @mysu8_end
      0075 mysu8_case3    #34 = 3
      0076                if #29 != #34: jump @mysu8_case4
      0077                #30 = "3"
      0078                jump @mysu8_end
      0079 mysu8_case4    #35 = 4
      0080                if #29 != #35: jump @mysu8_case5
      0081                #30 = "4"
      0082                jump @mysu8_end
      0083 mysu8_case5    #36 = 5
      0084                if #29 != #36: jump @mysu8_case6
      0085                #30 = "5"
      0086                jump @mysu8_end
      0087 mysu8_case6    #37 = 6
      0088                if #29 != #37: jump @mysu8_case7
      0089                #30 = "6"
      0090                jump @mysu8_end
      0091 mysu8_case7    #38 = 7
      0092                if #29 != #38: jump @mysu8_case8
      0093                #30 = "7"
      0094                jump @mysu8_end
      0095 mysu8_case8    #39 = 8
      0096                if #29 != #39: jump @mysu8_case9
      0097                #30 = "8"
      0098                jump @mysu8_end
      0099 mysu8_case9    #40 = 9
      0100                if #29 != #40: jump @mysu8_nomatch
      0101                #30 = "9"
      0102                jump @mysu8_end
      0103 mysu8_nomatch  crash "Match failed"
      0104 mysu8_end      #2 = #22 + #30 :: String
      0105 mg37v_end      return #2 :: lang.String
    }
  }
  @F3 (lang.Int) => lang.Int {
    LVT {
      c #go :: (lang.Int) => lang.Int,
      a #in :: lang.Int,
      . #_523 :: lang.Int,
      . #3 :: lang.Int,
      . #4 :: lang.Int,
      . #5 :: lang.Int,
      . #6 :: lang.Int,
      . #7 :: lang.Int,
      . #8 :: lang.Int
    }
    CODE {
      0000              #3 = 1
      0001              #_523 = #in - #3
      0002 m8jew_case0  #5 = 0
      0003              if #_523 != #5: jump @m8jew_case1
      0004              #4 = 1
      0005              jump @m8jew_end
      0006 m8jew_case1  #7 = 1
      0007              #8 = #in - #7
      0008              #6 = #go(#8)
      0009              #4 = #in * #6
      0010 m8jew_end    return #4 :: lang.Int
    }
  }

}

The "bytecode" is a set of instructions grouped by function. Each instruction is pretty-printed in a pseudo-code sort of way so it's not that hard to read, although match expressions compile into massive unwieldy if-then-jump tables.

Each function has a Local Variable Table that details how many registers it will use and their types. Closure and function arguments get passed in through the first few registers. Any function can have closure (most do), so to call one, you must first instantiate it with its closure. The function value can be passed around and will hold onto that closure.

If all user functions are instantiated, how does the runtime call main? Well, there's the concept of global bindings. A global binding is a lazily initialised value that use a special function (one that doesn't need instantiation) to create its value. When called upon, it calls that function and persists the result. The code that asked for its value (and subsequent askings) will use that result.

In this example, main is defined to be @G0. @G0 uses @F0 to instantiate the function @F1, and the interpreter invokes the result to run the program.

@F2 is the recursive intToString, and @F3 is the recursive go.

And the result is of course:

result :: lang.String = "value: 3628800"

Conclusion

There's a lot more I could say. The project took a lot of different turns to get here, depending on what articles I'd been reading or which language I was fascinated with at the time (right now it's Haskell). There were plenty of dead ends --- I rewrote the typechecker several times because it kept breaking. The latest branch tried a different optimisation technique that failed because I forgot about closure (...and I can't be bothered to handle it. Closure complicates everything).

I occasionally have dreams of the superoptimiser, eg: Why can't the optimiser simplify the factorial program down to "value: 3628800"?

Answer: recursion is hard to optimise, even if you have a deep CS/mathematics degree (I do not).

I've wanted to write a last stage that targets LLVM bitcode, but I never found the time (or desire).

And of course, I want to implement actors. But they're a lot of work too...

This was a fun project, and I learned a lot. If I ever do any more language design, however, I'll probably scrap the whole thing and start again, without the piled up twists and turns. And in a language less verbose than Scala.

Battling Maven repository hosting

Lachlan Phillips — Thu, 07 Aug 2014 22:23:07 GMT

Write your post here.

title: Battling maven repository hosting date: 2014-08-08 08:53:07 categories: - Software tags: - java - note-to-self

Recently I've been working on a closed-source application for a client. It's an Android implementation of a recently invented card game, with both single player and multiplayer modes. I had to write a custom server for it (I chose the Play framework with Scala). The fact that the system had effectively two server implementations --- one for single player coordinating bots, and one for multiplayer --- combined with my lack of experience building such systems meant I created a lot of duplication when implementing the rules of the game.

There were two implementations on the client, one in the mp server, and one in a mp test client. Even though I'd written them in a TDD style and had a good test coverage, there was still a lot of ground for bugs to hide in. Bugs that differed between single player and multiplayer, etc. Worse was the time I'd misinterpreted one of the rules, and had to fix it in several places.

Eventually, I decided to create one library of rules to be used everywhere. The past few days I've been refactoring all the Java code in the client to use one rules package --- then yesterday I ripped it out into its own SBT project. That all went well --- until I sat down to decide where to host the jar artifact.

The entire code base is closed-source, so I couldn't conveniently upload it to Maven Central or some other free open-source host. I Googled around, and heard suggestions including hosting on GitHub, uploading via FTP to a web server, running a dedicated Maven repository manager, etc. I really didn't want to dump the artifacts on GitHub --- it's an abuse of git (and possibly a breach of the GitHub terms of service). And, for various reasons, the other options were not to my liking.

Then I remembered --- I have a Cloudbees subscription! Cloudbees offers free repository hosting (up to a certain size). I set up a repo in Cloudbees, then googled how you deploy from SBT. Very quickly I saw that SBT by itself cannot deploy files via WebDAV. So I found a plugin that could. Five minutes later, I get a dreaded message on the SBT console.

Unresolved dependency...

Its repository was down. I went and found another plugin, installed it, and punched in my Cloudbees details. Another message on the console:

Error uploading file... HTTP 405

I had to Google what an HTTP 405 was. Then I fought, playing with permissions on the Cloudbees repo, credentials on my local machine... but to no avail.

That took a good afternoon to work out that Cloudbees was not worth the effort this time. So I finally decided to try somewhere else. I still did not want to upload to GitHub, so I decided to try uploading to my shared hosting via FTP.

Possibly an hour later, I worked out my hosting provider (VentraIP) does not support SFTP --- only FTP, and SBT does not support FTP.

So I gave up. I deployed my artifacts to the Archiva hosted on my local machine. Thus, builds would never work anywhere but on my local Jenkins instance --- and my boss across the country couldn't compile the Android code to try it himself without considerable effort.

A few sad commit messages appeared in the BitBucket repo.

Then, for some reason, I recalled one more place to host a maven repo --- in an AWS S3 bucket. I googled for a plugin for SBT, immediately found one and installed it. After a bit of fiddling with credentials, the deploy succeeded. Clean console.

Yes! Finally! But can I get the artifacts back again? I opened the bucket to the public (security through obscurity) and put the address into the Android Gradle build. At first, it didn't work --- but it turns out you have to make folders trees explicitly public as well as the bucket itself. Gradle downloaded the library with no complaint, and not long after the server's SBT build did so too.

The commits were counteracted with "build now works globally," and I was finished.

Note to self: use S3 buckets for private maven repo hosting. The sbt-s3-resolver from frugalmechanic works really well.

Rigid-body physics using particles

Lachlan Phillips — Wed, 15 Jan 2014 17:58:07 GMT

Towards the end of last year, I worked on a prototype physics engine based on a GPU Gems chapter --- a realtime physics simulation running on the GPU. So far I've only written it for the CPU as I've never done any GPU work before. (judging by the problems I've had with the physics side of things, it was probably a good idea not to heap on an order of magnitude more GPU problems...)

Here's a video of it in action:

After I managed to get rotation pretty much working' --- it still does odd things --- it worked pretty well. It's operating on the pixel/metre scale, because I was lazy... so everything here is massive skyscraper-sized entities. Quite by accident, it appears to have 'conservation of momentum' --- if I draw a 'C', and fill it with particles it will accelerate like a ~rocket~ snail.

Eventually, I'll work out how to port it to GPU, hook it up with OpenGL, and make it 3D... but that'll be a while.

In the meanwhile, read the code at Github, and... I've not got the build properly set up on Jenkins so you can't download it.

EDIT: another video from some time later:

Balloon Box

Lachlan Phillips — Fri, 27 Sep 2013 01:17:21 GMT

I've recently been working on a game called Balloon Box. It's a 2D desktop platformer in which the player controls a cardboard box, navigates through an abandoned factory which is known for its pneumatic material transport system. (When I implement it) you will have to avoid cardboard-dissolving acid and knife racks, and collect balloons and batteries. Read the story on Github.

The source code is available on Github. It's written in Java, using LibGDX and Box2D, via Maven (so the code is nested rather deeply...). Also, I'm using a library I developed called the Live Constant Tweaker (incredibly useful for tweaking physics constants) that only exists in my local Maven repo, so you'll have to get it to compile Balloon Box.

The latest build from Jenkins is available here.

A video of wip gameplay:

The performance monitor:

Some screenshots from yesterday:

Collecting a balloon - screenshot seems to have caught some tearing artifacts

Eventually I'll work out a way to make it build on other machines, and hopefully set up some kind of nightly build.

Growing Trees with Space Colonisation in 2D

Lachlan Phillips — Fri, 06 Sep 2013 01:50:27 GMT

Yesterday, I quickly hacked up an implementation of the Space Colonisation algorithm, for growing trees. By trees, I don't mean the binary trees – the living sort, just represented by segments and forks.

My inspiration was from Procedural World and this article. The idea of Space Colonisation is to colonise/fill a limited area. A tree like this would grow from the bottom up (as they usually do), gradually taking up all the available space. Trees generated by this algorithm (as far as I know) rarely have intersecting branches, and look organic.

I started coding from the top (as usual I didn't plan much beforehand, but it didn't turn out too bad) – a Main class that creates a Point Cloud in a spherical shape, grows a tree in that cloud, and renders the tree out to an image. Later I added a GUI for faster feedback.

After some work, I got this:

The green points are the Point Cloud, and the black dots with the yellow lines are the generated tree (segments).

When I constructed the GUI, I added the option of scaling the point cloud horizontally, which can give me trees like:

Here's a quick shot of the GUI:

I've still got more work to do, things like

Add different point cloud shapes
- Generate from an image?
Support for multiple trees growing in the same space
Biases
- Bias upwards
Rendering more tree-like
- Trunk should be thicker than topmost twigs
- Very close branches could be merged

The code is available on Github.