Notes on Unity LEGO Microgame Creative Mods

Once a Unity 3D beginner completed a tightly-scripted microgame tutorial, we are directed towards a collection of “Creative Mods”. These suggested exercises build on top of what we created in the scripted tutorial. Except now individual tasks are more loosely described, and we are encouraged to introduce our own variations. We are also allowed to experiment freely, as the Unity Editor is no longer partially locked down to keep us from going astray. The upside of complete freedom is balanced by the downside of easily shooting ourselves in the foot. But now we know enough to not do that, or know how to fix it if we do. (In theory.)

Each of the Unity introductory microgames have their own list of suggested modifications, and since I just completed the LEGO microgame I went through the LEGO list. I was mildly surprised to see this list grow while I was in the middle of doing it — as of this writing, new suggested activities are still being added. Some of these weren’t actually activities at all, such as one entirely focused on a PDF (apparently created from PowerPoint) serving as a manual for the list of available LEGO Behaviour Bricks. But most of the others introduce something new and interesting.

In addition to the LEGO themed Unity assets from the initial microgame tutorial, others exist for us to import and use in our LEGO microgame projects. There was a Christmas-themed set with Santa Claus (causing me to run through Visual Studio 2019 installer again from Unity Hub to get Unity integration), a set resembling LEGO City except it’s on a tropical island, a set for LEGO Castles, and my personal favorite: LEGO Space. Most of my personal LEGO collection were from their space theme and I was happy to see a lot of my old friends available for play in the digital world.

When I noticed the list of activities grew while I was working on them, it gave me a feeling this was a work-in-progress. That feeling continued when I imported some of these asset collections and fired up their example scene. Not all of them worked correctly, mostly centered around how LEGO pieces attached to each other especially the Behaviour Bricks. Models detach and break apart at unexpected points. Sometimes I could fix it by using the Unity Editor to detach and re-attach bricks, but not always. This brick attachment system was not a standard Unity Editor but an extension built for the LEGO microgame theme, and I guess there are still some bugs to be ironed out.

The most exciting part of the tutorial was an opportunity to go beyond the LEGO prefab assets they gave us and build our own LEGO creations for use in Unity games. A separate “Build your own Enemy” tutorial gave us instructions on how to build with LEGO piece by piece within Unity Editor, but that’s cumbersome compared to using dedicated LEGO design software like BrickLink Studio and exporting the results to Unity. We don’t get to use arbitrary LEGO pieces, we have to stay within a prescribed parts palette, but it’s still a lot of freedom. I immediately built myself a little LEGO spaceship because old habits die hard.

I knew software like BrickLink Studio existed but this was the first time I sat down and tried to use one. The parts palette was disorienting, because it was completely unlike how I work with LEGO in the real world. I’m used to pawing through my bin of parts looking for the one I want, not selecting parts from a menu organized under an unfamiliar taxonomy. I wanted my little spaceship to have maneuvering thrusters, something I add to almost all of my LEGO space creations, but it seems to be absent from the approved list. (UPDATE: A few days later I found it listed under “3963 Brick, Modified 1 x 1 with 3 Loudspeakers / Space Positioning Rockets”) The strangest omission seem to be wheels. I see a lot of parts for automobiles, including car doors and windshields and even fender arches. But the only wheels I found in the approved list are steering wheels. I doubt they would include so many different fender arches without wheels to put under them, but I can’t find a single ground vehicle wheel in the palette! Oversight, puzzling intentional choice, or my own blindness? I lean towards the last but for now it’s just one more reason for me to stick with spaceships.

My little LEGO spaceship, alongside many other LEGO microgame Creative Mods exercises (but not all since the list is still growing) was integrated into my variant of the LEGO microgame and uploaded as “Desert Dusk Demo“. The first time I uploaded, I closed the window and panicked because I didn’t copy down the URL and I didn’t know how to find it again. Eventually I figured out that everything I uploaded to Unity Play is visible at https://play.unity.com/discover/mygames.

But since the legal terms of LEGO microgame assets are restricted to that site, I have to do something else for my learn-and-share creation for ART.HAPPENS. There were a few more steps I had to take there before I had my exhibit Bouncy Bouncy Lights.

Notes on Unity LEGO Microgame Tutorial

To help Unity beginners get their bearings inside a tremendously complex and powerful tool, Unity published small tutorials called microgames. Each of them represent a particular game genre, with the recently released LEGO microgame as the default option. Since I love LEGO, I saw no reason to deviate from this default. These microgame tutorials are implemented as Unity project templates that we can launch from Unity’s Hub launcher, they’re just filled out with far more content than the typical Unity empty project template.

Once a Unity project was created with the LEGO microgame template (and after we accepted all the legal conditions of using these LEGO digital assets) we see the complex Unity interface. Well aware of how intimidating it may look to a beginner, the tutorial darkened majority of options and highlighted just the one we need for that step in the tutorial. Which got me wondering: the presence of these tutorial microgames imply the Unity Editor UI itself can be scripted and controlled, how is that done? But that’s not my goal today so I set that observation aside.

The LEGO microgame starts with the basics: how to save our progress and how to play test the game in its current state. The very first change is adjusting a single variable, our character’s movement speed, and test its results. We are completely on rails at this point: the Unity Editor is locked off so I couldn’t change any other character variable, and I couldn’t even proceed unless I changed the character speed to exactly the prescribed value. This is a good way to make sure beginners don’t inadvertently change something, since we’d have no idea how to fix it yet!

Following chapters of the tutorial gradually open up the editor, allowing us to use more and more editor options and giving us gradually more latitude to change the microgame as we liked. We are introduced to the concept of “assets” which are pieces we use to assemble our game. In an ideal world they snap together like LEGO pieces, and in the case of building this microgame occasionally they actually do represent LEGO pieces.

Aside from in-game objects, the LEGO minigame also allows us to define and change in-game behavior using “Behaviour Bricks”: Assets that look just like another LEGO block in game, except they are linked to Unity code behind the scenes giving them more functionality than just a static plastic brick. I appreciated how it makes game development super easy, as the most literal implementation of “object-oriented programming” I have ever seen. However, I was conscious of the fact these behavior bricks are limited to the LEGO microgame environment. Anyone who wishes to venture beyond would have to learn entirely different ways to implement Unity behavior and these training wheels will be of limited help.

The final chapter of this LEGO microgame tutorial ended with walking us through how to build and publish our project to Unity Play, their hosting service for people to upload their Unity projects. I followed those steps to publish my own LEGO microgame, but what’s online now isn’t just the tutorial. It also included what they called “Creative Mods” for a microgame.

Unity Tutorial LEGO Microgame

Once I made the decision to try learning Unity again, it was time to revisit Unity’s learning resources. This was one aspect that I appreciated about Unity: they have continuously worked to lower their barrier to entry. Complete beginners are started on tutorials that walk us through building microgames, which are prebuilt Unity projects that show many of the basic elements of a game. Several different microgames are available, each representing a different game genre, so a beginner can choose whichever one that appeals to them.

But first an ambitious Unity student had to install Unity itself. Right now Unity releases are named by year much like other software like Ubuntu. Today, the microgame tutorials tell beginners to install version 2019.4 but did not explain why. I was curious why they tell people to install a version that is approaching two years old so I did a little digging. The answer is that Unity designates specific versions as LTS (Long Term Support) releases. Unity LTS is intended to be a stable and reliable version, with the best library compatibility and the most complete product documentation. More recent releases may have shiny new features, but a beginner wouldn’t need them and it makes sense to start with the latest LTS. Which, as of this writing, is 2019.4.

I vaguely recall running through one of these microgame exercises on an earlier attempt at Unity. I chose the karting microgame because I had always liked driving games. Gran Turismo on Sony PlayStation (the originals in both cases, before either got numbers) was what drew me into console gaming. But I ran out of steam on the karting microgame and those lessons did not stick. Since I’m effectively starting from scratch, I might as well start with a new microgame, and the newest hotness released just a few months ago is the LEGO microgame. Representing third-person view games like Tomb Raider and, well, the LEGO video games we can buy right now!

I don’t know what kind of business arrangement behind the scenes made it possible to have digital LEGO resources in our Unity projects, but I am thankful it exists. And since Unity doesn’t own the rights to these assets, the EULA for starting a LEGO microgame is far longer than for the other microgames using generic game assets. I was not surprised to find clauses forbidding use of these assets in commercial projects, but I was mildly surprised that we are only allowed to host them on Unity’s project hosting site. We can’t even host them on our own sites elsewhere. But the most unexpected clause in the EULA is that all LEGO creations depicted in our minigames must be creatable with real LEGO bricks. We are not allowed to invent LEGO bricks that do not exist in real life. I don’t find that restriction onerous, just surprising but made sense in hindsight. I’m not planning to invent an implausible LEGO brick in my own tutorial run so I should be fine.

Checking In on Unity 3D

Deciding to participating in ART.HAPPENS is my latest motivation to look at Unity 3D, something I’ve done several times. My first look was almost five years ago, and my most recent look was about a year and a half ago in the context of machine learning. Unity is a tremendously powerful tool and I’ve gone through a few beginner tutorials, but I never got as far as building my own Unity project. Will that finally change this time?

My previous look at Unity was motivated by an interest in getting into the exciting world of machine learning, specifically in the field of reinforcement learning. That line of investigation did not get very far, but as most machine learning tools are focused on Linux there was the question of Unity’s Linux support. Not just to build a game (which is supported) but also to run the Unity editor itself on Linux. My investigation was right around the time Unity Editor for Linux entered beta with expectation for release in 2020, but that has been pushed to 2021.

For my current motivation, it’s not as important to run the editor on Linux. I can just as easily create something fun and interactive by running Unity on Windows. Which led to the next question: could I output something that can work inside an <iframe> hosted within Gather, the virtual space for ART.HAPPENS? On paper the answer is yes. Unity has had the ability to render content using WebGL for a while, and their code has matured alongside browser support for WebGL. But even better is the development (and even more importantly, browser adoption) of WebAssembly for running code in a browser. This results in Unity titles that are faster to download and to execute than the previous approach of compiling Unity projects to JavaScript. These advancements are far more encouraging than what Unity competitor Unreal Engine has done, which was to boot HTML5 support out of core to a community project. Quite a sharp contrast to Unity’s continued effort to make web output a first class citizen among all of its platforms, and this gives me the confidence to proceed and dive in to the latest Unity tutorial for beginners: LEGO!

ART.HAPPENS Motivates Return to Unity 3D

I’ve been talking about rovers on this blog for several weeks nonstop. I thought it would be fun to have a micro Sawppy rover up and running in time for Perseverance landing on February 18th, but I don’t think I’ll make that self-imposed deadline. I have discovered I cannot sustain “all rovers all the time” and need a break from rover work. I’m not abandoning the micro rover project, I just need to switch to another project for a while as a change of pace.

I was invited to participate in ART.HAPPENS, a community art show. My first instinct was to say “I’m not an artist!” but I was gently corrected. This is not the fancy schmancy elitist art world, it is the world of people having fun and sharing their works kind of world. Yes, some of the people present are bona fide artists, but I was assured anyone who wants to share something done for the sheer joy of creating can join in the fun.

OK then, I can probably take a stab at it. Most of my projects are done to accomplish a specific purpose or task, so it’s a rare break to not worry about meeting objectives and build something for fun. My first line of thought was to build a follow-up to Glow Flow, something visually pleasing and interactive built out of 3D printed parts and illuminated by colorful LEDs controlled with a Pixelblaze. It’s been on my to-do list to explore more ideas on how else to use a Pixelblaze.

Since we’re in the middle of a pandemic, this art show is a virtual affair. I learned that people will be sharing photos and videos of their projects and shown in a virtual meeting space called Gather. Chosen partially because the platform was built to be friendly to all computer skill levels, Gather tries to eliminate friction of digital gatherings.

I poked my head into Gather and saw an aesthetic that reminded me of old Apple //e video games that used a top-down tiled view. For those old games, it was a necessity due to the limited computing power and memory of an old Apple computer. And those same traits are helpful here to build a system with minimal hardware requirements.

Sharing photos and videos of something like Glow Flow would be fun, but wouldn’t be the full experience. Glow Flow looked good but the real fun comes from handling it with our own hands. I was willing to make some compromises in the reality of the world today, until I noticed how individual projects will be shared as web content that will be hosted in an <iframe>. That changes the equation. Because it meant I could build something interactive after all. If I have control of content inside an <iframe>, I can build interactive web content for this show.

I briefly looked at a few things that might have been interesting, like three.js and A-Frame. But as I read documentation for those platforms, my enthusiasm dampened by shortcomings I came across. For example, building experiences incorporating physics simulation seems to be a big can of worms on those platforms. Eventually I decided: screw it, if I’m going to do this, I’m going to go big. It’s time to revisit Unity 3D.

Unity 3D Editor for Ubuntu Is Almost Here

So far I’ve dipped my toes in the water of reinforcement learning, reinstalled Ubuntu and TensorFlow, and looked into Unity ML-Agents. It looks like I have a tentative plan for building my own reinforcement learning agents trained with TensorFlow in an Unity 3D environment.

There’s one problem with this plan, though: I have GPU-accelerated TensorFlow running in Ubuntu, but today Unity editor only supports MacOS and Windows. If I wanted to put them all together, on paper it means I’d have to get Nvidia GPU support up and running on my Windows partition and take on all the headaches that entails.

Thankfully, I’m not under a deadline to make this work immediately, so I can hope that Unity brings their editing and creation environment to Ubuntu. The latest preview build was released only a few days ago, and they expected that Linux will be a fully supported operating system for Unity Editor by the end of the year.

I suspect that they’ll be ready before I am, because I still have to climb the newcomer learning curve of reinforcement learning. I first have to learn the ropes using prebuilt OpenAI environments. It’ll be awhile before I can realistically contemplate designing on own agents and simulation environments.

Once I reach that point I hope I will be able to better evaluate whether my plan will actually work. Will Unity ML-Agents work with GPU-accelerated TensorFlow running in a Docker container on Ubuntu? I’ll have to find out when I get there.

Researching Simulation Speed in Gazebo vs. Unity

In order to train reinforcement learning agents quickly, we want our training environment to provide high throughput. There are many variables involved, but I started looking at two of them: how fast it would be to run a single simulation, and how easy it would be to run multiple simulation in parallel.

The Gazebo simulator commonly associated with ROS research projects has never been known for its speed. Gazebo environment for the NASA Space Robotic Challenge was infamous for slowing far below real time speed. Taking over 6 hours to simulate a 30 minute event. There are ways to speed up Gazebo simulation, but this forum thread implies it’s unrealistic to expect more than 2-3 times as fast as real time speed.

In contrast, Unity simulation can be cranked all the way up to 100 times real time speed. It’s not clear where the maximum limit of 100 comes from, but it is documented under limitations.md. Furthermore, it doesn’t seem to be a theoretical limit no one can realistically reach – at least one discussion on Unity ML Agents indicate people do indeed crank up time multiplier to 100 for training agents.

On the topic of running simulations in parallel, with Gazebo such a resource hog it is difficult to get multiple instances running. This forum thread explains it is possible and how it could be done, but at best it still feels like shoving a square peg in a round hole and it’ll be a tough act to get multiple Gazebo running. And we haven’t even considered the effort to coordinate learning activity across these multiple instances.

Things weren’t much better in Unity until recently. This announcement blog post describes how Unity has just picked up the ability to run multiple simulations on a single machine and, just as importantly, coordinate learning knowledge across all instances.

These bits of information further cements Unity as something I should strongly consider as my test environment for playing with reinforcement learning. Faster than real time simulation speed and option for multiple parallel instances are quite compelling reasons.

 

Quick Overview: Unity ML Agents

Out of all the general categories of machine learning, I find myself most interested in reinforcement learning. These problems (and associated solutions) are most applicable to robotics, forming the foundation of projects like Amazon’s DeepRacer. And the fundamental requirement of reinforcement learning is a training environment where our machine can learn by experimentation.

While it is technically possible to train a reinforcement learning algorithm in the real world with real robots, it is not really very practical. First, because a physical environment will be subject to wear and tear, and second, because doing things in the real world at real time takes too long.

For that reason there are many digital simulation environments in which to train reinforcement learning algorithms. I thought it would be an obvious application of robot simulation software like Gazebo for ROS, but this turned out to only be partially true. Gazebo only addresses half of the requirements: a virtual environment that can be easily rearranged and rebuilt and not subject to wear and tear. However, Gazebo is designed to run in a single instance, and its simulation engine is complex enough it can fall behind real time meaning it takes longer to simulation something than it would be in the real world.

For faster training of reinforcement learning algorithms, what we want is a simulation environment that can scale up to run multiple instances in parallel and can run faster than real time. This is why people started looking at 3D game engines. They were designed from the start to represent a virtual environment for entertainment, and they were built for performance in mind for high frame rates.

The physics simulation inside Unity would be less accurate than Gazebo, but it might be good enough for exploring different concepts. Certainly the results would be good enough if the whole goal is to build something for a game with no aspirations for adapting them to the real world.

Hence the Unity ML-Agents toolkit for training reinforcement learning agents inside Unity game engine. The toolkit is nominally focused on building smart agents for game non-player characters (NPC) but that is a big enough toolbox to offer possibilities into much more. It has definitely earned a spot on my to-do list for closer examination in the future.

Unity exploration on hold

After an intense few weeks with Unity, I’ve decided to put it on hold and come back to it later. The biggest problem is that, even though it can output to web browsers via WebGL, it is extreme overkill for projects that can be easily handled via Plain Jane HTML.

As an Unity exercise, I pulled art and assets from the old game Star Control 2. This was possible because (1) the creators of the game released the code in 2002, and (2) I’m no longer working at Microsoft. I was aware of the code release when it occurred, but my employment as a Microsoft developer at the time meant I could not look at anything released under GPL. Even that of an old game with zero relation to the work I’m paid to do.

Since it was a simple 2D game, bringing the assets to life using Unity severely underutilized Unity capabilities. That isn’t a problem, the part that gave me pause is that the end product is a lot heavier-weight than I thought it would be: Even when there’s only a few 2D sprites animating with some text overlay, it is a larger download with correspondingly slower startup than I wanted to see.

Given what I see so far, Unity is the wrong tool for this job. Now I have a decision to make. Option #1: Keep digging in on an old favorite, and work on SC2 code using some other framework, or Option #2: Find a project that plays more to Unity’s (many) strengths.

Option #1 goes to the past, option #2 goes to the future. (Specifically VR…. Unity is great for exploring VR development.)

Today, I find that I’ve caught the old SC2 fever again, so I’m going with #1. I’ll come back to Unity/VR/other explorations later.

 

Unity Mecanim Animation Notes

Unity’s learning center has a lot of information, I chose to start with the headliner tutorials. These appear to be full-day lectures during the Unite conference, where they take the class through building a game from beginning to end.

Since real games take more than a day to build, many shortcuts were taken. One of these shortcuts were the use of built art assets. All came already created, complete with their own associated animation sequences. The tutorial only covered how to import the items and write a few lines of code to trigger the animation sequences.

I have no illusions about being an artist but I also know I don’t have an artist to call upon as I learn Unity. So I had to know something about creating these assets for myself. I thought I would start small with a few simple sprite animations… that turned out to be not so simple.

The Unity animation engine (sometimes called Mecanim in the documentation) is a very complex machine optimized to work with humanoid figures in 3D space. It can certainly do simple sprite animations, but trying to do so became an exercise in figuring out what to turn off in the big complex machine. It keeps trying to do too much, blending and interpolating and trying to be helpful when all I really want was to put a few 2D images on screen at discrete coordinates at specific points in time.

It took way more time than it should, but (1) I got my simple sprite animations working, and (2) I learned a whole bunch about what the animation engine can do for me, down the line, when I’m ready to move beyond sprites.

It was a bit frustrating, but now that I’m through it, I’ll call it a win. Time to move on.

Onward to Unity Adventures

After deciding to move on from Phaser, I looked around for other game engines to generate web games. I came across Unity again and again. Unity is not new to me, but I also knew their web support was via the Unity web player browser plug-in. This is a problem, as browser plug-ins have fallen into disfavor. All the major desktop browsers Firefox / Chrome / IE, are moving away from plug-ins.

So… dead end (or at least dying). Plug-ins are not the future of the web.

Because of the web player, I dismissed every mention of Unity for web development I encountered, until I came across information that slapped me upside the head and woke me up. The web player is not the only Unity web target: Unity now has another web-friendly build target: WebGL, no plug-in required! Wow!

This warrants a closer look. While WebGL might be immature and inconsistent across browsers, it is a shared goal of many interests to evolve and mature the platform and I share their high hopes. The browser plug-in model is going away. WebGL may or may not take off but all the signs today look promising.

Unity had been on my to-do list of research topics, but it ranked lower than HTML-related technologies. (This is why I started with HTML, JavaScript, Node.JS, etc.) There are many interesting areas of development I can explore with Unity. Graphics, networking, user interface design, virtual reality, Android development, iOS development, the list goes on.

And now I learned that I can even explore some parts of the web world with Unity. It is quite the Swiss Army Knife of software development. With this latest discovery, Unity has moved to the top of my to-do list. It is time to roll up my sleeves and dig in.

This should be fun.