Updates from July, 2010 Toggle Comment Threads | Keyboard Shortcuts

  • Dark Liquid 6:50 am on April 9, 2012 Permalink | Reply
    Tags: , , , , radio   

    My First “Proper” Node.js app 

    So over the weekend I cranked out my first proper node.js application. It’s a very simple frontend for controlling MPD (the music playing daemon). When I say simple, I mean just a start and stop button and a button to ask it to update its track database. It’s designed as a simple tool for G3 Radio to control the automated jukebox they run between their live DJ shows. More features are on the cards as per the wiki for the project, but at the moment it does the job.

    It’s using express.js as the webservers and mpd-socket for talking to MPD. I have it load JSON files for its configs. I’m using the lovely everyauth middleware for auth against a flat file, which takes all the pain out of that process. It was pretty easy to write after I wrapped my brain around some annoying issues I was having.

    Anyway, if you fancy seeing some bumbling attempt to make a node.js application and/or need something like what I’ve described,  check it out on github: g3hal on g3radio @ github

     
  • Dark Liquid 9:28 pm on March 31, 2012 Permalink | Reply
    Tags: cucumber, , , testing   

    Automated Web Testing With Cucumber, Selenium and a custom fixtures server 

    Recently I’ve been working on a website that is a single page AJAX driven application that’s essentially just a frontend to an API, rather than doing any of it’s own processing.

    For testing it I wanted to be able to test without hitting the API but since the application is so javascript heavy and makes use of various complex workarounds for various browser issues, testing it using the usual methods never worked and only “real” browsers cut the mustard.

    I’ve tried a number of headless browsers such as zombie.js but they’ve all had issues so I’ve settled on using selenium for testing in a real browser. The other issue is that most web application testing systems assume at least some kind of backend system that you can then mock the http calls of. However in my case the entire app is client side and has no server at all, so doing that wasn’t an option.

    The solution I came up with was a cobbled together selection of rake tasks, shell scripts, a custom node.js server and some hooks in cucumber to set up fixtures.  I run my tests against a live server through a recording proxy to get the results of the API calls, then format the results into little fixture files to get loaded by the fixtures server based on tags on my cucumber scenarios. It’s quite a flexible setup now it’s running, the only chore is writing the fixtures, though hopefully I can knock up a quick script to generate the fixtures automatically from my proxy recordings.

    I wont go into the details of running cucumber with capybara and selenium, since those are already well documented elsewhere, such as on the capybara github page. However, I’ll run through my scripts.

    First off, I have my bash script to actually setup my server and run all my tests, it goes a little something like this:
    UPDATE: Made some amendments thanks to suggestions from Ralph Corderoy in the comments.
    UPDATE 2: Removed set -eu line since breaking out on errors means not shutting down the fixture server when tests fail.

    What is happening in the above should be fairly obvious, but essentially I’m running my fixtures server and capture it’s PID so I can shut it down later. I wait for a bit, then execute my tests via a Rakefile, then I send a command to the fixture server to shutdown, wait for a bit, then force a shutdown in case it’s not done it already. Simples!

    The fixture server can be found in my random scripts repository on github. It’s hard-coded to run on port 7357 but that’s easy to change. I’ve tried to document how it works in the code but a gist of it is that the fixture server will server static assets but fall through on all other requests to look at whatever fixtures it has loaded. By default, it will have loaded no fixtures so will 404. However, it has some special URLs for loading in fixtures. On /_fixtures/load/:fixture the server will load a json fixture file with the name :fixture.json. Multiple fixtures can be loaded for the same path and the server will work it’s way through them, serving each one once until it has no more where it will then 404 again. Fixtures can be cleared from memory with a call to /_fixtures/clear and you can inspect the currently loaded fixtures by calling /_fixtures/inspect. A call to /_fixtures/shutdown will instruct the server to shut itself down. At the moment the fixture server only supports loading fixtures for GET and POST requests as that’s all I’ve needed, but it’s easy to extend that.

    The fixture files themselves are very simple. Here is an example one:

    In the above fixture file, the server is setup to respond with a failure message the first time it is accessed with a GET on /test, a redirect the first time it is accessed with a POST on /test/post and a success message the second time it is accessed with a GET on /test.

    In cucumber, using the stuff I’ve mentioned above, we can now do cool stuff like configuring fixtures for each scenario. I use something like this to setup my fixtures:

    I slap that in a .rb file in my features/support directory and I can now use the @authed and @notauthed tags to load the appropriate fixtures into the server, like in the following feature:

    You’ll notice that I can do cool things like load multiple fixtures by specifying multiple tags. The fixtures get loaded in the order that that tags appear so I can create fairly complex fixture scenarios from simple individual fixtures.

    All in all I’ve found it a reasonably nice simple way of being able to test client-side only apps that drive APIs without having to hit the actual API server. Obviously if you are developing your own API server as well, you should be writing tests for the API server in addition to the client, but this way to can keep them nicely de-coupled in your tests so you don’t have to hit live data, which based on your project might be impossible to test against anyway.

     
    • Ralph Corderoy 11:16 am on April 1, 2012 Permalink

      Nice write-up, though I stumbled where you nouned apply! My brain is wired to know that word has one meaning and sound. Perhaps ‘appli’, if not ‘app’? On the bash you may like to know &> does the job of > followed by 2>&1, e.g. &>/dev/null, and ‘set -eu’ is always handy at the start of a bash script, e.g. curl(1) fails, though you’d expect the kill(1) to fail; kill $PID &>/dev/null || true.

    • Dark Liquid 3:32 pm on April 1, 2012 Permalink

      That use of apply was a mistake caused by writing the original draft for this article on a phone. Bloody auto-incorrect!

      I wasn’t familiar with the set flags, but yes, having looked them up that makes a lot of sense. As for the &> shortcut I was aware of that but for some reason I keep finding myself using the long form regardless. Old habits I guess :)

      Thanks Ralph, I’ll amend the script accordingly.

  • Dark Liquid 10:37 am on April 30, 2011 Permalink | Reply  

    Upgrading to Natty 

    I’m currently upgrading my desktop machine to Natty and once again it did the same thing that almost broke my last install – it trashed the gdk-pixbuf loaders.cache, making many gtk apps unusable. Considering the update program itself is a gtk app, this is not good.

    Luckily, I caught it doing it in the terminal of the update program, so I ran the following to fix it:

    Also, the update servers were so slow, I ended up downloading the dvd via torrent, mounting the iso and then using the cdromupgrade script on the image to do the update. getting the torrent was quick (600+kbps) but downloading packages manually was going at a max of 21kbps. Something to keep in mind.

     
  • Dark Liquid 9:24 am on April 30, 2011 Permalink | Reply  

    I get fed up occasionally of the million and one old kernels I have installed after multiple updates and upgrades of my Ubuntu install, so I have the following shell script snippet to sort it out for me:

    That pumps out a list of all the installed kernels and associated images/modules/etc apart from the one you are currently using. It also keeps around suffixed kernels at the same version as your current one (so if you have the -pae, -rt, -smp, -generic and -preempt kernels installed, it will leave the ones that match the version number of your current kernel).

    Now, if you want to be dangerous and not check the output, you can bin them all with a single command, like so:

    Et voila! Goodbye superfluous kernels!

     
    • Ralph Corderoy 10:08 am on May 10, 2011 Permalink

      Don’t think the searching aptitude needs the sudo?

    • Dark Liquid 11:33 am on May 10, 2011 Permalink

      I think you’re probably right.

    • Dru 11:26 am on March 23, 2012 Permalink

      :( It’s not working any more. I get this output:


      bash: syntax error near unexpected token `newline'
      Penny:~$ sudo aptitude -F '%p' search linux-{image,headers,restricted}-2.6~i | grep -v
      uname -r | sed -r 's/-[a-z0-9A-Z]+$//'
      bash: syntax error near unexpected token `newline’

    • Dru 9:57 am on March 25, 2012 Permalink

      Ignore that last comment, wordpress fucked with the code snippet, and inserted bits that shouldn’t be there.

  • Dark Liquid 11:41 am on March 6, 2011 Permalink | Reply  

    Another belated update 

    I’ve been neglecting my poor blog again and I’m sure the hoard that is all of maybe 3 people are clamouring for an update (by which I mean, don’t care).

    So what’s been happening in darkliquid-land? Well, knowing my memory, more than I can recall, but the things that immediately come to mind are:

    • Failing my 1st driving test
    • Buying a car
    • Getting back in PHP, primarily WordPress development
    • Hacking on C++ code for a minecraft map renderer
    • Playing various indie games
    • Playing some main stream demos, and enjoying them immensely
    • Watching TV and a few films
    • Discovering new music, primarily Ugress and side-projects
    • Hacking up my laptop’s partitioning scheme

    So yeah, I failed my first driving test. It was almost a pass, but I made a really stupid mistake halfway through that killed it. I have my next one on Wednesday and I was looking forwards to it and feeling confident until yesterday, which was one of the crappiest days ever.

    Yesterday started off with a pre-test driving session to get some practice in. What was practice turned more into a demonstration to myself of everything I shouldn’t do. Simple manoeuvres took 5+ attempts, I almost got the car T-boned on a roundabout, all sorts of retarded stuff. I have no idea where my head was at, but it certainly wasn’t on driving. Even coming back and parking up, I crossed traffic without indicating, waiting or anything and I know that’s completely wrong. I felt so utterly terrible after that, there was absolutely no redeeming features of that session at all, other than surviving it, which is one step up from getting myself killed in an accident.

    Following that, I accidentally deleted about 135GB of data whilst doing something for work and needed to spend time fixing it in yet another example of epic failure.

    Since I was on a roll of self-destruction and failure I thought, hey, I already feel shit, I can’t feel any worse so rather than crawl into bed and calling the whole day off as a bad idea, lets see what else I can fuck up, lets go for the record, eh? With that in mind, I decided to resize the partitions on my laptop.

    When I’d first set it up, I’d accidentally only allocated 40GB to Win7 and 560GB to Linux. I was using maybe 8GB in my Linux partition so it seemed like a waste, considering Win7 alone without much installed except for all the C++ game development bits was already taking up 30GB. The plan was to shrink the Linux partition and then resize the windows partition to be larger and shuffle everything around. Alas, it is never quite that simple.

    Because I am an early adopter/technology masochist I’d formatted my Linux partition as btrfs, for which there is currently no partition resizing support. Also, my Linux and Win7 partitions were not next to each other, so expanding one into the other wasn’t going to happen.

    So, my plan changed to imaging my Linux partition, then deleting it, splitting it in half, with one half for Linux and another half for shared files between Linux and Win7. To image it, I used clonezilla, backing it up with partclone to an external USB harddrive. That all went well, so I was happy. I then used a gParted live CD to repartition the disk and then proceeded to go back into clonezilla to reimage the partition.

    Except it refused to do so. Clonezilla refused to allow me to image a partition backup of sda6 onto what was now sda7. Hacking around, I essentially renamed the to be called sda7 instead of sda6 and then hit another problem. Since sda6 was 560GB and sda7 was 280GB, it wouldn’t image the backup due to the size difference, even though the image was only 8GB, since it was a backup of only the used part of the partition.

    This was getting ridiculous.

    So, after looking up various issues and solutions on the internet, I discovered how to run partclone manually to force it restore the image anyway, since clonezilla didn’t offer any facility for doing so through it’s own interface. I had to cat together the 3 sections of the image backup (it had been split into 2GB chunks), pipe that through gzip and then pipe that to partclone.restore with a flag set to tell it to ignore size mismatches. It promptly, much to my shock, started imaging without asking for confirmation at all. Note to self: partclone is a dangerous tool that will trash your disk if make even the slightest mistake.

    So, now the image was on the partition, I could even mount it from the clonezilla live CD. Woohoo! But of course, I knew there would be issues. The drive layout was different which would make grub go doo-lally. The entries in the /etc/fstab wouldn’t match up, so the system wouldn’t boot even if grub would work. Before I did anything, I made sure to use the btrf tools to make sure the filesystem was aware of the new extents on the disk, by telling it to expand to the max for the partition (considering only 8GB had been imaged). That seemed happy enough and I saw no problems. I’m not even sure it was necessary, considering it seemed to get the free space calculations right even before I ran that command, but better to be sure I suppose.

    Now, on the more modern distros, your /dev is basically empty from start so it’s an arse if you want to chroot into another OS and run things like grub or anything else that needs the block device files for the disks. So, I had to bind mount the clonezilla Live CD /dev onto the mounted Linux install, then chroot, and then mount all the other filesystems, etc into that (/proc, /sys, /boot in my case). Once that was done, I edited the /etc/fstab to be correct – I would have used the drive UUIDs, but it was an arse copying stuff from the blkid command into the fstab file on the console, so I couldn’t be bothered – I can do it later from within the OS when I have a graphical environment. After that, I needed to reinstall and then update the grub config and then reboot.

    And finally, after hours and hours of messing around, it was all fixed, working and I had a nice chunk of extra space for sharing stuff between Win7 and Linux. I could boot into Linux or Win7 without any problem. That gave me a nice boost after the horribleness that had been Saturday.

    Now, in Win7, I wanted to shift some stuff to the new partition to make space. Namely Steam, especially since I’d succumbed to the special deal offer the other day and brought Red Alert 3 + Uprising for around £7 and 16GB of game data was not going to fit on the main Win7 partition. Annoyingly, Steam has no way of storing games on multiple partitions and while I could use NTFS’ native symbolic linking support to make it use the other drive, considering Win7 doesn’t expose that functionality at all and you have to use a command line tool, I didn’t trust it. Instead, I followed the instructions on the Steam site and moved the whole install to the new drive, which seemed to work perfectly.

    I also decided to tweak my Win7 install and theme it using the Maverick Ambience theme dpcdpc11 on deviant art created, so make it as little Windows-y as possible. Hit some issues with that due to the SP1 update breaking the stuff I did last time I enabled custom themes. Had to run sfc /scannow to regen the files and then go through the process again. Now it’s all looking nice. Hurray!

    Going back to the topic of games, I’ve been playing the Bulletstorm demo quite a lot. Its immensely fun, completely over-the-top and very tongue-in-cheek, satirising it’s own genre. I’m not normally an FPS fan, but Bulletstorm is everything that is right and fun about FPSes all put in one place. It’s a shame they did some stupid stuff on the PC port – it’s clear they targeted consoles first and the poor PC gamers got a little screwed by silly issues. I’ve been playing the PS3 demo mostly, but had the chance to play an echo on the PC version and it was very fun, even if I didn’t know half the controls.

    I’ve also been playing through the Dragon Age II demo on PS3 and likewise, it’s very fun and looks very pretty. The combat system feel a lot more punchier now and I found mages in particular feel a lot more powerful, in the first game they felt very weak.

    However, I doubt I’ll be buying either of them for a while, I still have several games on my list to go through and without any free time to play the games, I’m progressing through that list very slowly.

    I’ve also been playing Inside A Star-Filled Sky, a cool little indie game that revolves around the concept of infinite recursion and the act of becoming. Each level is the creature that you later become, but also, you can enter other creates, power ups, even yourself and by changing things inside them, change their abilities and attributes back on the outer level when you jump back up. It’s similar in concept to the movie Inception with it’s multiple layers. I’ve also been playing a lot of fake-scrabble on my phone via DroidWords, which seems the best of the lot in that genre and is a very polished little application, far better than the bug-laden monstrosity Words With Friends by the evil company known as Zynga.

    Since we’re on the concept of media, let me move onto music and film/tv. I recently discovered the artist Ugress, a Norwegian grungy-electronica one-man setup and I fell in love with his music. It’s absolutely divine. He has several side-projects including Nebular Spool, Ninja 9000 and Shadow of the Beat. What’s especially cool about Ugress is the way he communicates with his fans and really leverages the new digital age. He tries to get on all streaming media platforms, he offers his whole albums for free streamed listening directly on his site and offers some albums directly for download free of charge. The albums he does sell he uses a pay-what-you-want model with a minimum value (sometimes, that minimum is 0). His music and his attitude to the industry is inspiring.

    Speaking of which, when I listened to Ruins by Nebular Spool I was captivated. I was instantly catapulted into a state of intense creativity. The album seemed to be the perfect accompaniment to a story I am writing with Archi Teuthis on Protagonize called Neon Bible. I was consumed with the idea of creating a story-focused 2D platform game as a prequel to the story – filling in the blanks of the journey the Priest at the start takes to the city of Eden and covering the hard choices he has to make, telling the story of how the world the game is set in came to be the way it is and challenging the player emotionally and intellectually. I only wish I had the time and skill to create it.

    I still have yet to see most films this year. I just don’t go to the cinema. The cinema sucks, mostly because of stupid costs, other people and other things. At home I can pause the film or eat a pizza whilst watching it if I feel like it. Not so at a cinema. I also don’t have to worry about retards ruining it by talking, or if they do, I can at least tell them to shut the fuck up and rewind to hear what I missed. Also, most films haven’t really inspired me enough to go out and pay for. I still want to see Tron: Legacy even though I’m fully expecting to be disappointed but even that is still not worth seeing at the cinema.

    Two films I watched recently on the PSN were excellent though. The Other Guys, a Will Farrell film satirising typical buddy-cop action films. Hilarious, clever, deeply sarcastic. Just what I like. I highly recommend it. I also watched Red, the film about retired assassins/agents. Very funny and sweet, the action sequences were amusing in their audacity, but not over done. So many films try so hard to be cool that they don’t realising that trying so hard is what makes them decidedly uncool. Red got it all just right. A friend lent me Despicable Me and I have to say it was dull and uninspiring. I wasn’t really interested in it at all. However, maybe it’s the potential father in me, but I found Margo adorable and the film is worth watching just for her, she is awesome.

    I’ve been getting back into my C++ coding recently by working on extending mcmap, a renderer for minecraft worlds. I’ve been working on adding support for displaying signs on the map (as in the text) and struggling with the coordinate system transforms, but I’ve more or less cracked it now, so it’s on to the more taxing job of extending the renderer to blend text at various positions onto the image. I’m thinking of just not doing that at all and instead storing the text and coordinate data in a text file so you can apply the text later with your own process (which I’ll probably write a Ruby script for). I’ve also been playing with OpenZoom, an open source implementation of Microsoft’s DeepZoom system in Flash which in many ways is superior to Google’s Maps offerings, especially for the task of showing large images rather than actual maps, which suits the output of mcmap much better, since it isn’t really a map in the traditional sense.

    I’ve also been brushing up on my PHP again having been digging around in wordpress writing numerous plugins and changing some base functionality of wordpress itself to support various things and add new features for a site at work. I’m definitely not  fan of PHP or particularly of wordpress. they are both messy and ugly, but having worked with them for a while now, I’ve been fairly productive; so using wordpress as a CMS framework definitely has its advantages in terms of productivity.

    I’ve not been roleplaying much recently. The game our GM is running has ceased to hold any interest for me. I’m not interested in the other players characters, the story (or lack thereof), the settings or the seemingly meaningless jumps from one pointless job to another. Nothing in it seems gripping or interesting or has any meaning to the characters. I’m not even sure why the characters are doing what they are doing, there doesn’t seem to be any clear motivation to the game at all and that’s why I’ve bailed out of it. I think I’m becoming a bit of an Evil Hat fan boy. Their roleplaying games always seem very focused around characters stories, are strongly character-driven, thematic and designed specifically to create group stories. The Fate system as applied in their Dresden files game is fantastic in the way it does group setting and character generation and I hope to run a quick game of A Penny For My Thoughts to get the others to put their toes in the water in regards to that style of collaborative play. I’m bored with the traditional hack’n'slash GM/Player separation – maybe it’s the writer in me, but I want to tell a story, through the medium of a game, not play a game set to a story. The others all come from this more traditional roleplaying background, so it will be interesting to see how they react.

    Well, that’s it for now. Maybe it’ll be another 3 months before I update!

     
    • eric_the_girl 6:32 pm on March 6, 2011 Permalink

      To be honest, I’m on my third driving test and have resorted to taking a Kalms tablet before driving lessons to see if it helps… I had the Potts woman as my examiner last time and couldn’t hack it in the air of disapproval.

      Good luck, though! I have my next one on the 16th….

      (Plus, they say that you are a better driver if you pass second time than first…)

  • Dark Liquid 5:23 pm on July 9, 2010 Permalink | Reply
    Tags: , , , x11   

    Resolution Quick-Change for Games in Linux 

    I play the odd game or two under Linux and some games really, really hate running on a dual-screen system (using Nvidia TwinView), Either getting the resolutions completely screwed or having their windows or fullscreens positioned on one screen or the other, but the content of those windows pushed to the centre of dual-desktop meaning half the game is missing. The same thing occasionally happens only with mouse input in these games, rather than the graphics, so I can’t move the mouse past the invisible divide.

    Anyway, fed up with this and wanting a simple one-click solution I knocked up a shell script that I run from a button on my panel. First of all, I did a simple toggle with xrandr, but found I’d occasionally accidentally click it and switch res, annoying the hell out of me so I knocked up a slightly less automatic version. It uses zenity to give me a choice of one of two resolutions (a single screen one or my dual-screen one) and then uses xrandr to apply the changes. I would just use the Monitors app, but it complains about not supporting the driver and I just wanted something click and simple (the Monitors app is slow to start, plus that annoying not-supported dialogue adds an additional click).

    Here is the script:

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel