Friday, 25 December 2009

Reverse engineering a multimeter's language

Some time ago I got myself checking all the gadgets in our lab, to see which ones can I connect to a computer. This is mostly because I like our gadgets and I like our computers, but together they are just that more awesome... Been checking oscilloscopes, function generators, counters, even high power lasers. Most of them seemed to have a decent (well, at least usable) manual, so I could do some basic communication right away, until I come up with some more complete awesome plan to automate the lab. But there are always exceptions to the rule, even if tiny ones.

There was this Pro's Kit 3PK-345 Digital Multimeter (pictured on the right), that could do RS-232 serial communication. Of course, this is not the most crucial equipment of them all, but could be quite useful for rough-and-ready monitoring and logging, since it has so many different measurement functions.

The problem is that there's no clue in the manual, how to communicate with it, what are the settings, what's the "language", are there any quirks... There was, however, a little piece of monitoring software on an attached CD-ROM, that was able to talk the multimeter. But it was a very basic piece of garbage program. I think it was really just thrown together, and I base this theory on the fact that the program icon is the default Delphi project icon (that one on the left) - I know it because used it like 10 years ago for a while (oh, my, even for some pocket money, but that's another post). And if you don't bother to change the icon, I don't have much confidence in the rest of the code...

But at least it worked. At least there was an example, how I should do it, and could set the hounds serial sniffers on it and check out what are those two talking about... That was another adventure, serial sniffer... Tried quite a few under Windows until found the right one giving all the information I needed. But after all the testing I don't even remember what it was, maybe an evaluation copy of a paid software.

From the sniffing, I got the initialization:

Baud rate 600
RTS off
DTR on
Data bits=7, Stop bits=2, Parity=None
Set chars: Eof=0x00, Error=0x00, Break=0x00, Event=0x00, Xon=0x11, Xoff=0x13
Handflow: ControlHandShake=(DTR_CONTROL), FlowReplace=(), XonLimit=0, XoffLimit=4096
DTR on
RTS off
DTR on
RTS off

That tells me most of the settings, plus that the multimeter needs that few extra DTR/RTS cycles (indeed, without those I couldn't get it to talk). Another thing the sniff told me, that multimeter works in request/reply mode, that is I have to initiate a reading by sending the sequence "D\r" (\r is the "carriage return") to the device to which it will reply.

Now, let's get down to business. Using PySerial, it is very easy to set things up:
ser = serial.Serial('/dev/ttyUSB0',baudrate=600, bytesize=7, stopbits=2, \
       parity=serial.PARITY_NONE, timeout=1, xonxoff=1, rtscts=0, dsrdtr=1)
line = ser.readline()

  1. This is on Linux, but works on Windows too by replacing /dev/ttyUSB0 with COMx, where x=1,... the appropriate serial port number
  2. I just have to write "D" to the port, because it will automatically attach the carriage return
It's time to decipher the language. Trying the different settings, the multimeter's reading is returned as a 13 character string.

  • First two characters tell the type of the reading, though one needs the unit (last part of string) to completely determine the type for current and voltage measurements. It's followed by a space
  • Five characters contain the reading value (signed).
  • Last 1-4 characters tell the units of the measurement.
The more or less complete catalog is the following:

### Output (13 chars) and Line ending (+1 not shown: \r /carriage return,0x0D/)
DC -0.000   V  (DC Voltage)
AC  0.000   V  (AC Voltage)

OH   O.L MOhm  (Resistance, ohm mode, no connection)
OH  0.008kOhm  (kOhm measure)
OH  080.8 Ohm  (Ohm measure
OH   OL.  Ohm  (Resistance, short mode)

DI    OL   mV  (Diode)

    0000       (hFE /Forward Current Gain/ mode)

TE -  OL    C  (Temperature, without connection)
TE  0024    C  (Temperature, with thermocouple)

CA  0.011  nF  (Capacitor, 4nF scale)
CA  000.3  nF  (400nF scale)

DC -0.000  mA  (DC current, 4mA scale)
DC -000.0  mA  (400mA scale)
DC -00.00   A  (10A scale)
AC  0.000  mA  (AC current, 4mA scale)
AC  000.0  mA  (400mA scale)
AC  00.00   A  (10A scale)

The main annoying thing I found, that there are a number of different notations for "over limit" or invalid values, such as "O.L", "OL.", and "OL". And here it is a "capital-O", not a "zero", thus it needs extra checking in the code, not just simple conversion. Nevertheless....

I wrote to the company as well to ask for some inside info on this "reverse-engineered" language, and they are promised to give me a complete reference. Yeah, "getting back" to me since the end of August. Fortunately they help is probably not needed anymore, I just wanted to double-check it, and also to test them how do they respond to such request. Initial impression was very favorable, but that has kinda decayed since then...

Now, because the language is so simple, it is very easy to implement the code "speaking" multimeter. A very bad version Python version can be found in my GitHub repo, where I keep all the other hardware related stuff as well. I seriously should get some Bad Code Offset for this one, but at least it works so far.

Sunday, 29 November 2009

Offline media

Today I went Magazine shopping to one of the 24H open Eslite stores in Taipei. It's a fun place to go, in a multitude of ways:

One big floor for books & magazines.
Two big floors for designer stuff, gadgets and artsy things....
Two big floors for food, with one of the ever best cake stores I have every eaten (Awfully Chocolate).
One final floor with music, stationery and toys....
(Oh, and today a concert in front of the store by a band Wonfu Loves You [旺福爱你])

Anyway, went there to get my monthly / quarterly dose of magazine, since we still love offline media, even if most of my knowledge comes from online.

Make Magazine - Volume 20

This was the primary objective was this one, the new issue of Make. The last one was about DIY Drones (automatic planes:) and got me fired up enough. Didn't get too much done about them, but definitely made a few entries to my Someday/Maybe list. Hopefully with some time it will be even better, it's all a process... Anyway, how can I resist if one of my heros is on the frontpage: Adam Savage from MythBusters. :) He's someone who can really spread the love for science and critical thinking and tinkering... 

You know about the "20% time" at Google? All the developers having 20% of their time to develope their own ideas, completely free. How about having something like for physics researchers? 20% of the time for tinkering and developing things that are necessary connected to the current project of the research group, but worthwhile anyway?
Maybe a bit tougher sell, because we (should) already do our own ideas - that's what research is about, isn't it? But still, there are always room to improve....

Anyway, if one is interested, there's also Instructables, no more excuse to be idle. Coming up later: reviews of Make: content.

Wired UK - Ideas Issue

In the section where Make: was I found some other goodies too... Wired is one of my favorite only magazines, but couldn't really justify buying the paper version if I have already read most of the online contents. But! Behold, there's a new UK edition as well! And they had a special issue on big ideas.... Well, really couldn't justify getting all 3 of them, but let's get one. Always looking for inspiration from those people cleverer than me. :)

Reading through it, I really admire the whole design and execution. Still "magazine" meaning relatively disposable content, so it's not going to be very long-lasting for me, but definitely high quality brainfood. Gadgets that I start yearning at first sight, architecture, everyday chemistry, technology, company insights, people profiles, and much more... And a lot of advertisement that (almost) works on me... :P

Not surprised about the awesomeness... After all, being a writer for Wired is one of the entries on my Dream Jobs list... And you know what, this still can be done. Astronaut, probably not anymore.

The Celebrity Twitter Directory

However there are magazines there that feel like a complete ripoff. How about The Celebrity Twitter Directory? Didn't people know about e.g. WeFollow, or Twitter Lists? Why would someone pay more for this thing more than my Make: and Wired combined is beyond me... But certainly Twitter is one of the hottest thing out there right now: Wired had at least 2 proper articles in connection with it  (about 6 pages), while the only Facebook related thing I could find was the page-bottom tiny text  ticker of the biggest FB groups this month...

There are still some magazines that I would have chosen on an unlimited budget.... The Economist, Nature, Science, Newsweek, Scientific American... Have to check out the library, maybe they have them. And at the university campus where my office is, seem to have quite a few "magazine ladies" who try to sell subscriptions to a large number of these. Pretty good deals as well. Maybe treat myself for Christmas? Got to think about it....

Sunday, 22 November 2009

Fixing someone else's code

All started with another round of "I'll organize myself!". This time get my books in order. Maybe music and movies as well, but books first. So been looking around the interwebs and the Ubuntu Software Center [sic], to find some suitable candidate for the job.

Alexandria popped up and seemed to have great reviews. It only does books, but seem to do it the way it should be, they said. Giving it a test-ride myself - man it was painful. The very first thing: it should be able to add books from Amazon (and other collections too, but those are tiny compared to that). Well, it doesn't. Just throws big hissfits when trying and that's all...

Now, instead of going and trying out a different program, what did I do? Grab git-svn, and check out the code. It's all ruby in the end... Poking around for about an hour, it turns out, that the library Alexandria uses processing Amazon's XML-formatted answer (Hpricot) just do some very very stupid things. It completely borks even simple XML structures. Not sure how this got through any testing, but probably it got through because there's no testing... And the problem is there for months, and even if there's a new version of Hpricot on Ubuntu, it's available for the next version (10.04 Lucid Lynx) but not the current one (9.10 Karmic Koala). Even Debian (Ubuntu's "daddy") has the new version...

So, there were three options that I could do:
  1. Just wait until someone comes up with a solution
  2. Repackage Hpricot for Ubuntu Karmic
  3. Change the code for something else than Hpricot
Comments on these in turn:
  1. Sure, but then where's the fun?
  2. Could do, but I'm pretty annoyed with the Ubuntu packaging system now, because I don't know it well enough to do even simple things. Tried, but failed. It's for another post, but it's a but too versatile and complicated for me at the moment. Keep trying some time, though....
  3. So this is what we gonna do, especially because there's another library, called Nokogiri. Seems to be pretty similar, except it was working.
Getting down to business, there was still some problem... Don't know much about XML (except what it is and more or less how it looks like, but much less about terminology and such, which kinda breaks "googleablility"). Also, Nokogiri didn't seem to have much of a documentation. The wiki on their github page was giving helpful suggestion to look around StackOverflow for nokogiri tagged questions. Not ideal....

Anyway after a few hours, it was working just fine, I could add books, I could add books with multiple authors, it had the pictures and all the goodies... Not a good-looking fix since there's still a lot of code there which I don't understand the purpose - since I never seen it ever properly work before... But it was a start. Happy, commit the changes, clean it up later.

Then I checked the original bug-report again, and someone just made Option 2 into reality, thus making all my efforts pretty much obsolete. Tried, and works...

Now, lessons from the story, and this fine weekend:
  • XML seems to be fun - once understood, and a lot of possibilities to interact with websites if one can use it well. (you know XML and JSON and can handle most sites)
  • Ruby seems to be fine but it's not different enough and too different from Python in the same time, that I don't yet see the point of trying even deeper. No offence, though.
  • While so far I found Python-related projects have very extensive documentation, these to days I'm yet to find any comparably well documented Ruby project. It might just me, but the culture seems to be just a tad different.
  • If I ever have some time, might do some benchmarking and if Nokogiri is really faster than Hpricot, might rewrite that code a bit better and see what those guys say.
  • Even when this one bug was fixed, run into so many others. Not showstopper, but bad enough not to recommend using Alexandria just now. Maybe if the development picks up a little more...
  • It's interesting to check out other people's code and even to be annoyed by it. That actually shows where I should improve as well. There's plenty of that....
  • ... and lastly: maybe I should re-check my priorities, and spend my next weekend in a bit more useful. :P

Thursday, 19 November 2009

Bad Computer Day

I think I'm getting a bit old geek now, or I have too high expectations? Many days I just hope the computer I us would just work. Amazing, these days all three systems I use regularly have "Critical" or "Showstopper" level bugs that I cannot seem to be able to fix (yet).
  • Windows XP SP3
    For a while now, the system keeps being unresponsive, svchost.exe crashing and throwing "Access Violation" errors whever the machine is started. Well, 2 times out of 3. It's enough of a problem that makes the computer really-really annoying to use. Some people suggested it is because the machine is "too old". I call BS on that one.
    Searching around there was some advisory from Microsoft, related to a problem of the AutoUpdate. Oh, great, all those auto-updating programs at startup...
    It is pretty much what is said at KB927385 (You receive an error message after a Windows XP-based computer runs an automatic update, and you may be unable to run any programs after you close the "svchost.exe - Application Error" error message dialog box) and they have a few full page of solution, most of it is in the console. Great. The Windows version I have to use is in Chinese, so even greater. Never mind, GoogleImages is actually quite helpful in figuring out what setting means what. (related note: how much I wish for copy-pastable text on Windows dialogs and settings windows....) Okay, done that, the services.msc, the REGSVR32 the net start *whatever*, the everyting... No good...
    Well, today I found some related advisories that I have to try, like KB932494, but it is still stressing out.
  • Ubuntu 9.10 (Karmic Koala)
    I really want to love Ubuntu, I want it to succeed, I want to become abundant... That's why I was really cheering when I've found the Windows Ubuntu Installer (WUBI). The promise is great: people don't even have to jump head first into Linux, and they don't have to know years worth of Linux admin training to set up dual-boot.... WUBI downloads the correct CD image from the web, installs it on a windows partition and sets up proper dual-boot. Perfect!
    Except, that some kernel or grub (the software that handles the boot process) updates keep making my system unbootable... What the hell? The machine works fine, run the updates, everything finishes fine, reboot - nothing. Sometimes even worse than nothing, don't like Kernel Panics at all, that's no better than the Blue Screen of Death... Not helpful, you are....
    Done this three or four times: install, update, broken. Reinstall update, works, update, broken.... and so on.
    Have nice little bug-report submitted, but the people who reply are in two categories so far: users who try to save their data but clueless about fixing the problem, and people who supposed to be maintainers but their comments are just plain wrong most of the time, and if even I know that, it must be very obvious... Yesterday that happend again. Not sure if I'll try to reinstall once more, or just give up for a while until there's some fix. Argh...
  • Arch Linux
    My favourite "geek distro", that I enjoy using a lot, but they have a track record of doing updates that break lots of people's computers - for the sake of cutting edge. Many times I don't mind too much, and they have a very responsive Forum, but sometimes it's still just plain bad.... The latest grief is that some latter day update made my laptops touchpad unusable. In the logs I can see some complains of psmouse.c about the Synaptic touchpad, but after "trying to reconnect" there's nothing... I can plug in a USB mouse and that works, but come on!! That's why the touchpad is there, and it works under Windows just fine, so not some more hardware break....
This makes me quite a bit frustrated. A bit more frustrated with Windows (as they do not have a proper bug reporting system that I know of). But things break down for the Linuxes as well: the usual argument is that if it's broken you can dive in yourself and fix it. I think in this case the Arch Linux problem will be like that when I have the right mood. But the Ubuntu thing - it's just so random, and so crippled and I so don't know enough about that part...

And I didn't even mention that my EeePC just does not start - lights are on for the wireless and the power but nothing at all...

Maybe I just go out for a while and read a book or something....

Sunday, 7 June 2009

some thoughts on open source project deaths

I've been reading Joel on Software for quite a while now (few years back, as much as I can remember) and he had a post in 2000 that seems to be still the most popular read: Things You Should Never Do, Part I.

He was saying:
[Why Netscape died? They committed] the single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch.
The argument goes that by writing from scratch you put yourself at disadvantage at many different levels.
  • Don't have any code to ship while rewriting, that can be a long time (I might add Hofstadter's Law to be considered). You lose customers/users and give an advantage to all your competitors.
  • You are basically choosing a level of technology that you are aiming at, and cannot keep up with the new developments while you are writing.
  • You lose the advantage of previously fixed bugs, thus potentially committing the same mistakes again.
  • You throw out a lot of knowledge as well, good algorithms, solutions, etc....
So, why do I think about this? Recently I had at least two open source projects that I liked but which run into similar silliness...

Exaile was a quite good music player (for my Linux system), had syncing with my iPod, internet radios, album art, music collections, plug-in system... Almost everything I needed was in their version 2.0.... But there were plenty of bugs and there were some issues that seemed to be hard to solve because of the design of the code. I thought there could be a little bug-fixing (yey, Exaile is written in Python) and it could be better and better, while keeping it's light weight. I was so wrong. By the time I started to use the program the developers already moved on and for version 3.0 they wanted to REWRITE the whole thing. Asked them about fixing bugs in 2.0. They said the 2.0 code is all but dead, nobody is working on it, so I should consider instead contributing to 3.0. The problem is that it is not very functional, even after years. There are so many features missing compared to 2.0, that I just gave up on using it (now rocking with RythmBox, but don't like that much)... Maybe I'll go back, but I'm still disappointed.

The other project was Mitter, a desktop Twitter reader-updater. Also written in Python, lightweight, easy to use, quite convenient. I really liked version 0.4.5. Then they decided to rewrite the whole thing once again. That much that at the moment it does not even run. They use version control, different branches for developement, but the top level "release" branch does not run... How useful is that? Since I liked to use it, I did a fork, and started to add a few changes to version 0.4.5 myself, like sending proper "in reply to" information, of spell-checking with gtkspell, and been trying to debug it so it can run on Win32 well too (just have to clear up some threading issues). These are all gradual changes and all what they seem to do can be done gradually too. I'm a bit said that it's like that, the Mitter developers seem to know their trade pretty well, and I'd love to learn much more from them by working together. But I don't want to wait for the whole interface to be fixed just to have "in reply to"....

Am I too impatient? Unfortunately my attitude to programming is as of puzzle solving. There's a problem that needs a solution. When the solution is found it is not very interesting anymore. That makes me a pretty bad maintainer, but and enthusiastic (even if not very good) coder. But seeing these projects commiting the Netscape-error again, and potentially losing users and developers over it - that's not much fun. I have no influence over what they do, and the "advantage" of open source is that, just as I did, one can fork a project and try to do a better job. And in the meantime everyone loses....

The availability of better (more user-friendly) programming tools and languages, tutorials, information and means to collaborate made it easier to make some amazing applications. But it didn't make people to be better programmers, unfortunately... Let's see of these two can achieve their "second-coming".... Hope so because once I felt a lot of affection for them. :)

Saturday, 16 May 2009

facebook's walled garden is still walled with open API

I'm not sure why, but recently got on Twitter again. But I don't like to keep a website always open in my browser, much more convenient to have a desktop app do the useful work for me. Personal preference, I guess... Going minimalistic, I found a reasonably nice one for the job, called Mitter.

Then I got thinking, since I spend much more time on Facebook (FB), why not have the same thing for that? An app that let me see my friends' status, comment on them, update my own, see their comments on the update, these kinds of things.... FB has its own API, so there should be so many opprotunitiesn to do just this. Well, cannot say I found many... I heard that Tweetdeck will have some integration, but that's just too big for me - don't want to install all that Adobe AIR at the moment for this (and wouldn't have that much space for that on my EeePC anyway).

So what should we do? Check out how to make one ourselves... Actually, switching out Mitter's internals and replacing it with FB calls could work - in theory. But replacing something written by another person is always more complicated than one thinks at first. That leaves "from skratch".

Next step, check out the FB API documentation. Yeah, status.set is for example the first thing I want to be able to do. How to get around to do that?

Well - nothing is as easy as it seems. Twitter lets you use any app you desire for any purpose. You can just hack together your own and distribute it, and it will be just fine....
FB, however wants to control exactly what happens, who have access, and how much. And in the process, it makes certain apps impossible.

To illustrate what I'm talking about, here's the process I figured out, how to use a simple desktop app, writen in Python, to update my status:
  1. Create a new application on FB - Developers. When done, set its type to "desktop".
  2. On your app's page, note the "API Key" and "Application Secret" (Secret Key). The API Key identifies your app, the Secret Key is needed for your app to make API calls. As much as I could figure it out, this is so that your app does not get hijacked by someone just by knowing your API Key (that is public). But the need for this Secret Key makes all the difference (more of it later)
  3. Setup the permission for this app to update your status. See Extended Permissions in the wiki. More specifically, point your browser to a special web address, and set the permissions there. After that you can remove permissions from the usual settings tab, but there at the settings there is no list of all available permissions, only those that your app specifically asked for. I guess "publish_stream" and "read_stream" should be enough most of the time.
At this point you are ready to start coding and hope for the best.

Now the things that bug me:
  • Since you have to keep your Secret Key, well, secret (by the terms of service as well, I think), one cannot really make an open source desktop app - it wouldn't be able to make any API calls without the key... In the wiki some people argued that even closed source apps are safe at all - just because you compile something it does not mean that hackers cannot rewerse engineer it... What could one do?
    Suggestions include e.g. to have a web interface for your desktop app that would handle the login and the login only. This is doubly annoying, since I didn't want to write a web app, but a desktop one, and the login authentication will be still done by FB anyway - so a lot of complexity for nothing.
    Other suggestion is that all your users have to have their own version of the app, with their own API and Secret Keys, and everything... That pretty much rules out all of the non-geeks, and say bye-bye to any branding or community dreams...
  • The login is always handled by FB. No way around it. Have to have a browser, your app has to go to a special address and get some info back from FB (a "session key") to be able to operate...
Well, so far this much, since I didn't get much into things yet. But these are already enough to put me off a little...

Never mind, at the moment there's a very tiny and badly written, but functioning version of my status updater minibook. The source can be found at the minibook github repo. Does not do much, log you in, able to send status updates. It is badly written because it's basically a gtk.Entry example, modified. So any comment is appreciated, but don't think it will stay like this for long. If you feel like you can even fork it and help making it better... At the moment it works under Linux, but haven't tried under Windows. Will try that next time I get around to boot into Windows ;) .

In the end I got my twitteresk status updates, from my own app, but only with jumping hoops and not really being able to share the results with many people. Just following the usual motto: "Why do you do it? Because I can."