Re: Testing strategy for Tk application?

From: Bryan Schofield (bschofield_at_users.sf.net)
Date: 04/09/04 

Date: Fri, 09 Apr 2004 02:40:09 GMT

Bryan Oakley wrote:
> BH wrote:
>
>> I'm working on a Tk application and the code is getting bigger
>> everyday. I'd
>> like to know how people go about testing Tk applications. Can the
>> testing be
>> automated?
>
>
> Absolutely. Tcl/tk is perhaps the ideal language for implementing highly
> testable graphical applications. After all, it has a built-in scripting
> language already in place!
>
> Now, before we go too far I need to explain the environment I work in.
> If all you do is work on one platform you can pick something like
> android or winrunner (I assume.... I've never actually tried it on a
> wish script). For my purposes I needed a solution that could work on
> windows, many flavors of unix, and macosx. Proprietary or
> platform-specific testing products were out of the running.
>
> My strategy is to build testing into the app. Specifically this means to
> add plenty of introspection commands, and to create an "API" that both
> the testing harness and the GUI use. My terminology for the API is
> "actions". Bryan Schofield independently came up with an almost
> identical concept with an identical name and identical code, which he
> posted to this group back in November 2003 [1]. Actions have more
> benefits than just testing, so they are definitely worth considering. In
> fact, his comments in comp.lang.tcl didn't even mention testing IIRC.
>
> The idea is, your GUI should be nothing but a bunch of widgets that call
> an API. This may not work for legacy apps, since often buttons have
> their functionality hard-wired in. This is a bad practice, but we can't
> control what our predecessors have done...
>
> Once you have this API, your tests can call it directly. This can be
> used to test the inner workings of your code. It doesn't test the UI per
> se, but rather what the UI is designed to accomplish. A test case might
> look like this:
>
> action invoke focus main
> ::test::input "Hello World"
> action invoke select 1.0 1.5
> action invoke copy
> action invoke moveto end
> action invoke paste
> set actual [action invoke getData]
> set expected {Hello WorldHello}
> if {$actual ne $expected} {
> puts "unexpected result: $actual
> }
>
> This oversimplifies things a bit, but I think it illustrates the point,
> which is that most user actions are implemented as tcl "actions" which
> really are nothing more than glorified procs. With the addition of some
> helper procedures (::test::input, for example), you can write complex
> scripts that exercise almost all the functionality of your app. Almost,
> because this scenario can't simulate mouse presses on the window manager
> controls, it can't work with native windows file selection dialog, etc.
> For me, having a testing strategy that works for 90% of the code is far
> better than having no testing strategy at all.
>
> That leads to the question "fine, but how do I run the test?". At my
> company I recently wrote a small test harness that is about 250 lines
> long. All it does is define some helper procs (eg: input, pressButton,
> selectMenu, etc), sets up the environment (eg: makes argv and argc be
> what it would be for normal operation), source the program to be tested,
> and then read a test script from stdin. All of this with just 250 lines
> of code.
>
> We then run the tests by writing a small shell script (due to historical
> reasons; this could easily be another tcl script, perl script, tcltest
> [2], etc) that runs a suite of tests and logs the results. In tcl, a
> trivial example of this script might look something like this:
>
> # script to test "MyApp"
> foreach test [glob -nocomplain tests/*.tcl] {
> wish test.tcl MyApp < $test >> test.log
> }
>
> This yields a very powerful yet simple mechanism for testing the GUIs.
> You can write scripts that directly invoke the functionality (via
> actions), but you can also directly invoke specific widgets, generate
> keystrokes, compare actual versus expected values, etc., all with a
> scripting language you are already familiar with.
>
> The only thing this scenario doesn't do is verify that what is on the
> screen looks right. There are test programs that can run scripts, make
> snapshots, and compare the snapshots. You may want to use that, though
> I've found it's not too onerous to do that manually. This is really just
> a small piece of the overall testing picture; after all, the appearance
> of buttons don't usually change. For the software I work on, I have a
> set of manual tests I run where I can visually inspect each GUI.
>
> On a final note, you might want to investigate the testing mechanism
> tcl/tk itself uses. tcltest [2] is very powerful in that it makes it
> easy to define constraints, match output against fixed strings or
> regular expressions and so on.
>
> [1] http://groups.google.com/groups?th=ddf110816a167ca9
> [2] http://mini.net/tcl/tcltest
>

I'd like to add one other thought... Once an application that I've
written has become sufficiently complex to point where I've done all the
unit level testing that I can do (I favor tcltest and the "action"
concept just as Bryan describes), I'll embed a built-in debugger. The
debugger is, of course, hidden but can be revealed with a secret command
line option or through obscure key or mouse binding...like
Control-Shift-Button-3 on an innocent looking label. Since a majority of
the applications I write are clients of some sort or another relying on
data provided by someone else, having the built-in debugger comes in
handy for diagnosing oddities.

The company I work for is quite large and the product I work on is
supported by about 100 engineers in about a dozen different groups. Each
group has their own priorities, and often it's easy to misdiagnose
problems. Having the embedded debugger allows me to investigate the
problem as it happens on any machine (and platform) the app runs, with
no fuss and find the root cause quickly. Sometimes it's my bug, but most
of time I find there's problem in the data be pumped in. Catching before
someone submits an internal bug report to me is so much easier than
spending half a day trying to reproduce it. The point I'm trying to
make, is that no amount of unit testing can protect you from all the
weird scenarios that can happen in a fielded system where the inputs are
unreliable and having a way to figure the problem is invaluable.

An embedded debugger can be as simple as an entry widget to enter tcl
commands in and a text widget to print results, or be complex, highly
customized solutions that give graphical representations of the state of
application. As an added bonus, I usually get an "oh wow..." from the
suits and other developers when they see me whip out the debugger on
*their* system.

Anyway, that just something that I've come accustomed to doing, and it's
saved me so much time in the past, perhaps it is a fit for you, perhaps
not. Best of luck.

-- bryan schofield