Programming Analytics

Cleaning up SQL Server 2008 Simple Databases

We make lots of databases here at the office.  Often they’re play databases, or restore databases, or test databases.  Since they rarely progress from development status to production status, they need to be set up with simple recovery model so they don’t collapse under the weight of the dozens of gigabytes of transaction logs my programmers pump in every day (at least, until Western Digital’s factory comes back online and we start buying hard drives again).

I was annoyed that my scripts were hardcoded to shrink transaction logs on a database by database basis.  This meant that anytime a programmer added a new database and didn’t tell me, that one could grow and fill up the disk.  So instead I wrote a single stored proc that resolves all transaction logs at one go:

DECLARE @Script nvarchar(max)

SELECT @Script = @Script + ‘USE ’ + QUOTENAME(d.name) + ‘;

DBCC SHRINKFILE (’ + QUOTENAME(f.name) + ‘); 

‘

  FROM sys.databases d

       INNER JOIN sys.master_files f ON f.database_id = d.database_id

 WHERE d.recovery_model_desc = ‘SIMPLE’

   AND d.state_desc = ‘ONLINE’

   AND f.type_desc = ‘LOG’

   AND d.name <> ‘master’

   AND d.name <> ‘tempdb’

   AND d.name <> ‘model’

   AND d.name <> ‘msdb’

IF (@@ROWCOUNT > 0) BEGIN

 PRINT @Script

 EXEC (@Script)

END

Why does this work?  First, we use the SQL server internal data structures to identify databases with simple recovery which are currently online.  Then we exclude the system databases which shouldn’t be part of this process, and prepare a SQL statement to call DBCC SHRINKFILE() on each developer’s database.  SQL provides the nifty ‘EXEC’ ability to run that query - and you’re off to the races!

Social/Phone Gaming and Bugs

With the comments from some analysts today that social gaming is flat or unsustainable, let me mention something that may be clear to people on the inside but might not be visible to the outside:

Social games, and mobile / iPhone games, are designed to exploit platform bugs.

What does this mean?

• Some facebook games became popular because they exploited the ability of applications to generate unlimited spam on facebook feeds, when facebook was becoming popular.  They gained free advertising exactly at the moment when people were receptive to it.
• Some iPhone games exploited the difficulty users had navigating the iPhone app store.  The apps that were in the “top free” lists generated lots of eyeballs, and eventually lots of conversions, simply by being near the top when people were searching.  It’s much easier to settle for the top app than to sift through page after page of stuff.
• In the past, old mobile phone games and services created phenomenal amounts of revenue by tricking phone users into texting something to a shortcode.  They then enrolled the phone subscribers into a monthly service that they didn’t understand, and they benefited from the fact that phone bills are hard to read - many of these customers didn’t know why their costs increased.

All of these bugs - and many others - are in the process of being resolved by the platform owners.  When these bugs are corrected, games will have to compete in an entirely different ecosystem than the one they were familiar with before.  Their process of natural selection will suddenly have to adjust to different fitness criteria.  Some games may persist, other games may not.

What many companies do, however, is constantly push the edge of the bugs and attempt to refine their viral appeal.  This kind of A/B testing and focus-grouping generates the highest possible effectiveness out of every exploit, but separately publishers still have to generate the kind of lasting appeal that builds a franchise people want to play.

Reed Hastings Claims To Be Moneyball

Multithreading a User

Programmers spend lots of time writing multithreaded applications that can get work done while the user is thinking, or reading, or whatnot. We know, almost instinctively, how to make the program fast. But we often don’t think about making the user fast. Why?

Well, the first reason is that computer science classes often say “Waiting for I/O is slow;” and leave it at that. The implied conclusion is that the program should be doing real work while it waits for the user to decide what to do. Often this makes sense, because a person thinks on the order of 100 milliseconds or so, and a PC can compute millions of instructions during this time.

But let’s use a real world example here: try depositing a check at an average ATM. Here is how it works:

- Click deposit
- Click check
- Insert check
- Confirm check amount
- Click confirm deposit
- Click receipt yes or no

In between each of these steps, the ATM has to contact the server to get approval from the bank to proceed. This means each page takes 300ms round trip, then 150-250ms for me to make my choice, then another 300ms for the next page to load. The end result is glacial.

Contrast this to Amazon’s checkout page. You click checkout, and Amazon selects all your default choices and presents you with a confirmation screen. If you want to edit anything, you click on the “Change This” button by each section. The end result is that an order is easy.

Why does this work so much better? There are a few principles at play.

- Show the user all the necessary information at once. Don’t hide necessary information behind a second button click. Organize the information in a clear fashion and hide anything not immediately useful.

- Present sensible defaults and allow the user to change them. Keep in mind that it’s very tempting to steal from your users using defaults: if Amazon did something sinister like add an opt-out “Donate $1 to Cancer Prevention” checkbox, or worse, the end result could be a major lawsuit.

- Gauge how much choice is necessary. Perhaps you don’t always need to ask if the ATM user wants a receipt; it might be better to show a button on the login screen that says “receipts: on” and allow the user to click to change their settings, which they will probably only do once in their lifetime.

Too many surveys!

If you visit enterprise hardware and software vendor websites frequently, there’s a good chance that you’ve been sent a survey. They often appear as a pop-behind window that detects when you leave the main website and asks you a few dozen vague questions. These questions give you a 1-10 point scale to grade them on such topics as “is this site easy to use?” or “how does this site match your expectations?”

Perhaps there is something else going on behind the scenes, but these questions don’t seem useful for improving a website. What change would you make if your site scored a 6.3 average on “matches expectations” that would be different than the one you would make if you scored 8.2?

This is the a similar mistake to that of Microsoft’s statistics focused design of Windows 8. Statistics and design don’t interact in the obvious way. You don’t survey all the people who walk in your company’s front door, ask them their favorite color, and paint the door the average of all the colors everyone picked. Instead, try picking a philosophy first, then using statistics to gauge whether you met it. Decide on a reason why the door should be blue and use a solid metric that is objectively correct to measure it. Perhaps the door to your company is meant to make the club feel exclusive to insiders, or to invite passerby to walk in. Then you can measure the number of new or returning customers to determine whether your ideas are right.

The best example of this is the wrongheaded way Microsoft is thinking about the “Task Manager” in Windows 8.  They surveyed everyone who opened task manager and figured out what they did.  This number is useless.  Let’s examine why:

Looking at the data and talking with customers, we determined that the most common usage of the tool was to simply end or “kill” an application or a process.

The reason the most common usage of the tool is this is because, historically, this has been the key purpose of Task Manager, a feature that was not provided anywhere else in Windows.  They had a few other buttons: ”Switch To”, ”Arrange Windows”, “Cascade”, “Tile”, “New Task”, etc.  But during the Windows 95 switch, cascading, tiling, and arranging windows became superfluous since it was now better handled via the Windows taskbar.  Starting new tasks became the function of the Start menu.  There was no need for the Task Manager to do anything but end a task that had become stuck.

So if you survey people to figure out “What do they do here?” you will get an answer that is dependent on what you trained people to do in the past.  That’s bad statistics.  That results in lame design like Windows 8’s “Fewer Details View” - I have never seen a more useless dialog box in my life.  Why should it exist?

If you instead hire a good UI designer, the designer will ask you “What feature is this dialog intended to accomplish?”  You then realize that Task Manager exists to find a problem application - one that is stuck or consuming resources - and to allow the user to kill it; this is a function that no other part of Windows performs.  The UI designer will next ask you, “What information is needed to allow the user to make that determination?”  

With this task in mind, you can research the list of information that’s helpful to the user, and you can then evaluate other feature proposals by how well they match the goals set forth by the UI designer.  Should we add a new button?  Does that new button complement the purpose of this dialog, or does it just add needless complexity?  

When all is complete, you can design a survey to discover if customers understand the goal of the dialog, if they found the information they needed, and if they were able to kill the task that was misbehaving.

Toys for Bob produces: Skylanders!

Two of my videogame heroes, Paul Reiche III and Fred Ford, have developed a fantastic new game series: http://www.skylanders.com/. Given how many classics they’re responsible for (Archon, Mail Order Monsters, and Star Control II, among others), they are way overdue for a massive hit. These guys deserve their success.

Turbo Pascal

I learned serious programming on Turbo Pascal, on the advice of one of my father’s friends.  This was in the mid-to-late 1980s - maybe 1987 or 1988.  I had experience with Microsoft Basic - yes, the last program Bill Gates wrote - and was doing okay there, but I knew I needed to become a better programmer if I was going to write video games.  So I learned Turbo Pascal, and I even learned how to do inline assembly on the advice of Richard Garriott.

Just saw this come up today: Things Larger Than Turbo Pascal v3.0

Let this be a reminder to you that we did things differently in the ’80s.  

I was terribly happy when I modified my sprite drawing code to be 10 bytes shorter by using REP MOVS - of course, I wasn’t in the same league as the big game developers but it was good for an amateur.

My father was proud of a great text-mode printer firmware he wrote in the early ’80s that did run-length compression in the buffer.  He used a special code to indicate the beginning of a run-length value. The three bytes 0xFF count byte meant that the buffer stored count repetitions of byte; when the code saw 0xFF 0xFF he had special logic to output the byte 0xFF. I pointed out one day that he could have represented the marker as the combination 0xFF 0x01 0xFF - which would have eliminated all the special conditions and shrunk his firmware.  We cared about such things!  Nowadays we just pass it through GZIP and dust off our hands.

Programming hand-offs

Doesn’t it suck when you build something great, a real shining jewel of a program, only to see the next owner mangle it with careless maintenance and poor planning?

The only thing worse is when you do it to yourself!

The necessity of ownership

Successful organizations make sure that programmers have the ability to own their programs.  It’s vitally important for a creative person to have a stake in the fruits of their invention.  It takes an incredible investment in energy, diligence, thoroughness, and continuous attention to turn a random scratchpad full of block diagrams into a functional, useful program.  Good programmers may make it seem easy, but there’s no faster way to discourage a talented person than to trash their creation.

But, ownership has a downside too.  You can overload your programmers and wear down their motivation if they get stuck on the same work for too long.  You’re probably not Google, and you don’t have a few million dollars lying around to just hire new programmers when the old ones get full.  You’ll also notice that employees do occasionally leave, and to keep the remaining ones happy, you’ll need to give them new challenges from time to time.

Mixing needs

So let’s try to avoid burnout while supporting efficiency: we can rotate ownership, say, every six months or so.  If we do it right, we should be able to encourage many programmers to leverage their own unique insight in turn, and get a better product than if it had been built by one person alone.

• Use code reviews to prepare a project for a transition.  Try to bring in different people periodically.  Watch their enthusiasm levels, and ask them to share ideas for future improvements.
• Cross-develop test suites to encourage healthy competition between groups.  Challenge someone new to produce an better validation test, or to investigate new use cases.
• Performance tuning, or new feature development can also be a good excuse to get new programmers involved.  Try to use a clear, simple, one-time goal in mind for the project to make it easier to gauge success.

Once you get a rotation in place, watch for reactions.  Some employees may be happy to get new tasks, others may regret losing control of their key contributions.  Balance out your well-thought-out plans with the practical realities of day to day management, and don’t be afraid to change your plans if needs dictate.  Happy rotating!

Advantages of Bad Code

Not every program is a masterpiece of ingenuity.  Software doesn’t exist to be a perfect shining diamond – it facilitates something else, and that something else is what we care about.  The intense 3D game you developed – fantastic!  Your massive accounting report - it should rock!  Your child’s digital storybook – it should be glorious!  But the program you’re going to write today?  It’s messy, it’s complex, it’s slow, it’s O(n^2), it’s unrefined and awkward, and that’s okay.  Let me tell you a story:

The Parable of the Two Programmers

A manager asked two programmers to compete to write a program to solve the same problem.  “This is a tricky problem,” the manager said.  ”The one of you who solves this problem will get a promotion.”  One of the two programmers thought, “This is my opportunity to show how good I am!,” and set to work analyzing the problem thoroughly, refining data structures, selecting the correct algorithms, and isolating every edge case.  The programmer labored over the program carefully, wrote unit tests, validation suites, and finally delivered a fully documented, functional masterwork.  The manager replied, “This is great, but I’ve been using the other guy’s program for a few weeks now.”

Why is bad code okay?

We all want to have good code, but good code has a cost.  It takes lots of extra work, time, and refinement to produce ideal code.  In the story above, I purposefully left out the criteria that the manager used to decide whether the program worked or not.  Ask yourself, the next time you start a project: how do you know for certain that what your client asked for is actually what they want?

Unless you’re gifted with perfect foresight – and all evidence suggests that nobody is – you’ve just encountered part of the reason that SCRUM and AGILE exist, the reason that people warn you against premature optimization, and the reason that most IT projects fail. 

They all come down to a key cause: it’s really difficult to identify what will make a project succeed in advance.  In some cases, ideal algorithms matter.  In other cases, exhaustive correctness matters.  In other cases, flexibility matters.  You can guess, and you can listen to your client, but you just don’t know which item will turn out to be the key when you start.  Clients are often just as wrong as we are!  That’s why it’s useful to produce a solid program, one that may be a work in progress, then refine it.

After decades of lessons in professional software development, the general consensus today is that rapid iteration creates better software than attempting to perfect your design in advance.  The point is to get your program to a healthy state, where the program is usable and the code is maintainable, then use and maintain the heck out of it!  If you want to provide the illusion to your customer that you produce perfect code on the first try, release it a dozen times internally, use it, refine it, and only then show it to the client.

Rather than trying to perfect your software up front, learn how to write something simple, easy to understand, and modular.  The programmer who’s tinkering with a finished product will have a much easier time deciding on optimization than the programmer who is still living in theory land.

Lessons from the Classic Mac

We all do tons of user interface design, both in games and in business software.  One of my favorite books on usability design is “Macintosh Human Interface Design”. It’s a classic; I’ve been reading it off and on since as long as I can remember.  Much of it is woefully out of date.  But, as with all visionary products, it has lessons to teach us to this day.

So, what can we learn from a user interface three decades old?  

Design for the worst case scenario

In the era of the original Macintosh, they spent a lot of time designing their applications to handle the worst-case scenario smoothly.  For them, the worst-case graphics meant two colors – quite a stretch when we nowadays think in floating point 32-bit color.  The Macintosh team solved this by requiring each icon to be designed in black and white first, and converted to color later.  This ensured that all the icons had a consistent, striking shape that was easily recognizable, regardless of your monitor’s color depth.

Nowadays, you need to identify your user interface’s worst case scenario.  Your user interface will display information and receive information – which things will change?  You may design it for a small number of columns now, but need to support more later.  You may design a graph to work great with three lines, but when fifty lines are added it may be unreadable.  What growth will cause your interface to break?

The lesson is, even if your designs works now, what will happen if your assumptions change?

Have a reason for everything

The Mac design book is full of lots of interesting little explanations: The little “…” that appears on a menu item tells you that clicking it will open a dialog box.  All controls related to each other should be inside a rectangular line called a group box to indicate that their functions overlap.  Text labels should be evenly aligned because it makes reading faster.  Any menu selection should be flashed so the user knows what they clicked.

Having an explanations is important, more so than the quality of the explanation.  When you’re designing, you need to have a reason for everything, even if you have to make one up!  A bad reason - even if it’s really lame - allows your colleagues to debate constructively.  Someone who can make a better argument can help improve your design, and improved design is what we want.

Have standards, and stick to them!

Way back when, the Macintosh had a standard that said that the “Save” button was on the bottom right hand corner of a dialog box, and the “Cancel” button was immediately to its left.  They had a reason for doing that, because they said the default action should be in the bottom right corner.  When Microsoft came out with Windows, they swapped places.  They also had a reason: they wanted the default choice to be harmless, so if a user clicked without paying attention no damage would be done.

Which standard is right?  It doesn’t matter!  The important lesson is to be consistent.  If you make a nifty decision in one part of your application, go look at the rest of the application and see if you can apply that lesson elsewhere.

Keep thinking

Keep reading, and discussing, and debating your design.  If you cultivate an atmosphere of appreciating suggestions, you’ll wind up with a better result.

Happy designing!