Scripts and wizards, too much script, not enough wizards

By Barry Harmsen

ScriptVsWizardsI recently read an interesting post by James Richardson over at the Business Discovery Blog: Wizards vs Scripts. In the post James makes the case that QlikView scripting is not old-fashioned or too hard, but is evidence of the power of QlikView as a platform.

Let me first state that I love QlikView scripting. I’m a guy who writes script for fun. I also agree that scripting offers much more flexibility than a visual solution ever could. With those things in mind, I would like to present a different viewpoint: I think that QlikView places too much emphasis on scripting. In my opinion, the default approach should be much more visual.

An old discussion

The discussion around scripting vs wizards (or, a visual approach) is not a new one. It is a variation of the age-old command line (CLI) vs graphical user interface (GUI) debate, which has been held about many different platforms and applications. With fierce proponents for either side, these debates usually end in a flame war that rages on until, often sooner than later, someone invokes Godwin’s Law. Given this, you can imagine that I was somewhat reluctant to bring this up, but I feel I must as I think some changes could improve the QlikView experience and open up the tool to new audiences.

In a nutshell, the pro’s and cons of both interface types can be summarized in the following table:

Pro's and con's of GUI vs CLUOr, to summarize it even further, scripting is good for experienced users while a visual interface is good for novice users.

TheDeepEnd

Scripting is the deep end of the pool

The question then arises; who is QlikView’s primary audience? The slogans I often hear are “Simplifying decisions for everyone, everywhere” and “There are no end-users“. To me, this implies that non-technical (“business”) users should at least be able to develop moderately sophisticated solutions on their own.

In the real world, I rarely see that happen. In my opinion, this is because new users are thrown into the “deep end” of QlikView that is scripting (and to a lesser extent, expressions). Not all of them float.

I'll have those QVD's ready for you this afternoon, fool!

I’ll have those QVD’s ready for you this afternoon, fool!

Those that do swim often have a technical background and an aptitude for scripting. Within an organization these people form a “crack commando unit“, developers and power-users who have invested a significant amount of time and effort in mastering QlikView scripting. These developers supply the rest of the organization with data models or applications. While they are awesome (yes, you are!) and valued, they’re also a bottle-neck in the analytical process. Very much like the data warehouse developers and cube-builders that came before them and that QlikView was meant to displace.

Or as the 2012 Magic Quadrant for Business Intelligence Platforms puts it:

One of the key issues QlikTech needs to address is QlikView’s weakness in data integration (currently requiring IT scripting)

I believe that QlikView could be a lot more accessible to novice and occasional users if it had more wizards and a visual interface. Learning QlikView scripting is hard and most users will not invest the time, they have a business to run.

But there are wizards!

“But” you may say, “QlikView does have wizards! Can’t novice users just use those?”

While it is true that QlikView has quite a few wizards, I see two issues with them:

  • There aren’t enough wizards. For example, why would you need to script a calendar table in each and every application when one could just as easily be created via a simple wizard? (actually, I’d prefer if QlikView handled dates a little smarter on the front-end so that we wouldn’t even need a calendar table)
  • QlikView wizards still generate script. This is my main issue with QlikView wizards; they still require the user to handle pieces of script. It doesn’t offer a true visual interface, just dialog boxes that generate script. To draw a parallel, what if, when using the “change desktop background” dialog on your favorite OS, instead of actually changing the background the OS just handed you the settings to change in the configuration file?

These two factors make it hard for novice or occasional users to achieve simple tasks and create QlikView data models, even when using the wizards.

90% of the work in QlikView scripting is about the simple stuff…

In his post, James states that “Wizards are great – and QlikView has them too for data integration – but they’re an option for simpler scenarios“. I would argue that 90% of the work done in QlikView scripting is about the simple stuff; loading data, filtering rows, splitting or merging tables, looking up values, etc. These are all things that are perfectly suited to a visual interface.

…but a visual interface can do much more than simple stuff

I also strongly disagree with James’ claim that wizards and visual interfaces are only for simple scenarios. They can do much more, and in many cases much easier.

Before I started working with QlikView, I worked with many visual ETL solutions. Besides supporting simple scenarios, most of these tools also support many complex transformations out of the box. This includes transformations that are quite complex if you have to script them in QlikView. For example, try scripting the logic to keep track of slowly changing dimensions. This is not trivial in QlikView and something I even see otherwise experienced QV developers struggle with. (shameless plug: if you join us at The Masters Summit for QlikView, I’ll show you how to do it)

A visual interface benefits experienced developers too

A visual interface is not only good for novice and occasional users, it also offers advantages for experienced developers. The main benefit is that a visual solution will give you a much better overview of how the data actually “flows” through the transformation process. It is much more “self-documenting” than script.

Using the slowly changing dimension example mentioned earlier, what do you think is easier to follow? The data flow for SQL Server Integration Services shown below, or the 200+ line script that is required to achieve the equivalent result in QlikView? And would you grasp from the script that there are actually different paths that the data can take?

Slowly Changing Dimensions Type 2 data flow in SSIS

We’re all convinced of the value of data visualization. Why should the process leading to those visualizations, arguably just as important as the end result, be captured in flat text?

Better structure – easier to maintain and more resilient to changes

QlikView script is organized in tabs that is processed top to bottom and left to right. A best practice is to organize these tabs by the tables that you are processing or by functional area. While this approach provides some structure, it can quickly become overwhelming for bigger, more complex applications. Or to use another quote from the 2012 Magic Quadrant for Business Intelligence Platforms:

Although quick to develop simple or moderately complex dashboards, when it comes to building large, complex reports from various data sources, involving detailed logic or calculations, QlikView users reported the second slowest turnaround time of any vendor on the Magic Quadrant (only Pentaho was worse).

Hierarchically structured data transformation jobs in BusinessObjects Data ServicesIn my opinion, the main reason for this is that, as more script is added, it becomes increasingly harder to organize and keep an overview. Even for experienced developers it becomes difficult to maintain the script and apply new changes. The 2 level (level 1: tab, level 2: script) hierarchy used in QlikView is just not sufficient to properly organize complex scripts.

A visual solution could solve this problem as it allows for hierarchies with more levels. An example of this, from BusinessObjects Data Services, is shown on the right. You can simply drill down to “sub-processes”, making it much easier to group transformations into relevant blocks. This makes it easier to maintain your application and absorb changes.

Auditing, error handling, traceability, debugging, lineage, metadata and much more

Besides better overview and better structure, a visual solution offers many more advantages for experienced developers and the rest of the organization. I won’t go into all the benefits here (this post is already long enough as it is), but there are many to be had. Including better options for auditing, error handling, traceability, debugging, lineage and metadata. A visual interface can have many features that will increase developer productivity.

But I want script!

“I see where you’re going with all this, but surely it’s impossible to achieve X, Y and Z using a visual interface? I just want to use script”

Yes, there are cases where a visual interface will just not cut it. However, using a visual interface does not exclude the use of script. Most visual ETL solutions support the creation of custom code blocks. Have a requirement that cannot be solved with the standard objects? Create a custom code block for it. My point is not that we should get rid of script, but that it shouldn’t be the default choice for building data models. “Under the hood” QlikView could still be using scripts, there would only be a visual interface on top of it.

Stop complaining and just use Expressor!

With its 2012 purchase of Expressor, QlikTech already has many of the capabilities that I described above. So if you want a visual interface, you can just use that, right?

While that could certainly be a valid route, in practice I see a few issues:

  • Expressor is a stand-alone application that needs to be purchased separately. It is not cheap (US$ 50,000 for the Standard version) which puts it out of reach of many clients in the small to medium business category. In these situations, alternative tools like QVScriptor could be an option, but most will elect to just write script. If/when they finally decide to switch to a visual solution they already have lots of script that needs to be converted, causing massive rework. The solution should be visual from the get-go. (I did a review of QVScriptor in 2011, you can find it here. I’ve not used it since but understand that it has been much improved in the mean time).
  • Companies that can afford Expressor, those in the Enterprise category, usually already own other tools that can achieve the same effect, such as Informatica PowerCenter. (as a side-note; in most Enterprise environments, QlikView is not the only game in town. In those environments it makes more sense to move most data transformations and metadata management upstream, instead of creating a stovepipe just for QlikView. But that’s a subject for another post.)

Most importantly though, Expressor (and the alternatives) are not (yet) truly integrated with QlikView. They still require script to import the data. In my opinion a true visual interface should be completely integrated with the core QlikView product and not require any scripting, unless you want to.

Conclusion

I believe that QlikView is the best Data Discovery platform on the market today, but it could be even better. QlikView scripting is awesome, but it shouldn’t be the default choice. A visual interface for extracting, transforming and loading data could open up QlikView to a much wider audience. This would allow it to truly achieve its goals of “Simplifying decisions for everyone, everywhere” and “There are no end-users“. Additionally, it would also enhance the productivity of more experienced developers and will shorten development times for bigger, more complex applications. At the moment there are add-ons that provide a visual interface for QlikView, but to achieve its true potential these capabilities need to be integrated into the core product.

Yeah well, you know, that's just like, your opinion, man.Your thoughts?

This has been quite a long read, so if you made it this far; kudos!

What you just read is my opinion on the script vs wizards (or more accurately, script vs visual) discussion. If you are a QlikView developer or a QlikView user, I am sure you will have an opinion about this matter as well. I’d be very interested in hearing it, so feel free to leave a comment below.