Looking at Tools

Its been a month, and I've spent it playing around with different front-end web application development tools. Specifically, in previous incarnations of this blog, I've stated that I'm not fond of javascript. Really not fond at all in fact. This has been debated endlessly online, but frankly I think its language which has several specific technical faults, and which just generally doesn't serve well as an application development tool. Even for small-to-medium projects I'd rather have strong typing, ONE kind of representation of 'null' instead of several, and a more clear and established sort of OO paradigm (to be fair javascript has maybe 5 different ones, and that's the problem!). While there are some 'standards' for modularity in javascript it isn't baked into the language, the various strategies don't mix well, and basically the result is a mess. While you could argue that ES2015, aka ES6, rectifies some of the deficiencies in a mild way, it leaves many of them intact and creates a C++-like mess of backwards compatibility hacks. This means you'll always be dealing with bad practices in libraries and tools forever. ES6 is also not available in browsers, meaning you still would have to compile/translate it to ES5, just to gain access to a few of the desired advantages.

The upshot is, I don't want to web application development in javascript; I just don't! I would argue against it even if I were doing work for a client, unless it was some extremely trivial amount of code. The lack of a good module system, strong typing, and a single consistent OO model are in my mind disqualifying. Of course there's no other option except javascript as an executable format for client-side application logic. Luckily we've progressed now to the point where we can write code in a better language and compile it to some sort of javascript which will run on the client side. This comes with many advantages:

1. We avoid the badness of javascript, a good implementation language will simply give you strong typing, good OO, modules, etc.
2. We get better unit testing and integration with existing mature IDE technology.
3. We can, if we choose correctly, share code between the client-side and the server-side of our application.

There are many choices for this implementation language, probably far too many to evaluate and almost too many to even list them all. Personally I've encountered, and sometimes used, the following: TypeScript, clojure, kotlin, scala, java, groovy, Ceylon, and Dart. Of these I exclude TypeScript and Dart from consideration due to their lack of JVM-based implementations. They could be used to create client-side code, but not to share it with server-side. This might not be a fatal consideration in a larger project, particularly where these languages might already be in use, but for my purposes I might as well have a JVM language, since I am planning to use the JVM in the back end.

Beyond wanting a JVM language, I would like one that is reasonably Java-like. Java may be criticized in various circles for this and that, and it has its faults, but its a very vanilla language, easy to understand, easy to write, very familiar. I also need something with good support for compilation to javascript. All of the above listed JVM languages have reasonable javascript target compilation support, so I did significant analysis of clojure, scala, and java itself. I also looked into the state of support for kotlin, ceylon, and dart.

Clojure is a little too functional for me. It lacks any real OO model at all. Clojurejs is a very mature ecosystem that provides all the functionality required to build a front end for a web application, compile to javascript, invoke existing javascript libraries, etc. However I just don't feel that the learning curve is worth climbing, and it is a steep one. If I was a LISP guy then I'd probably just go with this, but I'm going to pass.

I did a pretty extensive examination of Scalajs, including making a PoC of a front-end. Its a rather tempting idea. There's good support for React, which is an excellent front-end, and a good bit of support. Still, I find scala to be dense, obtuse, and hard to read. While I was able to master the language to a decent extent it would require a lot more time to fully master. I also found that the Eclipse support for scala/Scalajs is middling at best. Syntax highlighting, completion, and other features work intermittently and I spent many hours fiddling with things, and still never got Scalajs to consistently build my project every time. The resulting code is obtuse looking and I'm far from certain I've really achieved a good design. It would clearly require a lot of head scratching to get to a full web application this way. While scala is closer to a standard OO language than clojure, it still has a steep learning curve.

Of the three languages I checked out thoroughly that leaves java itself. Now, many years ago Google produced GWT, a large client-server app framework which relied on target compiling java to javascript. There were two issues with this, the framework was very heavy, mandating specific ways of accessing data, a specific application model and structure, etc. Secondly, javascript and browsers were much cruder in 2006 than they are now. As a result the target compilation was fairly unreliable and insuring cross-browser functionality was problematic at best. This earned GWT, and the notion of java as a front-end language, somewhat of a bad rep. Google eventually developed AngularJS as an alternative, and turned to Dart and then TypeScript in an attempt to make a lighter weight and more reliable construct.

However, as much as the entire mass of the GWT framework has no appeal for me and doesn't fit my idea of what I want to do, the GWT compiler itself is an excellent tool! Several different groups have continued to evolve the compiler and various light weight approaches to client-side java which simply use it as a nothing but a compiler. This is a very attractive concept. Currently I'm looking at VertxUI, which uses this strategy along with some simple libraries which provide something very similar to React, but written in java instead of JSX/javascript.

As for kotlin, ceylon, and dart... There doesn't appear to be a JVM incarnation of dart, though it seems like an obvious idea and dart has targetted other VMs, oh well! Ceylon has what appears to be a pretty solid javascript target compiler, some libraries, etc. Kotlin also targets javascript and has some active development. Neither of these two languages seems currently to have quite the level of maturity on the client side that java, scala, and clojure have. Ceylon seems to have limited traction in general, although it is a decent language. Kotlin appears to be gaining momentum, is a lot easier to use than scala as a java programmer, and has a decent amount of client-side activity. Still, neither of these seems to have anything quite like scalajs or java itself.

A final consideration is integration with my tool chain. I touched on Eclipse a bit earlier with respect to scalajs. I didn't get a chance to delve into kotlinjs, but kotlin is very heavily Intellij biased. I'm sure there's Eclipse support, but it is more of an afterthought. With scalajs there is a full suite of plugins for Eclipse and Gradle (my build tool of choice) although the community seems largely focused on the sbt command-line tool. I didn't evaluate ceylon deeply enough to discover exactly how well I could use it with Eclipse/Gradle.

VertxUI/GWT compiler seem to be rather winners here. Vertx itself has very solid Gradle support, and this seems to extend to VertxUI as well. Given that the implementation language is java, the Eclipse language support is sterling. I'm really looking forward to digging in and generating a PoC with this on a par with what I did for scalajs. It looks like the code itself will be MUCH more straightforward than the equivalent scala.

Client Process

OK, so, I'm my own client here. Perhaps this is not the ideal situation in an engineering process, but at least its pretty clear to me what my process IS. I need to decide what tools I'm going to use to approach this task I'm on. Its important when you work with a client to work within their process. You can always enhance that process, or maybe even create it, but the process should be owned by the client. If they don't own it, if they can't master it and live with and make it serve them, then ultimately it will wither. What you see then is some later contractor comes along and looks at the dusty remnants and sweeps it all away to start again! This isn't really, usually, a cost-effective result for your client.

In my case there are tools and technology which I'm comfortable with and know how to employ. These are things that I can pick up with relative ease, and with which I have, generally speaking, some degree of mastery. Now, in the web-app development space there are a truly vast number of tools, frameworks, libraries, platforms, etc. At this point, 20-some years into the web era you could likely find something that does almost exactly what you want. The problem is finding it isn't going to be easy. Once you do find it, its likely to be something obscure (most stuff is). Anyway, part of my motivation is going through the process and mastering some newer techniques. Besides, I don't want to spend the months it takes to learn Wordpress (or whatever) well enough to write a plugin for it, and then need to be hooked up into that world forever. Particularly when the feature set I need is not that extensive.

To get more specific, I've been a Java guy for a long time now. Nothing is more conveniently portable and likely to remain supported than JVM-based code. Today you can write this code in a wide variety of languages, leverage a huge number of libraries, and work with a vast array of tools. There are other ecosystems out there. At one time we might have compared the Java ecosystem to that of .NET/Mono, but to be perfectly blunt I just don't see it anymore. .NET has essentially disappeared from serious consideration in Enterprise web-application development circles.

Of course Node.js has, at the same time that .NET faded, risen to significant prominence. There are now more libraries, frameworks, languages, and toolkits residing in the Node.js/NPM space than you can shake a stick at. Of course this is kind of the problem, the noise is deafening. Now, I've used Angular and done my share of Javascript (blech, what a horrible little language) development. It is what it is. I just can't help feeling kind of like its another fad. Sure, I'm as biased as the next guy, but I did say I was a Java guy!

So, between the JVM and Node.js there are various pluses and minuses. The tools most commonly used for web client development now tend to be in the Node.js space. Java has a very solid Javascript interpreter in Nashorn, but it isn't the primary target for most of the currently popular tools. On the flip side, there are JVM-based tools that simply have no compare in the Node.js world. Frankly I'm happy sticking to the JVM world for now.

This also means I can stick with Eclipse. Crufty though it may be Eclipse 4.7 (Oxygen) is a pretty darn good IDE. It may be huge, and the quality of Eclipse plugins is pretty varied, but it is pretty solidly a part of my process. Remember what I said? Process is important. I might change some day, but for now Eclipse holds sway. There is certainly no IDE with the same level of support for Node.js as Eclipse has for the JVM and everything in it.

So, I feel like I'm going somewhere here. I have a process, Eclipse, JVM, and Gradle. I didn't talk about build tools. There are, once again, many build tools. I just like Gradle. Maven is a little more common in the JVM world, but I find its approach too rigid. You end up spending 50% of your time wrestling with Maven to get it to let you copy the file to the place when and where you want. Gradle is a LOT more flexible, and its DSL is pretty nice, vs the ugliness that is the POM.

This is a tool set that is well-supported, unlikely to vanish into obscurity any time soon, and which has a rich ecosystem. In any case, we can always bring in other technologies if we need to. There's nothing stopping us from installing a copy of Node.js and NPM on our dev box. I'd rather use a JVM as a runtime, but I'm not hostile to using any good tool for what is good for.

Further Exploration of Needs

Last time I talked about requirements in terms of some user stories and a general description of what I need. Now, taking the hat of a developer/analyst I really need to dig a little deeper and start to put together a design. The design activity will then feed into some system architecture, which is mostly going to consist of assessing various tools and picking some.

So, what do I actually want here? What really is the genesis of the need, the itch that wants to be scratched which leads to putting in the time and effort to design a whole web-based application? In a nutshell I want to be able to author content in a way that is reasonably simple and yields good results. I'd like the material to be in a format which could potentially be useful in other contexts (IE some sort of fairly standard and well-supported markup that has support for doing things like generating PDF, Postscript, LaTeX, or something similar that can potentially be published or merged with other content). This desire comes from an appreciation of the value of flexibility in information representation and the great utility of reuse. I'm going to author content, I want it to have maximum value and yet be simple to create and straightforward to reuse. Again, this is why most CMS/Wiki solutions don't work, the results are anything but portable!

One big area of research this brings up is obviously what sort of content data representation to use. The world is replete with different options for markup languages, WYSIWYG editor/word processor formats, etc. Word processors don't seem to be of much use here. Sure, you can author most anything in Word, or OO Writer, or utilize something online such as Google Docs, which can support export/import in several formats. Most of these tools will inter-operate to some extent, but they all eventually fall short in some area or other. All of them are difficult to use to produce online content with much quality. Beyond that, I haven't seen much evidence that word processors are really very adept at producing book-length content at the level of quality needed to do something like print publishing. Certainly its a huge challenge, and if you've invested your time and effort into carefully formatting/laying out your content in, say, Word, then you're likely to have to do it all again in Scribus or InDesign to produce high-quality output. Word processors tend to be difficult applications to get your data out of as well. You can export, but the target formats and results are usually too limited and messy to be useful in a toolchain, or else you simply lose all the formatting you created in the first place.  So, actual content generating, writing text, could be done in OO Writer, for example, but you'd be basically using it as a text editor, so why bother?

The alternative are markup languages. HTML is one which we're all familiar with, but its not an easy language to author, and it doesn't tend to lend itself to particularly well-structure results which can be easily moved into other formats and re-used. There are MANY 'wiki formats', basically small markup languages which are intended to render as HTML, such as TWiki markup. These are OK, but are generally only well-supported by one or a very few applications, with options for conversion to other formats either limited or of low quality.

This leaves roughly three contenders for a good markup language that will support flexible reuse and work pretty well with HTML, LaTeX, Docbook, and Markdown. LaTeX is the grand-daddy of all markups (well, there is runoff and its ilk, but these are little-used today). It certainly has the expressive power required to do anything you want. You could author the Encyclopedia Britannica in LaTeX (and maybe they do, I have no idea). Of course this is also a two-edged sword. LaTeX is powerful, but it is also really complex. You pretty much have to be a LaTeX Guru, or have access to one, to do a good job of authoring LaTeX documents.

Docbook is another XML-based markup; this one is much better structured and was designed with publishing tool chains in mind. It naturally converts to HTML or other XML-based markups pretty easily. It can also be converted to LaTeX without a vast amount of trouble. Its fairly standardized, and it inherently supports breaking down projects into chapters/sections, which fits pretty well with the idea of displaying our content in a sort of wiki-like form.

Markdown is the other option that I've really looked at. It is another of the sort of 'wiki editor' languages. It's main claim to fame is ease of authoring. Like most wiki type formats it can be typed in quickly using pretty much any sort of text editor, or even a basic HTML textarea control. Its main weakness is more limited expressiveness than other options. While you can create a lot of content in something like GFM (Git Flavored Markdown) it lacks features like being able to specify headers and footers, floating elements left and right, borders and backgrounds, and any number of other fairly basic formatting capabilities you might want in a published document. Still, it is widely supported, albeit in a number of semi-compatible dialects, and works well online.

I don't think there's a perfect answer here. As with many such cases, maybe it is best to work with a few of the options and see how they pan out. Its not clear that we must lock ourselves into a single choice that cannot ever be revisited. It might be hard at some point to pull together content in several formats into a single cohesive document -if we were to desire to publish our contents in print for example- but at the same time premature decisions are bad decisions, so lets consider what we might do about this and look at the solution space and architecture, then perhaps perform some experiments (spikes as we say).

Project Definition

OK, so we're going to work on a modern software development project. Nowadays that means its web-based and its pretty much going to want to work in the mobile space, although there are certainly projects which specifically target desktop, tablet, and/or mobile but not all three. Heck, if you're fortunate enough, you may even get to do some middleware and not have to worry about the cataclysm that is webapp UI design paradigm land. Lucky you!

In our case I have my own itch to scratch. I would like to be able to create some documents using a simple markup language, give each document a URL, and provide a reasonably decent, but simple, SPA front-end that can view them, allow me to authenticate myself and paste in new pages, etc. Now, I could utilize any of many wikis, but I have yet to find one that is simple, reliable, and has as few features as I actually want! I've used TWiki, XWiki, MediaWiki, etc, as well as various popular CMS systems in the past. They're all hard to configure, require separate database servers, are difficult to install, upgrade, etc. They have large attack surfaces, which requires you to keep updating too.

I don't want to have to setup and maintain a separate MySQL (or whatever) that I also have to backup, upgrade from time to time, and generally manage either. Ideally I'd like the whole setup to be simple enough that I can just zip it up and move it a machine somewhere, unzip, and start it up and have it work with nothing more than setting a pointer to Java somewhere, at most. Once I've built a document and finished it up, I can hand it off to people and they can use it offline through their browser.

I won't really go deeply into the genesis of these requirements. These documents will be 'book like' and as long as each section has a URL that I can build links to from other sections, it will do what I want. As far as a UI is concerned, the details will be worked out as part of our process, but I'd like to have some basic navigation features (breadcrumbs and a sidebar where I can place links to key sections). A full index and TOC can simply be sections in their own right with lots of links in them, they need not be special beyond having a sidebar type link that stays on screen for quick reference.

It would also be nice if the whole thing can be crawled and indexed. So, while an SPA is good from a UX perspective (IE it is snappy) the back end should also respond properly to the notional URLs which the application displays in SPA operation. Strictly speaking it doesn't even need to be an SPA, it just needs to be snappy and crawlable.

I want a responsive design which will produce a usable result on a desktop, tablet, or phone. Ideally the phone UX should be excellent, but it shouldn't detract from the utility of a larger screen. If this cannot be achieved in a single UI, then it would be OK if there's a separate SPA that has mobile URL and presents a variant interface. Such a mobile-only interface wouldn't need to be crawlable either.

So, we have a few user stories here:

1. displays content written in some sort of fairly simple markup.
2. allows definition of new content sections which get their own URLs and insertion of content into them.
3. user story 2 needs to be reasonably secure so only I can access it.
4. needs to be crawlable (IE each section needs a distinct URL)
5. simple high performance UI.
6. responsive for the whole range of screen sizes.
7. breadcrumbs
8. sidebar with links to specific sections
9. either no database or some sort of embedded database which can simply be packaged up along with the application as desired.
10. simple install/backup/reinstall on the order of 'move a zip file to a new place and unzip it'.
11. Ability to display any content section as raw markup so we can download it in source form. 
12. ability to upload image files as content sections and link to them as well.

OK, so that seems like both a small enough list of things to be doable, and large and complex enough to constitute a genuine engineering challenge worthy of some attention. I'd like to also keep in mind a couple of possible enhancements that might be useful in the future:

1. Some kind of in-place editor for our markup (right now we only need to be able to paste it in and save it, but being able to bring up an online editor is obviously potentially handy).
2. ability to version content sections so that older versions can be displayed if desired.
3. Some simply access control logic to allow for protected sections of content.

We don't have to implement these last 3 things, but lets keep them in the back of our minds as we come up with a design and implementation.

What's Going On Here

Recently I came to a conclusion of several parts. First of all I just don't have the energy to keep ramping up my level of output as a developer. No longer is it enjoyable to attempt to figure out exactly how to pump out 12 change requests this week vs the 11 that went out last week. There's vast room for improvement in process where I ply my trade, but most of the factors which would need to be worked on in order to fix that are either beyond my control or beyond my (admittedly limited) skill set to fix.

Secondly, I know a really stupidly ridiculous amount about the craft of software development. The people I work with are generally highly professional and seem to have received a good bit of training (most have some sort of 4 year degree in our field, some have Master's Degrees). Some of them have considerable experience (far less than me as a rule, but that would be a high bar). Still, they seem quite narrow in their focus, not having a really broad view of the industry, their role in it, and the vast array of trade craft which has been amassed over the last 50 or so years.

Thus it seems to me at this point that my knowledge and experience outreach my capacity to use them, and probably constitute the more valuable portion of my own professional assets. So, what is one to do when one finds that he or she is no longer particularly interested in, or inclined to, earn a living using the one skill that separates him from the local car wash attendant?

Well, perhaps I can pass on at least some of what I have picked up. Here we are in 2017. Twenty five years ago I built my first web application, and I already had a pile of software development experience at that point. Yet I've never ventured to blog or spent any vast effort thrusting words out into the great Internet Void. However, I can write, so I will. I will write about what I know, the craft of engineering and implementing applications.

Being a practical sort of fellow, I'm just going to show and tell. I have some other interests, and one of them suggested a reasonable use case for such an application. So, what we'll do is we'll see exactly how the process unfolds. Whereas most people who blog about these sorts of topics try to demonstrate fairly abstract cut-and-dried textbook examples of carrying out some specific task, or a recipe for setting up some tool, etc. instead I will talk about what the tools are, how I chose them, and describe all the ugly pitfalls, quandaries, dilemmas, and perhaps even solutions which arose.

For now I think this is enough chatter. Its probably not that interesting in any case. I will get on to the meat of it in my next post, where I will discuss my selected need, the problem I wish to solve. For now, its on to catching up with Schlock Mercenary!