DivvyStats: An Excercise in Architecting and Full Stack Development

My Objectives

Let me begin by saying that this project is a non-revenue project. I haven't been paid for the work. However, I felt that the benefits would justify the effort. Here's what I wanted to accomplish.

• Architect an entire system, taking the design from conceptualization to the specification of technologies used for implementation.
• Develop an entire system, e.g. become a "full stack" developer able to start at the beginning and work through to the actual deployment and launch.
• Learn and develop some proficiency in a number of technologies that were to me, mostly mysterious acronyms before I began to learn and employ them.

I leave it to the reader to decide how well I succeeded in my effort. My personal feeling is that I did rather well given my somewhat inexperienced starting point. Clearly, I have more to learn; but learning more about what you don't know is part of the benefit of the exercise. Learning about things you don't know you don't know is an even greater epiphany!

Foundation Elements

Most systems are built around some fairly known and widely used technologies. This effort was no different. Here's my foundation technology list. All other technologies complimented these:

• Delphi and its related technologies such as FireDAC, LiveBindings and FireMonkey.
• Interbase RDBMS along with a couple of user functions (written in Delphi) used to make things a bit easier and more maintainable.)
• Node.js, your savior or your nemesis depending on your viewpoint.
• HTML5, an absolute must if you're doing modern web development.
• CSS3, another "must" for modern web development.
• JavaScript, because if you use Node it's pretty hard to use anything else. It's ubiquitous despite all of its nay-sayers.

Of course, there were lots of ancillary technologies used to support the effort, among them uniGUI (FMSoft's web application development framework), npm (Node Package Manager), Express (a Node based framework) and Bootstrap (that helps trying to style responsive websites.) For a more complete list and discussion, see the website (link below.)

Major Topic for Development

One of the decisions to make for a project of this kind is what I call "topic." Just what do you choose for development? The choice should be:

• Complex enough to be challenging and interesting
• Interesting and relevant to a reasonably sized audience
• Solvable using available technologies and skill level of the project team (in this case me)
• Related to something already of interest.

I finally settled on DivvyBikes, Chicago's bike sharing system. Not only does it satisfy my criteria, but it also offers:

• A large repository of data free for the taking (for any legal purpose)
• Serious possibilities for logistic complexity
• The possibility of having an impact on urban life and transportation
• Personal interest since I'm a Divvy user and to say I'm enthusiastic about it would be an understatement. (I recently received an award from Divvy for being their champion senior rider.)

The Results

The result of this effort consists of two major deliverables:

• The DivvyStats website, that you can visit here: DivvyStats. Note that the link uses a nonstandard port.
• The DivvyStats Dashboards, that you can visit here: DivvyStats Dashboards. Note that this link also uses a nonstandard port.

The website provides a lot of detail and imagery about the system, how it is architected, what technologies are employed and screen shots of expected outputs and displays. This includes things like the system overview diagram, a list of technologies with links for more information, and an E/R Diagram for the database used by the system.

The dashboards are a uniGUI application. The various displays available are explained in greater detail by the website page "Dashboards" accessible from the System Architecture dropdown of the main menu.

Effort

The entire effort occupied about 600 hours. Unfortunately, I did not track "learning curve" separately from application of technology. In some cases, much of the time spent was actually trying to learn a new technology in order to apply it effectively. I developed a half-dozen or more separate, small projects along the way to test various ideas before actually implementing something in the final product. Here are the major time investments, from most time-intensive to least:

• The maintenance and administrative program, written using Delphi Tokyo, FireMonkey and LiveBindings. Also included were new experiences accessing Divvy APIs, using the Zip components to unzip files and then process the .csv files that came as a part of the zipped download. Along the way I developed a crude ORM and gained some practice in using some of the more common design patterns. Data cleansing was also a bit of a time-consuming effort as the Divvy data covers several years, and the files generated by Divvy early on differ in some ways from files in later years. No external files were used. The Divvy data was downloaded into RAM, unzipped, and the .csv files converted to my mini ORM entirely without writing to external media. This enabled me to achieve about a 700 row/second insertion rate on the Interbase database installed on another server locally, accessed via a gigabit LAN connection. Overall, lots of details occupied time with this program.
• Next most time consuming was the uniGUI application, a totally new technology for me. While there is some documentation for uniGUI that is helpful, it still could use greater detail and more comprehensive content. Much of what there is to learn must be learned by reading the code and samples, and by reading and learning about the Sencha components that uniGUI deploys.
• The actual website was next, but since the website is basically a static site, not much effort was required beyond what I had already acquired from other efforts. One of the more interesting pieces of the website was the sending of email used by the "Contact Me" page.
• Data analysis and research followed website development time-wise. This involved determining data content, examining relationships and generally trying to clean up a few things that were ambiguous.
• The monitor service (a Windows service program that periodically takes a snapshot of the entire Divvy system) was next. Mostly this was simply learning the most efficient way to update the database. You download 574 records and pop them into the database every X number of minutes. Some editing is required along with data conversion. (I store everything UTC, the only reliable way to store timestamps, but Divvy provides data in local format that of course suffers from all the deficiencies of daylight savings.)
• A few other minor time chunks were taken up by architecting, database design and administration, deployment, graphics (using Adobe Illustrator) and version control (Atlassian and GitHub.)

Conclusion

Building a complete "system" is amazingly time-consuming and requires a broad base of technology familiarity. It has given me a much greater understanding of just what it takes to bring an idea from concept to Minimum Viable Product.

My suggestion is to go through the website and try the uniGUI application. I hope it provides you with a greater appreciation for the depth and variety of tasks it takes to realize a vision using IT resources.

Good luck with your own efforts!