Plan Application Stack for Dev and Test Scenarios
In the previous article Designing Application Stack for an Early Startup I briefly discussed what every application or system could be involved into the startup application stack. Even the most important is building the product, you as an architect has to also plan for future and design middleware and infrastructure to be easily extended. Business is everything but steadiness. You will deal with change every single day.
In this article I will focus more on details of the design and propose the solutions which could be both robust in production and cheap in dev and test. From the production point of view is important whole system is resistant to extreme workloads and prone to failures. The other specifics are more more or less important according to particular application or scenario. Some are more focus to throughput and some to data consistency. Even if stakeholders will like to have system of all attributes it is always trade offs. Eventually all this has impact on budget, delivery time and scope.
Definition of Environments
Before I start to design infrastructure let me define some kind of environments. Each has its purpose and are used in different stage of implementation cycle. You will use different number of them depends on if you are young small startup or medium and large enterprise with several independent teams.
Production / Live
This stage is obvious and it is what business and customers are used to accomplish their goals. The infrastructure for this stage is usually huge and includes all but not least components like: firewalls, load balancers, virtual networks, data caches, analytic tools, monitoring and of course application stack itself. Each critical part is running in several instances and is simply scalable as a response to workload.
These stages are usually mirrors of production infrastructure. The pre-production stage is used to finally test system before the change is deployed to production. Both environments usually share data stores, but it is not a requirement. The pre-production and production environment could be utilize in tandem in blue-green or flip-flop mode, when the release management quickly switches what environment is live and what is pre-production. It minimizes application downtime and eases rollback by switching environments back.
The stage is mirror of production and its purpose is to be ready for disaster recovery. The switch to this environment is often manual. Once the main data center is down, the IT department undertakes necessary steps to switch traffic to this spare system. Once the main data center is back alive the data are synchronized back to primary data stores.
Staging / Test
Previous environments are used for production in different scenarios and thus it is important having them in full infrastructure. On other hands the infrastructure of test environments is usually more lean. There is probably not need to have several instances of same component and thus load balancers, because it is not often accessible from outside of intranet, there is not need to have firewalls and other security components. The questionable is presence of data caches or analytic tools.
The staging environment is usually full application stack, however the infrastructure of test environment can consist only part of whole system. It is reasonable to consider mocking of subsystems and data stores, it could make tests faster. The test environments are often short-live when automated and running on IaaS.
Development environments are used by developers and sometime by automation testers. The size of infrastructure depends on the development task and / or on team the developer works for. The dev environments have short live cycle and they must be synchronize with committed changes from other developers.
I suggest each developer has his own environment so he could verify his only change according to automated test. Once the change is committed the environment should be released and recreated for new implementation, it is easier to incorporate other changes.
On several examples I would like to present how each component could be design and setup for different environment. If you are small startup it could seem overkill to follow it, however you need to think someday your company will be grown into small enterprise and the importance of having several stages increases. And then it is too late and you need to reimplement all hard coded dependencies to allow you at least test changes before deployment.
I highly suggest to create REST service for each data store. I was involved to work on many systems and I could say it’s always hard work sometimes a nightmare if there is not gateway to data store implemented. Developers are creative and they call database using different frameworks and techniques. The REST service could help you having control on implementation details and compatibility.
Well but what means REST service from environment point of view. For instance you can implement caching as an option and switch it on and off by configuration. For production it will be always on and for dev or test environment off. Also in terms of caching you can implement in-memory cache as well as distributed cache. If the REST service will run in several instances the distributed cache is a choice and for single instance on staging and test environments the in-memory cache is the enough (or it could fully simulate the DB).
If you are all in the testing you know how long it take to provision database and restore to some point. The in-memory database is well suitable for scenarios of quick fast and repeatable testing. You will restore the DB into initial state just by reloading files into memory.
Note: there was not saying you have to have only one REST service for one data store, there could be a set of services to store or retrieve a data. But what I want to say is, that the application stack is using only those services to work with data store and the services do only data operation and data aggregation with this particular data store (no business logic, no other data store).
For the best integration of any 3rd party system I will suggest to implement proxy service(s) thru which the application stack will communicate with it and vice versa reverse service thru which will 3rd party system communicate with application stack.
The 3rd party systems costs differently and many of them will be available in production only. For the test and dev purpose you could need to implement some kind of simulators which will emulate behavior on origin system. The proxy and reverse proxy could help to integrate these simulators into the application stack, without need to change any other component.
Those who reads my blog know I favor messaging as a primary communication between components. Of course there is a place for synchronous communication as well. But if workflow could be implemented asynchronously it adds more benefits than inconveniences. Using message bus to manage message in production is absolutely must have. There are lot of great implementations of message bus for instance Azure has Service bus which is really awesome.
Although using message bus for dev and test scenario is not complicated, in some cases the asynchronicity could be annoying if not obstacle. If your own framework for message bus is well designed it could be possible to use local queues or direct calls. But this often requires the installation of all middleware components and services on one server. But for dev or test you don’t usually need full midlleware.
Live web application is typically running on several servers allows it to proceed more requests. The work load is equally divided between servers using hardware component called load balancer. The most using method is round robin, so load balancer directs next request to next server in the ’rounded’ list.
For most dev and test scenario having more web servers is a luxury, so there is not need to have load balancer as well. This simplifies infrastructure for the dev and test environments and safes resources of data center.
For the security reasons it is important to split application stack into several virtual networks. And each virtual network has its gateway with precise access control list defining inbound and outbound access for hardware components outside of virtual network. This best practice allows accessing your database only from server where REST data services are deployed, all other attempt of communication will be refused on level of virtual network and packet will never reach database server. The test and dev environments could be easily created without all those virtual networks and its components.
Another example is focusing on pricing. The working hours for the developers can be defined for instance from 7am to 10pm each work day. In the nights and during the weekends running dev environments are costly. Azure has its way how to automatically switch off and on resources in predefined time.
If the architecture of application stack is well designed from beginning it could solve creating and managing environments of different stages later. If your company is small or large you will still benefit from great architecture if you will have 1 or more environments. Even if you have only production good architecture could help you with future growth when creating at least staging or UAT environment will become a deal.