# Deployment
Well, you wrote your application. Now it has to run in the wild. How to deploy? What are good ways to setup the server architecture? What are the bottlenecks you'll face? Also... have you ever heard of our lord and savior, the backup?
# One-click or continuous deployment
There are only two sane options for deploying your application: you can have a one-click process to deploy your code, or you can have a zero-click process to deploy it.
Deployment consists of a series of steps. You may have to compile things. These days of webpack and css preprocessors make generating a frontend production build a process, unlike in the 2000s when you just copied files. There might be several steps on the backend as well. At least you should run all your tests to make sure this version has not regressed. You might want to bump versions, zip everything into a package, copy it to the server (possibly servers, many), perhaps restart a service. Even copying to the server might be an involved operation, unzipping the package to a new directory and then swapping the old one with an atomic operation if possible, to avoid downtime. Or if you have a server farm, you might want to take the server down, update it, and then take it back up, and do this sequentially for all servers, also to avoid downtime.
Don't do it manually.
Again, don't do it manually.
Use a script from your very first deploy. There are tons of ways of writing it. It might be a shell script, you might use something like Ansible. As long as you have a single click (or run a single command). If you need to run 17 commands for a deploy you are bound to make mistakes. Things crash and you might need to do a quick deploy or several in succession. You don't want to waste your time or make even more mistakes. We talk about deployment on the frontend section as well.
Even better than running a single command is to have continuous deployment. You are running a continuous development server to run your tests, right? Use it to deploy your system. Ideally you should deploy every time there's a commit to your master branch that passes all tests. This requires good coordination between teams and excellent tests, so you'll be sure that anything that is broken is detected before it can be released, which might be difficult to achieve when you are starting and doing heavy development (it shouldn't be like that, but let's be realistic). It's also more complicated for mobile apps, which you don't want to push so often, since they might need to be validated by the stores and deployed to users who will download the app, or for packages/libraries, which might need a better control over what constitutes a new release. But you can use tags in your repository to control this, and have automatic actions to publish new packages or deploy to app stores, so it's not an excuse to not automate things -- you still get a one-command deployment, whenever you tag your repo.
Your life becomes a breeze when your CI/CD server does it all for you.
# Scaling
This book is not about system administration. There are many technologies these days for scaling websites. The approach we take in this book, of a monolithic service, should fit most applications. In fact, Wikipedia is written this way based on MediaWiki and found a way to scale over many servers. This section will give an overview of common bottlenecks and ideas for scaling.
There are many services these days that handle all aspects of system administration for you. Called a "serverless" approach, you use services managed by your provider for everything. It's also known as PaaS (platform as a service). This is comfortable and vendors make it very alluring. There are two basic disadvantages with this approach: first is cost. You can rent a server these days for $5 or $10 that will handle enough traffic for the bottom 50% (yes, that is a random statistic that I just made up) of all dynamic websites, and for $100 you have a very powerful server or a few smaller ones that are more reliable. A well setup server will run without manual maintenance for years. So you should ask yourself if you really need to spend 10 times this cost at least when you are starting. Take also notice that PaaS that are setup wrong can easily run bills in the thousands of dollars before you even notice -- this is not their fault, it's just that they are powerful enough to allow you to create feedback loops that will incur in costs.
The second problem is that while in your server you have full control of your architecture, and you can easily migrate to another provider that gives you a better deal. You avoid getting stuck in a vendor lock-in. Some services follow a standard (like object storage, which ended up having the same API everywhere exactly to make migrations possible) and others are quite standalone (such as a CDN front), which makes it easier to pick and eventually change. There are plenty of technologies to help you with things. From systems to handle multiple servers such as Ansible to cluster tools such as Kubernetes, you should at least learn about what technologies exist to handle things yourself. Remember that if your application grows in usage your problems will change with time. Sometimes cost will be an issue, sometimes you need to scale quickly, sometimes you need to handle a very elastic load. So there's no single solution that will be best for everything.
When looking at scaling your system, you need to understand what is its bottleneck. From a low lever point of view, the possible bottlenecks of a web application are: network I/O, disk I/O and processing. When viewed from a higher level considering the parts of your application, they can be database, static file serving, dynamic file serving and dynamic processing. Let's analyse this.
# Static files
Static file serving means data that is not user-generated: static HTML, CSS, JS, static images like logos or icons. This data changes rarely (only when you deploy), making it great for caches at several levels. There is no processing done on them: servers get a request, read a file from disk and send it back over the network. It's rare that this is a bottleneck, but as you scale it starts to steal time, disk and networking I/O from other operations. One solution to this are either setting up a separate static server, usually with a fast HTTP server such as NGINX that is fit for serving files with a minimum of overhead. You can setup several servers in a round-robin or a fancier DNS arrangement, which can be spread in many geographical locations, which also makes your website have less delay for users.
Instead of setting all this up yourself another way to this is to get a CDN (Content Distribution Network) service, which will act as a middleman for your requests. This is easy to setup and very transparent: the service will handle requests for your website. If the file is cached it will serve it directly, otherwise it will fetch it from your website, cache it and send it back to the original requester. Setting up requires little less than altering DNS records. This is how Cloudflare, AWS CloudFront, Fastly and similar services operate.
To find if this is your bottleneck, watch your server disk I/O and the process responsible for it. If it's the webserver, it's file serving and time to find a solution.
# Dynamic files
Some sites depend heavily on user generated content, such as photos. This means that files are being constantly uploaded. There are two problems here: one is that if you have a lot of traffic you start to accumulate a lot of data, and a local disk starts to become unpractical. This is aggravated by being more difficult to distribute this data. While with static files you can easily copy all files to all your servers when deploying a new version, dynamic files are uploaded and have to be redistributed in real time.
There are a few local approaches to manage this, with distributed file systems. They work by dividing your data transparently over many servers. They can handle file distribution and replication themselves. You can setup data to be replicated, so if one of the servers goes down you don't lose data, similar to what RAID 5 does to multiple disks. Setup is not quite trivial, and scaling to new servers will require a management infrastructure. Don't use databases to store media: both relational and non-relational (such as NoSQL) databases are not made to store large binary data. Use a object store service if you don't want to use plain disk.
Like everything else, there are companies providing specific solutions for this. It became common to use object storage services. This is a service that allows you to send files which are stored remotely, manipulated by an API and accessible by third parties directly by HTTP. This means that you offload storage completely, and in your application you'll store only a URL or id for files. These services, like Amazon S3 or Linode Object Storage, handle distribution over CDNs, high availability, replication and backups. You pay by storage and by network traffic, and have zero headaches with sysadmin. You also have no limits: you are paying for what you use, which can be 1GB or 10TB, and you never have to allocate a new disk, block storage or server yourself.
You can also use object storage for static data, in fact, and serve your HTML/CSS/JS from it. Most frameworks have support or plugins to connect with these services directly, and S3 has became a de-facto standard API for them, making it possible to migrate from one to another with minimal headache and avoid a vendor lock-in.
The disadvantage of object storage systems is only cost. It can be quite expensive since you are paying for traffic, while with your own servers you pay a fixed amount for unmetered traffic.
Like static file serving, to find if this is your bottleneck, watch your server disk I/O and the process responsible for it. If it's the webserver, this is a bottleneck.
# Request processing
Moving on with our bottlenecks, even if you offload all file serving to other services you still have to process dynamic requests: logins, fetching lists and items data, upload and form submissions, etc. If you preferred to render HTML on the server (SSR), you also have to serve every page that users request. Depending on how complex your application is this might mean a considerable amount of processing.
To find if this is your bottleneck, check your processor usage rate. If your application is hitting the roof, this is it. There are a few ways to handle this.
Web applications have independent requests, so adding more servers or servers with more processors is a trivial way to scale, and grows almost linearly. It's easier to move to a single, more powerful server, which will only require adjusting the number of processes in your webserver. You might want to check its configuration too. Since you might be waiting for I/O (from the database) you'll want to have more concurrent processes than you have processors.
It's common these days to ignore optimizations and just throw more hardware. But if you disregard your application performance, making hundreds of database hits per request or using inefficient algorithms, such as fetching all data from the database and filtering it yourself. If you application is inefficient it will be slow no matter how many servers you allocate. To know if that is the case make a typically slow request to a server with no load and see how long it takes to run it. You'll benefit from more servers if individual requests are fast but you have a lot of concurrent requests. If you have few concurrent requests and they are slow, you need to work on your code. Run it through a profiler. For PHP Xdebug can do it.
If your database and dynamic files are in separate servers (or third party services), which you definitely should as soon as possible, allocating new servers is a breeze. Setup DNS for round-robin or add a node balancer, which is a service that routes HTTP requests to servers and can keep connections and handle servers going down automatically. Most cloud server companies have nodebalancers ready to be spun. You can handle it yourself with HAProxy if you prefer, but nodebalancers tend to be very cheap, so unless you need something special it's probably not worth it. Having multiple webservers is a good way to increase reliability, too. If you have a single server and it goes down, you are offline. If you have at least two you are safe.
If you opted to a microservices approach instead of the monolith that is presented in this book you can easily spread your services among several servers and use some technology to manage everything. Microservices, if well planned, will scale well regarding processing, since you can find out which services are bottlenecked and replicate them. If the problem is not load but delay, however, you have to optimize the software.
Of course you can use cloud services for request processing as well. Instead of handling yourself a server with a webserver you can use a managed approach, with images built with Docker. Or you can go the PaaS/serverless route. In this case your software is not allocated to a single, long running server, but it's added to a pool of servers. The advantages are not having to manage the server and being very elastic to traffic: if you suddenly have a spike of 20 times your usual traffic, which would kill static servers, your site is still online and as responsive as usual. The disadvantages of going to a serverless architecture are increased cost and probably vendor lock-in.
# Database
Database is the most common bottleneck in web applications. This stems from a number of problems.
Amazingly, very often it's because you are using the database in the wrong way. Is your query slow because you never created a key and it needs to sweep all entries? Database software allows you to easily check your slowest queries, which then you can analyze with EXPLAIN queries to see if there is a trivial optimization. Note that adding indices for everything might make your database slower, so know what you are doing.
Another common problem it to fetch too much data. Make sure you fetch only what you need. Doing SELECT * when you only need ids makes the DB hit the disk more and send more network I/O, and your webserver will get a lot of data (that will consume more RAM) just to throw it away. When using ORMs it's easy to load things inefficiently when you have relationships. This is known as the (n+1) problem. Let's say you want a list of posts, and each post has an author and you need their names. If you get all posts, then you fetch the author name for each post, you end up doing (n+1) queries. When writing SQL queries yourself you'd easily avoid it with a JOIN clause, but ORMs often need to be informed about it. (Laravel has the with keyword)[https://laravel.com/docs/7.x/eloquent-relationships#eager-loading] exactly to alleviate this problem. You can check which queries are performed by a request using Telescope to see if you are performing unnecessary queries.
If your queries are fine and your database schema is optimized, yet the database is still your bottleneck, you have a few options. Is it disk I/O or is it processing? Check both. Usually databases are I/O bound, and the trivial way to make them faster is to throw more RAM at them. If you have enough RAM to keep the entire database there your performance will be way better. This is not always feasible, but for most smaller sites you can have at least the same order of magnitude. Databases store text (they are not made to store media!) so a few GB means a lot of data. The English Wikipedia articles take around 77GB (July 2020), with all pages taking around 173GB. This may seem like a lot, but you can get servers with 1TB of RAM these days at a reasonable cost. If you need speed, vertical scaling is very likely to be possible unless you are dealing with so much data that you should be writing a more advanced book than this one and rather than reading it, and probably using a database made for scalability and volume such as Apache Cassandra.
This doesn't mean that you can't scale horizontally as well -- or even more, that you shouldn't. If your database server goes down your entire site will go down, and having multiple database servers is the sure way to handle high availability, either doing it yourself or getting a service that does it for you. There are many managed SQL services these days, such as AWS RDS, running common SQL software. You can do it yourself with tools for your favorite SQL flavor, such as Galera for MySQL and for MariaDB, or PostgreSQL tools. Managing a distributed SQL installation can be tricky. Even trickier with multiple geographical locations, when it becomes a serious problem, so think about it before you setup.
# Backups
There are three things you must know about backups.
- You need backups.
- You don't only need backups. You need a plan for restoring backups.
- Reliable and distributed storage is not a backup.
The first one is obvious. Things go wrong, hardware fails. You can't afford to lose all your data.
The second one is often forgotten. People setup backups and forget about them. When they need it, they have no plan and don't know how to restore things. Ideally you should be able to restore backups with a single command. More than that, are you sure your backups are actually working? It's easy to find horror stories of automated backups that were creating empty files, ignoring important directories, or being created and immediately deleted. If possible you should periodically run a backup exercise to make sure everything works. If not, at least add some checks to backups: email in case of errors, have a cron script that makes some sanity checks. I prefer to have a notification that backups are actually happening, since if I don't get a message I know that something is wrong.
The third is another catch that not all people take care of. You might be using a reliable storage system for your data, with replicated databases, or a third party service that guarantees a very high level of availability. There's no single point of failure, right? Except for humans. If you run a script that erases data, it's immediately erased on all your replicated copies. Backups are not only for hardware failures, but human failure as well. Besides, even though your provider guarantees multiple copies of your data, it might be all local. What if a catastrophe happens to the data center? As the saying goes, if you don't have backups at two different locations, you don't have backups. And some people say three.
There are many ways to backup data. If you are using your own servers, try to use snapshots. These are available for virtual servers, are a breeze to setup and guaranteed to have all the data. Restoring them is as easy as spinning a new server.
If you have a distributed application using PaaS, you should not only backup your data, but have a good way to rebuild the system if something happens, like someone deleting services by accident. This is a problem with distributed architectures, since you use several services and our natural tendency is to setup it manually once. If your server crashes and you have backup of the data, can you also rebuild the operating system/software/dependencies with a single command? You should -- this is useful to spin a new server for redundancy or to migrate data. If you use a PaaS, can you restore it if someone screws it up and deletes things? Have a backup of your settings and if possible control everything through an API, not creating things manually on web forms.