tl;dr: My development environment is packed into a docker image
Background
As a full-time developer, I have to regularly switch between 'work-from-home' and 'work-in-office' modes. Thus a major problem: how to synchronize the development environments between these 2 places?
Another problem I face everyday is the lack of computing resources. Very often my local machine crashes due to OOM by only a couple of jobs.
The ideal solution is to configure my development environment once and for all. The deployment should also be cheap in both time and economy. The cost of reset and migration should be minimal, so that I can easily recover from a broken instance.
Thoughts & Design
The most old-school approach is scripting all the setup that I had to do by hand. However, it is not trivial to script across different OS/platforms. I would have to write different scripts for different OS which do hopefully the same thing. Moreover, in the long term such a fragile system is not maintainable.
Now the Container comes to rescue. It suits my need for the following reasons:
- only 1 OS needs to be considered
- it can be run on most modern OS
- its behaviour is consistent across restarts
IDE
The first question is what IDE to use. Vscode does not fit well as it is fundamentally a desktop app and GUI support in containers is very limited. I chose Neovim as I am most familiar with Vim-like editors. Vim 8 is another possible candidate but I prefer Neovim because it's community-driven.
Neovim provides built-in language server protocol(LSP) support. All I need is to steal the configurations from all over the internet. Check my settings in the image.
Deployment
The next and the most difficult question is how to use the image I just built. It is not as simple as running docker run --rm -it onichandame/dev bash. Containers do not persist peripheral storage so all my codes not backed-up would be lost during restarts. Then mount a host path you may say. But remember that my initial problem is to access a single development environment from different locations. How am I supposed to access a project stored in my office from home?
Kubernetes(k8s) is the answer. I will not go into the details here, because it is quite straightforward once you decide to deploy it on k8s. I basically deploys to a k8s cluster on the internet. So I can access it from anywhere if only I have kubectl configured locally. A small hint: consider statefulset + ingress.
Local Setup
After all the hard work, I still need to setup my local machines in order to use the remote deployment. Only Windows setup is recorded here:
Other Attempts
Before the final answer was found, I have tried many approaches where many lessons were learned.
Local Container Runtime
Before discovering k8s, I tried running the container locally on different platforms.
Podman
Podman is a daemonless container runtime. It requires --security-opt label=disable to bypass SELinux for sharing files with the host.
Podman treats containers differently than Docker. Each plain container created by podman is not allocated with a new IP as it would require root permission. To communicate 2 rootless containers, a pod is needed to contain the 2 containers. The official guide tells how to setup pods and containers within pods. This guide tells how to setup communication between containers in the same pod. However, the creation of pod requires pulling image from k8s.gcr.io, which is blocked by GFW. Thus a mirror registry must be used.
According to man container-registries.conf, the system-wide configuration is /etc/containers/registries.conf. The location may differ on different platforms, check it carefully. In the default configuration shipped with CentOS 8, the format v1 is used. Mirrors are only available in format v2, therefore the configuration needs to be changed to v2:
unqualified-search-registries = ['docker.io', 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org']
[[registry]]
prefix="k8s.gcr.io"
location="k8s.gcr.io"
[[registry.mirror]]
location="registry.cn-hangzhou.aliyuncs.com/google_containers"
Thanks to this issue
After removing the old configuration and adding the above configuration, an AliCloud mirror is used. Now pods can be created.
The containers inside pods cannot communicate directly with host, namely port mapping. The port mapping needs to be set on the creation of the pods. For example:
podman pod create -p 3000:3000
Local Kubernetes
Before having access to a cluster on the internet, I had to play with local k8s deployments.
Minikube
The official method to setup a development cluster is minikube. But it heavily depends on the traditional VM stack, which means that it consumes more resources compared with other alternatives.
Kind
Kind starts a k8s cluster based on a local docker runtime. It provides an official executable to help manage the cluster. One can follow the official guides to install it.
The next step is to install an ingress controller. From my experience it would be very inconvenient if services are exposed to different ports instead of a single port but different domains.
According to the official doc, one can choose an ingress controller from 3: Ambassador, Contour and Nginx. Based on my experiments, only the ambassador works well in China. To install the ambassador controller, run the following commands.
kubectl apply -f https://github.com/datawire/ambassador-operator/releases/latest/download/ambassador-operator-crds.yaml
kubectl apply -n ambassador -f https://github.com/datawire/ambassador-operator/releases/latest/download/ambassador-operator-kind.yaml
After all the pods in the namespace ambassador are up, the controller is ready for use.
Another concern is that all the ingresses using ambassador should be annotated by kubernetes.io/ingress.class: ambassador.
To install other ingress controllers, you need to make sure that the gateway is installed on the node where the the extra port mapping rules are applied.
I choose to install Istio as it provides many more useful features relating to the monitoring stuff.
- Run
curl -L https://istio.io/downloadIstio | sh -to download the latest CLI of Istio. - Run
istioctl install -f istio.yamlwhere the istio.yaml can be found at here.
Now the Istio default profile has been installed.
The last manual setup is to label the namespace by istio-injection=enabled which tells Istio to inject sidecar proxy to every pod in the namespace. This is required to enable most core functionalities of Istio.
One caveat currently not resolved is the that the coredns service may stop resolving domain names after the host being restarted. This can be worked around by restarting the coredns deployment on host restart.
kubectl rollout restart deploy/coredns -n kube-system
K3D
K3d is a product of Rancher. It is based on k3s, a super-lightweight kubernetes distribution suitable for low-resource environments, such as IoT or local testing environment. Like Kind, K3d runs as many nodes as wished in containers to simulate a multi-node cluster. But this is much lighter than kind.
K3d relies on the command line arguments to customize the cluster. The command I use is:
k3d cluster create --no-lb --k3s-server-arg='--disable=traefik' -p 80:30001@agent[0] -p 443:30002@agent[0] -a 3 -v <path to workspace on host>:/git -v <path to registries configuration file>:/etc/rancher/k3s/registries.yaml dev
The arguments specifically disables the external load balancer and traefik. As I will install istio manually as a replacement. The port mapping argument is also for istio installation. the -a 3 argument instructs k3d to spin up 3 worker nodes. the volume mount argument maps the workspace on the host to all the nodes for persistent data storage. The configuration mounted to /etc/rancher/k3s/registries.yaml replaces the official registries of docker.io or gcr.io by the mirror registries to speed up the image pulling process.
The installation of istio follows the same instruction above.
Docker Desktop
Docker was originally made for Linux. Running on Windows is always non-trivial. Here I record the main issues solved for Docker Desktop on Windows 10.
Docker Engine
Docker Desktop mainly provides 2 engines: HyperV(legacy) and WSL 2.
HyperV works as a traditional VM hypervisor, which is utilized by Docker Desktop to maintain a VM where the docker daemon is run.
WSL 2 on the other hand, provides a deeply integrated Linux kernel. This is a faster solution.
The first step is to install WSL 2. Check this for the official guide.
Having WSL 2 installed, now check the Use the WSL 2 based engine option in the General settings of the Docker Desktop. Restart the docker daemon for this action to take effect. Note: by restarting the daemon, all the images in the old daemon will be lost!.
WSL 2 at this moment suffers from a fatal bug. If you try to start a Linux distro backed by WSL 2, and see the error message the attempted operation is not supported for the type of object referenced., it means that WinSock has caused the WSL 2 to collapse. According to this issue, running netsh winsock reset as admin works around the bug once and for all.
HyperV
No matter you choose HyperV or WSL 2, HyperV service must be enabled. However, MicroSoft made a very bad default behaviour that disables virtualization in BIOS once the HyperV related services are enabled. Thus it is required to go to BIOS and enable hardware virtualization after enabling HyperV every time.
Thanks to this issue
Misc
Keep Container Running
A container will exit as soon as the PID 1 process exits. This is not a problem for stateless containers. But on k8s, all containers have to run in background.
To run a container in background indefinitely, two options are available:
tail -f /dev/nullbash
The first option has a problem that it does not respond correctly to SIGKILL/SIGINT. Therefore the pod will be stuck in the Terminating stage forever.
The second option requires a pseudo-tty session to keep the bash from exiting.
To conclude, the tail approach has no elegant workaround for its problem, whereas the bash approach's problem can be fixed by simply providing a pseudo-tty session.
NOFILE Limit
When developing a project with plenty of source files and dependencies, the development server usually needs to watch the changes in all the source files. Sometimes it can exceed the system's watch file limit.
The watch file limit is defined by a hard limit and a soft limit. When the soft limit is exceeded, the user is warned. When the hard limit is hit, the watcher's process throws an error. Therefore both limits need to be raised.
The solution is to increase the host's NOFILE limit. The containers automatically inherits the host's limit. <a href={'./ulimit'}>Check this for the detailed setup.
UTF-8 Encoding
The base image is usually minimized for performance, so the default encoding isusually set to ASCII. This won't be a problem if the host machine has UTF-8 enabled and that container interface streams the output in the raw binary form. However, it gets problematic when using tmux inside the container as tmux needs to decode the characters then prints them.
The first step is to add locales to the container. In my case, I added Simplified Chinese by installing glibc-langpack-zh. Run locale -a to check if the desired language has been installed.
The last step is setting the default encoding to UTF-8 by adding export LANG="zh_CN.UTF-8" and export LC_ALL="zh_CN.UTF-8" to bashrc or the Dockerfile.
Port Forwarding
To access a web server running in a container from the host, port mapping is the only option. It is as simple as adding the flag -p <host port>:<container port> to the command creating the container. However, one caveat of this setup is that the port-mapping rules cannot be updated once the container has been created. Hence within the container, the web servers must listen on a predefined port. But this method does not solve all the problems. For example the dev server simply does not allow the user to specify a port, like Gatsby's fast refresh server.
One solution to this problem is to add routing rules inside the container. socat is a tool designed for this.
# redirect requests for the mapped port to the server port
socat TCP4-LISTEN:<mapped port>,fork,reuseaddr TCP4:localhost:<server port>