Setting up services using Nix and Nixops can actually be pretty easy and fast!
Prerequisites
- Have Nix installed in your system. Nix is required for you to run
nix-shell. - Have VirtualBox installed.
Create the following files in a directory
File: loki.yaml
Description: This file is used to configure the Loki logger. This is just an example configuration, change it as you please.
# Enables authentication through the X-Scope-OrgID header, which must be present
# if true. If false, the OrgID will always be set to "fake".
auth_enabled: false
server:
http_listen_address: "0.0.0.0"
http_listen_port: 3100
ingester:
lifecycler:
address: "127.0.0.1"
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
schema_config:
configs:
- from: 2020-05-15
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h
storage_config:
boltdb:
directory: /tmp/loki/index
filesystem:
directory: /tmp/loki/chunks
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
File: grafanaService.nix
Description: The configuration file contains all the services we want to set up.
{
network.description = "GrafanaService";
grafanaservices = {
# Expose the Grafana and Grafana Loki service ports
networking.firewall.allowedTCPPorts = [ 2342 3100 ];
# Set up Grafana
services.grafana = {
enable = true;
port = 2342;
addr = "0.0.0.0";
provision = {
enable = true;
# Set up the datasources
datasources = [
{
name = "Prometheus";
type = "prometheus";
access = "proxy";
url = "http://localhost:9001";
isDefault = true;
}
{
name = "Loki";
type = "loki";
access = "proxy";
url = "http://localhost:3100";
isDefault = true;
}
];
};
};
services.loki = {
enable = true;
configFile = ./loki.yaml;
};
# Set up Prometheus for metric scraping
services.prometheus = {
enable = true;
port = 9001;
# Export the current system metrics
exporters = {
node = {
enable = true;
enabledCollectors = [ "systemd" ];
port = 9002;
};
};
scrapeConfigs = [
# Scrape the current system
{
job_name = "GrafanaService system
static_configs = [{
targets = [ "127.0.0.1:9002];
}];
}
# Scrape the Loki service
{
job_name = "Loki service"
static_configs = [{
targets = [ "127.0.0.1:3100" ];
}];
}
];
};
}
}
File: grafanaVbox.nix
Description: This file contains our "target" machines, in this case it's a VirtualBox but it could easily be a Google Cloud Machine or a VPS server.
let
vbserver = {
deployment.targetEnv = "virtualbox";
# Memory size in case of VM
deployment.virtualbox.memorySize = 512;
# Amount of vCPUs assigned to VM
deployment.virtualbox.vcpu = 1;
# Set so we don't get Vbox popping up in the OS
deployment.virtualbox.headless = true;
};
in
{
grafanaservices = vbserver;
}
Command instruction
- Get Nixops in shell instance:
nix-shell -p nixops
note: Those examples work with nixops version 1. Version 2 is currently (at the time of writing) in unstable and I have not tested the above scripts using nixops v2. - Create a Nixops:
nixops create grafanaService.nix grafanaVbox.nix -d exampleDeployment - Deploy:
nixops deploy -d exampleDeployment --force-reboot
note: The --force-reboot flag is required when first booting up virtualbox, if it's not added the virtualbox instance will fail to reboot and you will get an error. - The deployment should now be up and running. You should find the IP address of the deployment in the logs. You can find your Grafana instance at
instance-ip-address:2324
The Grafana instance should have the Loki and Prometheus instance configured under the "sources" tab of Grafana and Prometheus should be collecting metrics from the system and from the Loki instance.
DONE! Thanks for reading!
Huh? Aren't you satisfied yet?
Nixops Cheatsheet
- To deploy after updating a something in the configuration simply run:
nixops deploy -d exampleDeployment(this also works with other configuration files, such as the loki.yaml) - To stop the deployment and shut down the virtualbox instance
nixops stop -d exampleDeployment - To restart the deployment:
nixops reboot -d exampleDeployment - To delete a deployment:
nixops delete -d exampleDeployment(deployment must be shut down first) - To list all deployments:
nixops list - To ssh into the instance with root access:
nixops ssh -d exampleDeployment grafanaservices
note: Only use for debugging (i.esystemctl status grafana.serviceto see if a service is running), if you want to change something it's better to instead change the grafanaService.nix and then run the deploy command. Nix will take care of applying the neccessary changes. - You can otherwise find other nixops commands by running
nixops --help
Further development and Nixops tips
To get things straight; Nixops (and also Nix!) have sections that are severely under documented. There's some good resources from conferance talks and some videos on Youtube but if you try to learn from any written documentation the documentation can often outdated (with breaking changes) or isn't addressing crucial concepts that you need to know. There's also some information to find in Github tickets but finding this information takes hours of research. That doesn't help much considering the Nix ecosystem itself also has a lot of concepts that differantiate if from other Linux distros, languages and package managagers.
That is why I've decided to write about Nix and Nixops and I hope it's helpful to others! As a small introduction to Nixops; Nixops actually adds a subset option to the nix configuration. One of those are deployment, the parameters for deployment can only be found in the code of nixops.
The same goes with Nix, for example there's a way to get the private IPv4 from a machine called config.networking.privateIPv4 (the public version is config.networking.publicIPv4 but this can be null at times)
Finding Nix packages and package options
If you want to find any packages you can find those at search.nixos.org
Normally you can find options by searching for them at the Nixops package search engine. Here's the options search shortcut. You can both search for Nix specific options and package options.
What if you want to add more machines with services?
This is a bit beyond the scope of the this guide. However Nixops has a nifty function which can create ssh tunnels between the different machines.
To create a tunnel you simply add a new deployment.encryptedLinksTo = [ "nameOfService" ]; on one of the machines. This will create a peer to peer connection between the two machines. It will also edit the host file accordingly so you can reference the hostname of the other machine fairly easily.
Want metrics on the machine? A trick is to add the Promtheus node exporter if you want metrics from that machine. To do that, simply add this to any service.
services.prometheus = {
exporters = {
node = {
enable = true;
enabledCollectors = [ "systemd" ];
port = 9002;
};
};
};
This won't add Prometheus itself but it will add a program that exports metrics that Prometheus can scrape at port 9002 of that machine. Tadaa! You suddenly got metrics for all the machines you want, super easy.
Actually let's do a more complete example, let's extend the grafanaService.nix and add a new machine with a single Cassandra node.
# ... rest of the grafanaService.nix
cassandra =
{ config, pkgs, ...}:
{
deployment.encryptedLinksTo = [ "grafanaservices" ];
services.cassandra = {
enable = true;
# Undocumented nix parameter
listenAddress = config.networking.privateIPv4;
# Set itself as seed
seedAddresses = [config.networking.privateIPv4];
};
services.prometheus = {
exporters = {
node = {
enable = true;
enabledCollectors = [ "systemd" ];
port = 9002;
};
};
};
# Enable to allow access to ports
# The SSH tunnel allows access to all ports between the two linked machines
#networking.firewall.allowedTCPPorts = [ 7000 7001 7199 9042 ];
};
# Remember to add closing bracket '}'
Change the virtualbox hardware configuration
let
vbserver = {
deployment.targetEnv = "virtualbox";
# Memory size in case of VM, increase memory if it's required!
deployment.virtualbox.memorySize = 512;
# Amount of vCPUs assigned to VM
deployment.virtualbox.vcpu = 1;
# Set so we don't get Vbox popping up in the OS
deployment.virtualbox.headless = true;
};
in
{
grafanaservices = vbserver;
cassandra = vbserver;
}
note: remember to increase the memory if the cassandra node requires it. It may need more than 1024MB of RAM.
Since this example also exports the metrics, you can also add the cassandra node to scrapeConfig inside the grafanaService machine configuration:
{
job_name = "Cassandra service"
static_configs = [{
targets = [ "cassandra:3100" ];
}];
}
After running nixops deploy -d exampleDeployment, you should be able to find the cassandra metrics through exploring prometheus in Grafana.
Thanks for reading! Be sure to leave a comment if you find any typos, got any questions or if there's something that just doesn't sit right