Loki / Grafana / Prometheus / GitHub Actions
Hire Senior DevOps
Engineers in Morocco
One test decides whether you have observability: when something breaks, can you say what broke, why, and how bad it is, in under five minutes? If the answer involves SSH, you do not.
We build the logging, metrics, error tracking and alerting that answers it, plus the pipeline that ships safely. Morocco on UTC+0, so a deploy window gets a live person and not a ticket.
Free / you keep the findings either way
01 / The five minute test
Four primitives, or you are guessing
Most teams have one or two of these and call it monitoring. Logs with no metrics tells you something happened and not how bad. Metrics with no logs tells you the graph moved and not why. Errors with no alerting means somebody finds out from a customer.
This is the stack we deploy. It runs on Docker Compose on a single host, and it is genuinely enough for most of the companies that have been told they need a platform team.
Logs
Loki + Promtail
What happened, in order, with the request id that ties it together.
Promtail tails the Docker socket and ships every container stream to Loki with labels for service, container and environment. You search by label, then narrow.
Metrics
Prometheus + Grafana
How bad, and since when.
Error rate as a share of 5xx, latency at p50, p95 and p99, uptime, queue depth, failed job count. The numbers you would want at 3am, on one dashboard.
Errors
Sentry or GlitchTip
The stack trace, grouped, with the release that introduced it.
GlitchTip is the self-hosted option when the data cannot leave your infrastructure. Same wire format, so the decision is reversible.
Alerts
Grafana alerting
Somebody finds out without staring at a screen.
Tiered by severity, and every one of them carries what to do next. An alert nobody acts on is noise that trains people to ignore the next one.
Why Loki and not Elasticsearch
Loki indexes the labels, not the log body.
That single design decision is why it runs on a box you already have instead of a cluster you have to fund. Elasticsearch indexes every field of every line, which is powerful and is why teams end up paying more to store logs than to serve traffic. Loki is to logs what Prometheus is to metrics, down to the query language, so anyone who can read one can read the other.
02 / Logs you can actually query
Structured, correlated, and free of your customers' data
A log line written for a human reading a terminal is close to useless at volume. The moment logs are JSON with consistent keys, Loki can filter on them and a question that took an hour takes one query.
The field that matters most is the correlation id. Without it you have a pile of events. With it you have the story of one request across the web container, the worker and the scheduler.
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"json": {
"()": "pythonjsonlogger.jsonlogger.JsonFormatter",
# Every line carries the same keys, always.
"format": "%(asctime)s %(levelname)s %(name)s "
"%(request_id)s %(message)s",
},
},
"filters": {"request_id": {"()": "app.log.RequestIDFilter"}},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"formatter": "json",
"filters": ["request_id"],
},
},
"root": {"handlers": ["stdout"], "level": "INFO"},
}
# stdout, not a file. The container runtime collects it,
# Promtail ships it, and nothing rotates logs by hand.scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
# Labels are the index. Keep them low cardinality:
# service and environment, never user id or request id.
- source_labels: ["__meta_docker_container_name"]
target_label: container
- source_labels: ["__meta_docker_container_label_service"]
target_label: service
- target_label: environment
replacement: production
# A label per user would give Loki millions of streams
# and bring it to its knees. That goes in the line body,
# where it is filtered at query time instead.Never log secrets or PII unmasked
Tokens, API keys, passwords, card numbers, personal data. Logs get shipped, retained, backed up and read by more people than your database ever will, and a log store quietly becomes the least protected copy of your most sensitive data. Masking belongs in the formatter, so it is not a thing every developer has to remember.
03 / Alerting
Every alert answers three questions
What broke. How bad it is. What to do right now. An alert missing the third one is a notification, and people learn to swipe those away. That is how a real outage gets missed by a team that had an alert for it.
| Severity | Goes to | Why that channel |
|---|---|---|
| Low | Logged only | Reviewed in the weekly pass. Nobody is woken up for a warning. |
| Medium | Slack | Someone sees it inside working hours and decides. |
| High | Slack and email | It is chased today, not this week. |
| Critical | Slack, email and phone | Twilio or PagerDuty. Reserved for things that are actually worth a phone call, so the phone still means something. |
Alert on symptoms, not causes
Page on the checkout error rate, not on CPU. High CPU with happy customers is not an incident. A healthy-looking box with a broken checkout is.
Every alert links to a runbook
Even three lines. The person woken at 3am is not necessarily the person who built it, and often will not be once we have rolled off.
Delete the ones nobody acts on
An alert that has fired forty times and been acknowledged forty times is training your team to ignore alerts. Fix it or delete it, and we will push to do one of the two.
Health checks that check dependencies
A check returning 200 whenever Python is alive will report health while Postgres is unreachable. Check what the request path actually needs, and nothing it does not.
04 / Delivery
GitHub Actions, with the guards that matter
Most pipelines run tests then deploy. The interesting parts are the four lines that stop a bad day: not cancelling a deploy halfway, catching schema drift before it reaches production, gating on a human where it matters, and not holding long-lived cloud keys.
name: deploy
on:
push:
branches: [main]
permissions:
contents: read
id-token: write # OIDC, so no long-lived cloud keys
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
options: >-
--health-cmd pg_isready
--health-interval 10s --health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: pip }
- run: pip install -r requirements.txt
- run: ruff check .
# Fails if a model changed and nobody made the migration.
- run: python manage.py makemigrations --check --dry-run
- run: pytest -q
deploy:
needs: test # never deploys a red build
environment: production # required reviewers live here
concurrency:
group: production
cancel-in-progress: false # never kill a half-done deploy
runs-on: ubuntu-latest
steps:
- run: ./deploy.shcancel-in-progress: false on deploys
The default cancels the running job when a new commit lands. On a test run that saves money. On a deploy it means killing a process midway through migrations, and you now have a database in a state no code expects.
makemigrations --check in CI
Catches the model change that shipped without its migration. That mismatch is silent in review and loud in production.
OIDC instead of stored keys
The workflow exchanges a short-lived token for cloud credentials at run time. A leaked repository secret stops being a standing key to your infrastructure.
Pin actions and images
Not @main, not :latest. A pipeline that changes behaviour because somebody else shipped is a pipeline you cannot reason about on a bad morning.
05 / The argument
We will probably talk you out of Kubernetes
Kubernetes solves organisational problems, not code problems. It exists so that many teams can deploy independently without coordinating. If you are one team deploying one product, it buys you a hiring requirement, a permanent tax on everyone's attention, and slower delivery.
That is an opinion, we hold it publicly, and you are welcome to disagree with it on the call. Here is the honest version of when we would tell you the opposite.
| Your situation | What we would say |
|---|---|
| One product, one deploy pipeline, predictable traffic | Docker Compose on a host you understand. Put the saved months into the product. |
| Several teams that must ship without coordinating | Now Kubernetes is earning its keep. This is the problem it was built for. |
| Traffic that genuinely spikes unpredictably | Autoscaling is a real requirement. Managed Kubernetes or a serverless platform, depending on the workload. |
| Compliance demanding hard workload isolation | A legitimate reason, and the compliance programme drives the design rather than the other way round. |
| Infrastructure is the product you sell | You need it, and you should hire a platform team rather than an agency. |
| Somebody senior says it is best practice | Ask them which of the four rows above you are in. If none, it is resume-driven development and it will cost you a year. |
The long version, with the Compose files, is on our blog: Django observability on Docker Compose, without Kubernetes.
06 / If your stack is already decided
We work inside the tools you have bought
Plenty of companies have a security team, an existing contract and a standard. If logs have to land in Splunk because that is where the SOC looks, arguing about Loki is not a service to you. The open-source stack is our default, not a condition.
| Layer | Our default | If your stack says otherwise |
|---|---|---|
| Logs | Loki and Promtail | Splunk, Elastic, Datadog Logs, CloudWatch |
| Metrics | Prometheus | Datadog, New Relic, CloudWatch Metrics |
| Dashboards | Grafana | Splunk, Datadog, whatever the SOC already opens |
| Errors | Sentry or GlitchTip | Rollbar, Bugsnag, or your existing Sentry org |
| Paging | Grafana alerting | PagerDuty, Opsgenie, ServiceNow |
| Pipeline | GitHub Actions | GitLab CI, Azure DevOps, Jenkins if that is the reality |
The part that transfers either way
Structured events with a correlation id, alerts tiered by severity that say what to do, and a pipeline with real guards. Those are design decisions, not products. Get them right and the vendor underneath is a procurement question. Get them wrong and the most expensive observability platform on the market will not save you.
07 / Scope
What we own, and what we will not pretend to
We own this
- The observability stack, end to end, in your infrastructure
- Structured logging in the application, not just collection around it
- CI/CD pipelines with tests, gates and reproducible deploys
- Containerisation and the deployment topology
- Incident response during an agreed window, and the runbooks after it
- Secrets handling and pipeline permissions
We do not
- Round the clock on-call as a standalone product. That is a managed service provider and we will say so.
- Large multi-cluster Kubernetes estates. Different firm, different bench.
- Act as your compliance auditor. We build to the controls you give us.
- Own a platform you cannot operate. If your deploys stop working when we leave, we built the wrong thing.
08 / Rates
What it costs
$4,800–$7,200
per month / one senior DevOps engineer / full time
| Observability review | Free | What you can answer today, and what you cannot. |
| Stack build, scoped | From $6,000 | Logs, metrics, errors and alerts, wired and handed over. |
| One senior engineer, full time | $4,800–$7,200 / mo | Embedded, on your roadmap. |
| Part time beside your team | From $2,400 / mo | When you have the people, not the depth. |
Moves on seniority, whether you need on-call cover, contract length and how regulated your environment is. Morocco is why the range sits where it does: senior rates here are the lowest of any region Lemon.io tracks in its 2026 data, and the cause is local cost of living rather than a discount on the work.
09 / Questions
Straight answers
What does your DevOps team actually own versus advise on?
We own the observability stack, the CI/CD pipeline, containerisation and the deployment topology, and we write the runbooks. We advise on organisational things like on-call rotations and incident process, because those only work if your own team owns them.
Can you cover on-call, and in which hours?
An agreed window inside an engagement, yes. On UTC+0 we cover most of a European working day and four to five hours of a New York one. Round the clock as the main deliverable is a managed service provider, and we will tell you that rather than take the contract.
What is your Kubernetes and Terraform depth?
Terraform for infrastructure as code, yes. On Kubernetes we are deliberately not the right firm for a large multi-cluster estate, and our published position is that most teams asking for it do not need it. If you are running one already and want help operating it, say so on the call and we will be straight about what we can own.
How do you handle production access from Morocco?
Named identities with MFA, least-privilege roles scoped to the engagement, audit logging on, secrets in a secrets manager, and same-day revocation when somebody rolls off. Your accounts, not ours. Geography changes nothing about that model.
We already use Splunk and Datadog. Is that a problem?
No. The open-source stack is our default, not a requirement. Structured events with a correlation id, tiered alerting and a guarded pipeline are design decisions that transfer to any vendor. If logs must land in Splunk because that is where your SOC looks, that is where they land.
What does a DevOps engineer cost per month?
$4,800 to $7,200 a month full time, or from $2,400 part time alongside your own team. A scoped observability build starts around $6,000, and the review before any of it is free.
How long until we can answer the five minute question?
For a single application on Docker Compose, usually two to three weeks to have logs, metrics, errors and alerts wired and a dashboard your team actually opens. Longer if the application needs structured logging added first, which it often does, and that part is the real work.
Will we be able to run it after you leave?
That is the point. Everything lands as code in your repository, the dashboards are provisioned rather than clicked together, and the runbooks assume the reader is not us. If it only works while we are there, we built the wrong thing.
Try the five minute test
Think of the last thing that broke. Walk us through how you found out and how long it took to know why. Thirty minutes, with an engineer, and you keep the findings either way.
Also for you
AWS engineers who have been on call / Django engineers who read the query plan / every role we staff