Linux Kernel Live Patching Engineer
Прямой работодатель Sardina Systems ( sardinasystems.com )
Опыт работы любой
Sardina Systems is an OpenInfra software platform company based on OpenStack and Kubernetes and is looking for a Linux Kernel Live Patching Engineer to join our team.
Position summary
Build and operate a live kernel patching feed for AlmaLinux 10. You own the full technical chain, from CVE triage to signed, tested RPMs published in a public repository. This is a hands-on kernel engineering role, not a management or release-coordination position.
Core responsibilities
Patch engineering
- Track AlmaLinux security advisories (ALSA) and upstream kernel CVEs.
- Triage CVEs against the published scope: critical, remotely exploitable or actively exploited vulnerabilities.
- Backport security fixes to the supported AlmaLinux 10 kernels, resolving conflicts across version differences.
- Generate live patch modules with klp-build. Initially, publish individual CVE patches; evaluate cumulative/atomic replace strategies as the pipeline matures.
- Identify CVEs that cannot be live patched (e.g. __init changes, data structure layout changes) and record them in the coverage-gap log.
Build and distribution pipeline
- Reproduce AlmaLinux kernel builds exactly (same source, config and compiler), e.g. in mock with the AlmaLinux buildroot, so modules match the running kernel.
- Maintain the automated pipeline: CVE scan, build plan, klp-build, RPM packaging, signing, test gate, createrepo_c publication.
- Plan and execute the move of support to new AlmaLinux kernel releases as they ship.
- Verify the exact client-side commands work on stock AlmaLinux 10 (dnf kpatch auto plus kpatch-patch-* packages).
Testing and release gate
- Run the mandatory pre-publication gate on every patch:
- load and unload on the target kernel in QEMU;
- run kernel livepatch selftests;
- for high-severity CVEs: apply and revert under load on a FishOS test cluster with running VMs and Ceph I/O.
- Document which tests were passed for each release.
- A patch that passes the gate is cleared for publication. Do not publish a patch that has failed the gate.
Release and withdrawal
- Publish approved patches with release notes listing the CVEs covered.
- Publish the source for every released patch alongside its RPM.
- If a published patch misbehaves: pull it from the repository, publish an advisory, and give users revert instructions.
- Maintain the runbook and coverage-gap log.
Secure Boot
- Manage the Sardina module-signing key and RPM signing key.
- Document and verify mokutil key enrolment for AlmaLinux users.
- Work with the Ops team to automate key enrolment on FishOS.
Required skills
- Linux kernel development: strong C, with hands-on experience backporting kernel patches or debugging kernel modules. Screening will test this directly.
- Live patching: practical experience with klp-build or kpatch, or demonstrated ability to learn them quickly from source.
- RPM and the RHEL/AlmaLinux ecosystem: kernel packaging (RPM/SRPM), mock, dnf, createrepo_c and repository metadata.
- Automation: Python or shell; building and maintaining CI-style pipelines.
- Signing: GPG key management and RPM signing; kernel module signature verification and MOK enrolment.
- English: working proficiency.
Preferred skills
- Spoken Russian and/or Mandarin, in addition to English. This is to work closely with Russian- and Mandarin-speaking team members.
- QEMU-based VM validation or canary testing.
- Upstream kernel contributions or involvement in the live patching community.
- Koji or similar build orchestration systems.
- Ceph I/O testing or storage-heavy workloads.
Key traits
- Boundary awareness: Willing to say plainly "this CVE needs a reboot" and "this kernel is out of scope" rather than forcing a risky patch
- Release discipline: Understands that a bad patch is the main reputational risk and will not skip the test gate under pressure
- Self-directed: Comfortable owning an end-to-end pipeline with minimal supervision
- Confidentiality: handles embargoed vulnerabilities and unreleased plans with discretion.
Working context
- Scope: AlmaLinux 10 only, starting with one kernel version.
- Infrastructure: runs separately from the existing Koji-based OpenStack build infrastructure, but follows the same automation approach.
- Primary tooling: klp-build, kpatch/kpatch-dnf, mock, createrepo_c, GPG, systemd timers, QEMU and a FishOS test cluster.
- Release authority: Routine patches are approved by the automated gate; no per-patch sign-off is required. A named release authority reviews weekly summaries, approves exceptions (e.g., a patch that cannot pass the full gate for a documented reason), and handles emergency withdrawals.
- Backup: one Ops team member is cross-trained on the test gate, publication and withdrawal steps early in the project.
Success Criteria
First milestone
- klp-build confirmed working on a real Alma 10 kernel
- End-to-end pipeline producing a signed, tested patch for one CVE class on one kernel version
- Test gate passed on at least two consecutive builds
- Coverage-gap log and runbook exist
Second milestone
- Pipeline operating reliably with the test gate on every build
- Secure Boot enrollment automated on FishOS
- Sources and tooling published under the appropriate license
- Runbook and coverage-gap log maintained and current
