Self Hosting Guide

A guide for getting started with Self Hosting devices including software and hardware that will make you a better and more efficient Self Hosting.

Note: You can easily convert this markdown file to a PDF in VSCode using this handy extension Markdown PDF.

Getting Started with Self-Hosting

Self-Hosting is the practice of locally hosting(on premises & private web servers) and managing software applications by a person or organization instead of monthly subscriptions from Software as a service (SaaS) providers.

Most self-hosted software can be installed using Docker, a packaging system which allows software to bundle their configuration and dependencies and isolate them from your operating system. Software using docker can be installed using the command line or via graphical interfaces such as Portainer. Software is installed with Docker by downloading an image file containing the application, then creating a copy that sets up its own dependencies and configuration within what is called a container. Without containers you would often need to install different versions of the same programming languages or tools to satisfy the dependencies for the software you want to use which can get complicated.

Tools for Self-Hosting

Back to the Top

Containers

Back to the Top

Container is a standard unit of software that packages up code and all its dependencies(including CPU, memory, file storage, and network connections) so the application runs quickly and reliably from one computing environment to another.

Application Container Security Guide | NIST (PDF)

Container Image is a lightweight, standalone, executable package of software that includes everything needed to run an application such as the code, runtime, system tools, system libraries, and settings.

Best places to get Container Images:

Docker Compose is a tool that was developed to help define and share multi-container applications. With Compose, we can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

Kompose is a conversion tool for Docker Compose to container orchestrators such as Kubernetes or OpenShift.

Containerd is a daemon that manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond. It is available for Linux and Windows.

Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images. Podman provides a command line interface (CLI) familiar to anyone who has used the Docker Container Engine.

Lima is a tool that launches Linux virtual machines with automatic file sharing and port forwarding (similar to WSL2), and containerd. It's a great free and open-source alternative for Docker Desktop.

Colima is a container runtimes on macOS (and Linux) with minimal setup.

Portainer Community Edition is a lightweight service delivery platform for containerized applications that can be used to manage Docker, Swarm, Kubernetes and ACI environments. It is designed to be as simple to deploy as it is to use.

Yacht is a container management UI with a focus on templates and 1-click deployments.

Kitematic is a simple application for managing Docker containers on Mac, Linux and Windows letting you control your app containers from a graphical user interface (GUI).

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Red Hat Quay is a project that Builds, Stores, and Distributes your Applications and Containers.

Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including OCI and Docker).

Shipwright is a WebUI to generate templates for Yacht, Portainer, Docker-Compose, and Unraid.

Autoheal monitors and restarts unhealthy docker containers.

Dozzle is a small lightweight application with a web based interface to monitor Docker logs. It doesn’t store any log files. It is for live monitoring of your container logs only.

Diun is a tool that receive notifications when a Docker image is updated on a Docker registry.

WatchTower is a process for automating Docker container base image updates.

Nginx Proxy is an automation tool that sets up a container running nginx and docker-gen. Docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped.

Visual Studio Code Dev Containers is an extension that lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code's full feature set. A devcontainer.json file in your project tells VS Code how to access (or create) a development container with a well-defined tool and runtime stack.

CI/CD

Back to the Top

CI/CD: Continuous Integration and Continuous Delivery

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to define and execute Pipelines inside Docker containers.

Travis CI is a hosted continuous integration service used to build and test software projects hosted at GitHub.

Circle CI is a continuous integration and continuous delivery platform that helps software teams work smarter, faster.

Buddy is a fully-featured DevOps platform with no learning curve that packs everything you need from a CI/CD tool.

Buildbot is a continuous integration tool which automates the compile or test cycle required to validate changes to the project code base. It queues jobs, executes the jobs when the required resources are available, and reports the results.

Development

Back to the Top

Proxmox VE(Virtual Environment) is an open-source platform for enterprise virtualization. It has a built-in web interface that you can use to easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

Code-Server is Visual Studio Code running on a remote server, accessible through the browser.

Visual Studio Live Share is a service/extension that enables you to collaboratively edit and debug with others in real time, regardless of the programming languages you're using or app types you're building. You can instantly and securely share your current project, start a joint debugging session, share terminal instances, forward localhost web apps, have voice calls, and more.

GistPad is a Visual Studio Code extension that allows you to edit GitHub Gists and repositories from the comfort of your favorite editor. You can open, create, delete, fork and star gists and repositories, and then seamlessly begin editing files as if they were local, without ever cloning, pushing or pulling anything.

Live Server is an extension for Visual Studio Code that launches a development local Server with live reload feature for static & dynamic pages.

Gitea is a community managed painless self-hosted Git service.

GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. Self-host GitLab on your own servers, in a container, or on a cloud provider.

Soft Serve is a tasty, self-hostable Git server for the command line.

Glow is a terminal based markdown reader designed from the ground up to bring out the beauty—and power—of the CLI. It's used to discover markdown files, read documentation directly on the command line and stash markdown files to your own private collection, so you can read them anywhere.

Node-Red is a low-code programming for event-driven applications.

Cloud

Back to the Top

Linode

Linode is a cloud hosting company that provides virtual private servers and variety of other cloud services.

Linode Tools

Linode Cloud Manager is a user- and mobile-friendly interface to deploy and manage virtual machines, configure networking, and control user accounts.

Linode API is a tool that makes easy to configure, manage, and deploy user management, billing, support tickets, and more with programmatic access to Linode products and services.

Linode CLI is a tool to deploy and manage Linux servers from Linode without leaving the command line.

Linode Images is a service to capture, store, and deploy your custom images across Linodes or data centers. Easily create your own raw disk image and upload a compressed .gz image file (up to 5 GB) using the Cloud Manager or API to easily deploy to the Linode size and data center you need.

Linode Integrations is a collection of integrations lets you connect infrastructure and dev tools to the Linode platform. That let's you manage your Linode resources using the tools you know and love.

StackScripts is a tool to automatically configure new Linode instances using simple scripts. Create your own StackScript or browse the community StackScript library.

Linode Bare Metal is the single-tenant solution for applications and organizations with security, compliance, and performance needs. Bare Metal combines direct hardware access and the flexibility of a virtual machine.

NextCloud

Back to the Top

Nextcloud is an industry-leading, on-premises content collaboration platform for file sync & share and communication server. It is fully open source and you can host it yourself or pay a company to do it for you. Also checkout the following links below:

Nexcloud login screen

Nextcloud Hub is a tool that allows you to share and collaborate on documents, send and receive email, manage your calendar and have video chats without data leaks. As fully on-premises solution, Nextcloud Hub provides the benefits of online collaboration without the compliance and security risks.

Nexcloud Hub

Nextcloud AIO (All In One) is a tool that provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Nextcloud Desktop Client is a tool to synchronize files from Nextcloud Server with your computer.

Nextcloud Deck is a kanban style organization tool aimed at personal planning and project organization for teams integrated with Nextcloud.

Nextcloud Files is a tool tool that allows your employees have easy access to their files, photos and documents to work and can share and collaborate with team members, customers and partners. So IT knows nobody besides those they shared with has access to those files.

Nextcloud Talk is a tool that protects your communication better than other team collaboration platforms like Microsoft Teams or Slack, making sure your data stays on your servers. It also goes further than other encrypted communication technologies by keeping even metadata from leaking.

Nextcloud Home is a tool that allows you store your documents, calendar, contacts and photos on your server at home, at one of at one Nextcloud's providers or in a data center you trust.

Nextcloud Enterprise is a service that gives professional organizations software optimized and tested for mission critical environments.

[Nextcloud Outlook Integration](https://nextcloud.com/outlook/ is a tool that automatically upload files to replace large attachments or integrate Calendars and Contacts in Microsoft Outlook.

Collabora Online in Nextcloud is a powerful LibreOffice-based online office suite with collaborative editing, which supports all major document, spreadsheet and presentation file formats and works in all modern browsers.

ONLYOFFICE integration in Nextcloud is a service that empowers your users to collaborate on office documents with team members in real time. It has compatibility with Microsoft Office formats means perfect documents, every time.

Nextcloud VM(virtual machine appliance) is a set of carefully crafted family of *nix scripts, which interactively guide you through a quality-controlled installation of a Nextcloud instance for Home/SME Server and scripts for Raspberry Pi 4. It is Community developed and maintained.

DigitalOcean

Back to the Top

DigitalOcean is a cloud infrastructure provider that provides developers cloud services that help to deploy and scale applications that run simultaneously on multiple computers with data centers worldwide.

DigitalOcean Tools

DigitalOcean API is a service that manages your DigitalOcean infrastructure with our RESTful API.

DigitalOcean Client libraries is a collection of libraries lets you use the DigitalOcean API in a variety of programming languages.

DigitalOcean CLI is a service that manages your DigitalOcean infrastructure through your terminal with our open source Command Line Interface (CLI).

Terraform provider is a service that allows the user treat their DigitalOcean infrastructure like code with Terraform.

DigitalOcean Custom images is a service that quickly builds your environment in the cloud by provisioning servers with your own custom image, or choose from various Linux distributions.

Container Registry is a service that easily stores, manages, and protects private container images.

Remote Access

Back to the Top

FreeRDP is a free remote desktop protocol library and clients.

Rustdesk is an open source virtual/remote desktop infrastructure for everyone. Display and control your PC (Windows, macOS, and Linux) and Android devices.

TinyPilot is a tool that enables KVM over IP letting you control any computer remotely.

X2Go is open source remote desktop software for Linux that uses a modified NX 3 protocol. It gives remote access to a Linux system's GUI.

Apache Guacamole is a clientless remote desktop gateway. It supports standard protocols like VNC, RDP, and SSH.

Remotely is a remote control and remote scripting solution, built with .NET 6, Blazor, SignalR Core, and WebRTC.

P2P Remote Desktop is a portable, no configuration or installation needed remote desktop tool.

Cloudflare Tunnel is a tunneling daemon that proxies traffic from the Cloudflare network to your origins. This daemon sits between Cloudflare network and your origin (a webserver). This attracts client requests and sends them to you via this daemon, without requiring you to poke holes on your firewall and your origin(webserver) can remain as closed as possible.

WireGuard® is a straight-forward, fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec while avoiding the massive headache. WireGuard is designed as a general-purpose VPN for running on embedded interfaces and super computers alike, fit for many circumstances. Initially released for the Linux kernel, it is now cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Tailscale is a WireGuard-based app that makes secure, private networks easy for teams of any scale. It works like an overlay network between the computers of your networks using all kinds of NAT traversal sorcery.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

MeshCentral is a full computer management web site. It can run your own web server to remotely manage and control computers on a local network or anywhere on the internet. Once you get the server started, create device group and download and install an agent on each computer you want to manage.

VNC Viewer is a free remote desktop application that use can use on your iPhone, iPad, Mac, Windows and Linux computers from anywhere in the world.

TightVNC is a free remote desktop application. It can see the desktop of a remote machine and control it with your local mouse and keyboard, just like you would do it sitting in the front of that computer.

Virtualization

Back to the Top

HVM (Hardware Virtual Machine) is a virtualization type that provides the ability to run an operating system directly on top of a virtual machine without any modification, as if it were run on the bare-metal hardware.

PV(ParaVirtualization) is an efficient and lightweight virtualization technique introduced by the Xen Project team, later adopted by other virtualization solutions. PV does not require virtualization extensions from the host CPU and thus enables virtualization on hardware architectures that do not support Hardware-assisted virtualization.

Network functions virtualization (NFV) is the replacement of network appliance hardware with virtual machines. The virtual machines use a hypervisor to run networking software and processes such as routing and load balancing. NFV allows for the separation of communication services from dedicated hardware, such as routers and firewalls. This separation means network operations can provide new services dynamically and without installing new hardware. Deploying network components with network functions virtualization only takes hours compared to months like with traditional networking solutions.

Software Defined Networking (SDN) is an approach to networking that uses software-based controllers or application programming interfaces (APIs) to communicate with underlying hardware infrastructure and direct traffic on a network. This model differs from that of traditional networks, which use dedicated hardware devices (routers and switches) to control network traffic.

Virtualized Infrastructure Manager (VIM) is a service delivery and reduce costs with high performance lifecycle management Manage the full lifecycle of the software and hardware comprising your NFV infrastructure (NFVI), and maintaining a live inventory and allocation plan of both physical and virtual resources.

Management and Orchestration(MANO) is an ETSI-hosted initiative to develop an Open Source NFV Management and Orchestration (MANO) software stack aligned with ETSI NFV. Two of the key components of the ETSI NFV architectural framework are the NFV Orchestrator and VNF Manager, known as NFV MANO.

Magma is an open source software platform that gives network operators an open, flexible and extendable mobile core network solution. Their mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. Magma is 3GPP generation (2G, 3G, 4G or upcoming 5G networks) and access network agnostic (cellular or WiFi). It can flexibly support a radio access network with minimal development and deployment effort.

OpenRAN is an intelligent Radio Access Network(RAN) integrated on general purpose platforms with open interface between software defined functions. Open RANecosystem enables enormous flexibility and interoperability with a complete openess to multi-vendor deployments.

Open vSwitch(OVS)is an open source production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).

Edge is a distributed computing framework that brings enterprise applications closer to data sources such as IoT devices or local edge servers. This proximity to data at its source can deliver strong business benefits, including faster insights, improved response times and better bandwidth availability.

Multi-access edge computing (MEC) is an Industry Specification Group (ISG) within ETSI to create a standardized, open environment which will allow the efficient and seamless integration of applications from vendors, service providers, and third-parties across multi-vendor Multi-access Edge Computing platforms.

Virtualized network functions(VNFs) is a software application used in a Network Functions Virtualization (NFV) implementation that has well defined interfaces, and provides one or more component networking functions in a defined way. For example, a security VNF provides Network Address Translation (NAT) and firewall component functions.

Cloud-Native Network Functions(CNF) is a network function designed and implemented to run inside containers. CNFs inherit all the cloud native architectural and operational principles including Kubernetes(K8s) lifecycle management, agility, resilience, and observability.

Physical Network Function(PNF) is a physical network node which has not undergone virtualization. Both PNFs and VNFs (Virtualized Network Functions) can be used to form an overall Network Service.

Network functions virtualization infrastructure(NFVI) is the foundation of the overall NFV architecture. It provides the physical compute, storage, and networking hardware that hosts the VNFs. Each NFVI block can be thought of as an NFVI node and many nodes can be deployed and controlled geographically.

Virtualization-based Security (VBS) is a hardware virtualization feature to create and isolate a secure region of memory from the normal operating system.

Hypervisor-Enforced Code Integrity (HVCI) is a mechanism whereby a hypervisor, such as Hyper-V, uses hardware virtualization to protect kernel-mode processes against the injection and execution of malicious or unverified code. Code integrity validation is performed in a secure environment that is resistant to attack from malicious software, and page permissions for kernel mode are set and maintained by the hypervisor.

NVIDIA virtual GPU (vGPU) is a software enables powerful GPU performance for workloads ranging from graphics-rich virtual workstations to data science and AI, enabling IT to leverage the management and security benefits of virtualization as well as the performance of NVIDIA GPUs required for modern workloads.

AMD MxGPU is a hardware-based virtualized GPU solution, is built on industry standard SR-IOV (Single-Root I/O Virtualization) technology and allows multiple virtualized users per physical GPU to work remotely.

Proxmox Virtual Environment(VE) is a complete open-source platform for enterprise virtualization. It inlcudes a built-in web interface that you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko.

QEMU is a fast processor emulator using a portable dynamic translator. QEMU emulates a full system, including a processor and various peripherals. It can be used to launch a different Operating System without rebooting the PC or to debug system code.

Quickemu is a program that quickly create and run optimised Windows, macOS and Linux desktop virtual machines.

Hyper-V enables running virtualized computer systems on top of a physical host. These virtualized systems can be used and managed just as if they were physical computer systems, however they exist in virtualized and isolated environment. Special software called a hypervisor manages access between the virtual systems and the physical hardware resources. Virtualization enables quick deployment of computer systems, a way to quickly restore systems to a previously known good state, and the ability to migrate systems between physical hosts.

Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of KVM. The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms. Cloud workloads refers to those that are usually run by customers inside a cloud provider. Cloud Hypervisor is implemented in Rust and is based on the rust-vmm crates.

VirtManager is a graphical tool for managing virtual machines via libvirt. Most usage is with QEMU/KVM virtual machines, but Xen and libvirt LXC containers are well supported. Common operations for any libvirt driver should work.

oVirt is an open-source distributed virtualization solution, designed to manage your entire enterprise infrastructure. oVirt uses the trusted KVM hypervisor and is built upon several other community projects, including libvirt, Gluster, PatternFly, and Ansible. Founded by Red Hat as a community project on which Red Hat Enterprise Virtualization is based allowing for centralized management of virtual machines, compute, storage and networking resources, from an easy-to-use web-based front-end with platform independent access.

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models. It runs workloads in lightweight virtual machines, called microVMs, which combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of containers.

Foreman is a free open source project that gives you the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage your servers life cycle, on-premises or in the cloud.

Harvester is an open source hyper-converged infrastructure (HCI) software built on Kubernetes.

Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.

HyperKit is a toolkit for embedding hypervisor capabilities in your application. It includes a complete hypervisor, based on xhyve/bhyve, which is optimized for lightweight virtual machines and container deployment. It is designed to be interfaced with higher-level components such as the VPNKit and DataKit. HyperKit currently only supports macOS using the Hypervisor.framework making it a core component of Docker Desktop for Mac.

Intel® Graphics Virtualization Technology (Intel® GVT) is a full GPU virtualization solution with mediated pass-through, starting from 4th generation Intel Core (TM) processors with Intel processor graphics(Broadwell and newer). It can be used to virtualize the GPU for multiple guest virtual machines, effectively providing near-native graphics performance in the virtual machine and still letting your host use the virtualized GPU normally.

Apple Hypervisor is a frameowrk that builds virtualization solutions on top of a lightweight hypervisor, without third-party kernel extensions. Hypervisor provides C APIs so you can interact with virtualization technologies in user space, without writing kernel extensions (KEXTs). As a result, the apps you create using this framework are suitable for distribution on the Mac App Store.

Apple Virtualization Framework is a framework that provides high-level APIs for creating and managing virtual machines on Apple silicon and Intel-based Mac computers. This framework is used to boot and run a Linux-based operating system in a custom environment that you define. It also supports the Virtio specification, which defines standard interfaces for many device types, including network, socket, serial port, storage, entropy, and memory-balloon devices.

Apple Paravirtualized Graphics Framework is a framework that implements hardware-accelerated graphics for macOS running in a virtual machine, hereafter known as the guest. The operating system provides a graphics driver that runs inside the guest, communicating with the framework in the host operating system to take advantage of Metal-accelerated graphics.

Xen is focused on advancing virtualization in a number of different commercial and open source applications, including server virtualization, Infrastructure as a Services (IaaS), desktop virtualization, security applications, embedded and hardware appliances, and automotive/aviation.

Ganeti is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. Once installed, the tool assumes management of the virtual instances (Xen DomU).

Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.

Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past. It provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

Password Management

Back to the Top

Bitwarden is a free and open-source password management service that stores sensitive information such as website credentials in an encrypted vault.

Bitwarden Server is a project contains the APIs, database, and other core infrastructure items needed for the "backend" of all bitwarden client applications. Checkout Bitwarden's self-hosted release repository.

Vaultwarden is an unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs.

AuthPass.app is an Open-Source Password Manager for mobile and desktop that is Keepass 2.x (kdbx 3.x) compatible.

pass is an open-source unix-based password utilitiy with various gui clients

Network Tools

Back to the Top

MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth.

Mongoose is a networking library for C/C++. It implements event-driven non-blocking APIs for TCP, UDP, HTTP, WebSocket, MQTT. It is designed for connecting devices and bringing them online.

Eclipse Mosquitto is an open source implementation of a server for version 5.0, 3.1.1, and 3.1 of the MQTT protocol.

Ejabberd is an open-source, robust, scalable and extensible realtime platform built using Erlang/OTP, that includes XMPP Server, MQTT Broker and SIP Service.

LibreSpeed is a network speed test tool that can be run on your LAN or hosted in the cloud.

SmokePing is a deluxe latency measurement tool. It can measure, store and display latency, latency distribution and packet loss. It uses RRDtool to maintain a longterm data-store and to draw pretty graphs, giving up to the minute information on the state of each network connection.

Cockpit is a web-based graphical interface for servers, intended for everyone. It uses your system’s normal user logins and privileges. Network-wide logins are also supported through single-sign-on and other authentication techniques.

NetBox is a leading solution for modeling and documenting modern networks. It combines the traditional disciplines of IP address management (IPAM) and datacenter infrastructure management (DCIM) with powerful APIs and extensions, NetBox provides the ideal "source of truth" to power network automation.

Nginx proxy manager (NPM) is a reverse proxy management system running on Docker. It's easy to set up and does not require users to know how to work with Nginx servers or SSL certificates.

Netdata is high-fidelity infrastructure monitoring and troubleshooting, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

Pi-hole is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software, intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations.

AdGuard Home is a DNS relay station with ad/tracker/other blocking, IP address redirections, and DNS-over-HTTPS.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Supabase is an open source Firebase alternative. It is building the features of Firebase using enterprise-grade open source tools.

Plik is a scalable & friendly temporary file upload system (Wetransfer like) in golang.

Restify is a framework, utilizing connect style middleware for building REST APIs.

Traefik is an open source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them. What sets Traefik apart, besides its many features, is that it automatically discovers the right configuration for your services.

DuckDNS is a free service that allows you to bind your own favorite subdomain under duckdns.org to the public IP address in use from your router, even though such address is dynamically allocated by your internet service provider and therefore changes over time.

Trust-DNS is a Rust based DNS client, server, and Resolver, built to be safe and secure from the ground up.

Hugo is a static HTML and CSS website generator written in Go. It is optimized for speed, ease of use, and configurability. Hugo takes a directory with content and templates and renders them into a full HTML website.

sshuttle is a transparent proxy server that works as a poor man's VPN that forwards connection over ssh. It works with Linux and MacOS and supports DNS tunneling.

NetHopper is a Multi-Cloud Application Network as a Service. The easiest way to visualize, connect, secure, protect, and monitor microservices across any cluster, site, cloud, or network.

Cypress is a tool that makes it fast, easy and reliable testing for anything that runs in a browser.

Kimchi is an HTML5 based management tool for KVM. It is designed to make it as easy as possible to get started with KVM and create your first guest.

ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere.

FreeRDP is a free remote desktop protocol library and clients.

Pimox is a port of Proxmox to the Raspberry Pi allowing you to build a Proxmox cluster of Rapberry Pi's or even a hybrid cluster of Pis and x86 hardware.

PiKVM is a very simple and fully functional Raspberry Pi-based KVM over IP.

Firezone is a self-hosted WireGuard®-based VPN server and Linux firewall.

Monitoring

Back to the Top

Proxmox Mail Gateway is an open-source email security solution protecting your mail server against all email threats from the moment they emerge.

M2MLabs MainSpring is an application framework for building machine-to-machine applications like vehicle tracking or machine remote monitoring. In such applications typically a remote device equipped with sensors (e.g. gps, temperature, pressure) and actors communicates with a server application that is running the device communication protocol, device configuration, storage of data sent by the devices as well as the application business logic and the presentation layer.

VictoriaMetrics is a fast and scalable open source time series database and monitoring solution which exists in a Single and in a cluster version. It is compatible with Prometheus pull model and supports a wide variety of ingestion protocols: Influx, Graphite, Prometheus remote_write, Prometheus exposion format, OpenTSDB put message, JSON line format, Arbitrary CSV data, native binary formant, DataDog agent or DogStatsD; as way as many ways to query data via PromQL or MetricsQL from Grafana or own VMUI.

InfluxDB is an open source time series database, purpose-built by InfluxData for monitoring metrics and events, provides real-time visibility into stacks, sensors, and systems. Use InfluxDB to capture, analyze, and store millions of points per second, meet demanding SLA's, and chart a path to automation.

Grafana is a tool that allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.

Wyze is a great security and monitoring application to live stream HD video from the security cameras from anywhere in the world.

Uptime Kuma is a fancy self-hosted monitoring tool.

Gatus is a developer-oriented health dashboard that gives you the ability to monitor your services using HTTP, ICMP, TCP, and even DNS queries as well as evaluate the result of said queries by using a list of conditions on values like the status code, the response time, the certificate expiration, the body and many others.

Upptime is the open-source uptime monitor and status page, powered entirely by GitHub Actions, Issues, and Pages.

HertzBeat is an open-source, real-time monitoring system with custom-monitor and agentless. It supports web service, database, os, middleware and more.

Tautulli is a python based web application for monitoring, analytics and notifications for Plex Media Server.

Flower is a web based tool for monitoring and administrating Celery clusters.

Weave Scope is a tool for Troubleshooting & Monitoring for Docker & Kubernetes. It automatically generates a map of your application, enabling you to intuitively understand, monitor, and control your containerized, microservices-based application.

Statping (Status Page & Monitoring Server) is an easy to use Status Page for your websites and applications. Statping will automatically fetch the application and render a beautiful status page with tons of features for you to build an even better status page.

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow.

Ciao is a tool that checks HTTP(S) URL endpoints for a HTTP status code (or errors on the lower TCP stack) and sends a notification on status change via E-Mail or Webhooks.

Server is a simple server for sending and receiving messages in real-time per WebSocket.

Ngxtop is a real-time metrics for nginx server (and others).

Blocky is a fast and lightweight DNS proxy as ad-blocker for local network with many features

Dashy is a self-hostable personal dashboard built for you. Includes status-checking, widgets, themes, icon packs, a UI editor and tons more.

Netdata is high-fidelity infrastructure monitoring and troubleshooting, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

Restic is a modern backup program that can back up your files: from Linux, BSD, Mac and Windows. To many different storage types, including self-hosted and online services. easily, being a single executable that you can run without a server or complex setup. effectively, only transferring the parts that actually changed in the files you back up.

Autorestic is a wrapper around the amazing restic. While being amazing the restic cli can be a bit overwhelming and difficult to manage if you have many different locations that you want to backup to multiple locations.

MinIO is a high performance object storage tool that provides the world's fastest object storage server. With READ/WRITE speeds of 325 GiB/s and 165 GiB/s on standard hardware with default parity (EC:4), object storage can operate as the primary storage tier for a diverse set of workloads ranging from Spark, Presto, TensorFlow.

Greyhole is a tool that uses Samba to create a storage pool of all your available hard drives (whatever their size, however they are connected), and allows you to create redundant copies of the files you store, in order to prevent data loss when part of your hardware fails.

Falcon LogScale is purpose-built to help any organization achieve the benefits of large-scale logging and analysis. Falcon LogScale has virtually no latency, even at ingest volumes of 1PB(Petabyte) per day.

Googerteller is a tool that makes an audible sound any time your computer sends a packet to a Google tracker or a Google service, which excludes Google Cloud users.

DeviceHive is a free, highly scalable open-source IoT platform for data collection, processing and analysis, visualization, and device management with the broad range of integration options.

Distributed Services Architecture (DSA) is an open source IoT platform that facilitates device inter-communication, logic and applications at every layer of the Internet of Things infrastructure. The objective is to unify the disparate devices, services and applications into a structured and adaptable real-time data model.

IoTivity is an open source software framework enabling seamless device-to-device connectivity to address the emerging needs of the Internet of Things.

Eclipse IoT Project provides open source technology that will be used to build IoT solutions for industry and consumers.

Social

Back to the Top

Mattermost is a secure, open source platform for communication, collaboration, and workflow orchestration across tools and teams.

Mastadon is a a decentralized social media platform that supports audio, video and picture posts, accessibility descriptions, polls, content warnings, animated avatars, custom emojis, thumbnail crop control, and more, to help you express yourself online.

Telegram is a cross-platform, cloud-based instant messaging service. It has an open API and source code free for everyone. Telegram also provides end-to-end encrypted video calling, VoIP, file sharing and several other features.

Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network.

Pleroma is a free and open communication for everyone. Pleroma is social networking software compatible with other Fediverse software such as Misskey, Pixelfed, Mastodon and many others.

Matrix is a tool that gives you simple HTTP APIs and SDKs (iOS, Android, Web) to create chatrooms, direct chats and chat bots, complete with end-to-end encryption, file transfer, synchronised conversation history, formatted messages, read receipts and more.

Element is a Matrix web client built using the Matrix React SDK.

Nostr(Notes and Other Stuff Transmitted by Relays) is a truly censorship-resistant alternative to Twitter that has a chance of working.

Diaspora is a privacy-aware, distributed, open source social network.

Hubzilla is a general purpose communication server integrated with a web publishing system and a decentralised permission system.

Expanse is a fully selfhosted multi-user web app for externally storing Reddit items (saved, created, upvoted, downvoted, hidden) to bypass Reddit's 1000-item listing limits.

Apollo is a beautiful Reddit app built for fast navigation with an incredibly powerful set of features.

Infinity is a Reddit client on Android written in Java. It does not have any ads and it features a clean UI and smooth browsing experience.

RedReader is an unofficial open source Reddit client for Android.

Communications

Back to the Top

Matrix is a tool that gives you simple HTTP APIs and SDKs (iOS, Android, Web) to create chatrooms, direct chats and chat bots, complete with end-to-end encryption, file transfer, synchronised conversation history, formatted messages, read receipts and more.

Element is a Matrix web client built using the Matrix React SDK.

Mattermost is a secure, open source platform for communication, collaboration, and workflow orchestration across tools and teams.

Mastadon is a a decentralized social media platform that supports audio, video and picture posts, accessibility descriptions, polls, content warnings, animated avatars, custom emojis, thumbnail crop control, and more, to help you express yourself online.

Telegram is a cross-platform, cloud-based instant messaging service. It has an open API and source code free for everyone. Telegram also provides end-to-end encrypted video calling, VoIP, file sharing and several other features.

Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network.

Pleroma is a free and open communication for everyone. Pleroma is social networking software compatible with other Fediverse software such as Misskey, Pixelfed, Mastodon and many others.

Nostr(Notes and Other Stuff Transmitted by Relays) is a truly censorship-resistant alternative to Twitter that has a chance of working.

Diaspora is a privacy-aware, distributed, open source social network.

Hubzilla is a general purpose communication server integrated with a web publishing system and a decentralised permission system.

Expanse is a fully selfhosted multi-user web app for externally storing Reddit items (saved, created, upvoted, downvoted, hidden) to bypass Reddit's 1000-item listing limits.

Mailroute is a great tool that provides the best email filtering & security( CMMC, NIST 800-171, DFARS, DISA, HIPPA). It protects your inbox, stop spam, viruses, ransomware, security threats & more with email filtering services. With an easy setup on Office 365, Google & more.

Docker Mailserver is a production-ready fullstack but simple mail server (SMTP, IMAP, LDAP, Antispam, Antivirus, etc.) running inside a container. Only configuration files, no SQL database.

iRedMail is a self-hosted email server.

iRedMail Easy is a web-based deployment platform, it offers an easy to use web interface to help you deploy iRedMail server, keep your server up to date, also get fast and professional technical support from iRedMail team.

Spider Email Archiver is an On-Premises Email Archiving Software.

MailCow is a self-hosted email server.

Nextcloud Talk is a on-premises, private audio/video conferencing and text chat through browser and mobile interfaces with integrated screen sharing and SIP integration.

Poste.io Email Server is self-hosted SMTP + IMAP + POP3 + Antispam + Antivirus Web administration + Web email. It is easy setup with a DNS guide for protect from spam.

Business Management

Back to the Top

Nextcloud is a suite of enterprise client-server software for creating and using file hosting services. It offers an on-premise Universal File Access and sync platform with powerful collaboration capabilities and desktop, mobile and web interfaces.

Odoo is a suite of open source business apps that cover all your company needs: CRM, eCommerce, accounting, inventory, point of sale, project management, etc.

Kanboard is project management software that focuses on the Kanban methodology.

Eden Workplace is a complete workplace management platform that lets you achieve more. Desk Booking Software to make desk reservations easier for your team, including assigning permanent and hybrid desks, providing wayfinding solutions for employees.

Matomo is an ethical alternative where you won't make privacy sacrifices or compromise your site. Matomo is the Google Analytics alternative that protects your data and your customer's privacy.

Plausible Analytics is a simple, lightweight (< 1 KB), open-source and privacy-friendly alternative to Google Analytics. It doesn’t use cookies and is fully compliant with GDPR, CCPA and PECR. You can self-host Plausible or have us run it for you in the Cloud.

Mailroute is a great tool that provides the best email filtering & security( CMMC, NIST 800-171, DFARS, DISA, HIPPA). It protects your inbox, stop spam, viruses, ransomware, security threats & more with email filtering services. With an easy setup on Office 365, Google & more.

InvoicePlane is a self-hosted open source application for managing your quotes, invoices, clients and payments.

Collaboration & Synchronization

Back to the Top

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers in real time.

Synology is a tool that allows you to easily access and manage files in your Synology Drive on the go. Apart from common file types, such as documents, images, videos and music, you can also open Synology Office document, spreadsheets and slides in the user-friendly viewer provided by Drive.

Nextcloud is a suite of client-server software for creating and using file hosting services. It offers an on-premise Universal File Access and sync platform with powerful collaboration capabilities and desktop, mobile and web interfaces.

Lsyncd (Live Syncing Mirror Daemon) is a tool used in Linux systems to keep directories synchronized. These directories can be found locally, within the same machine, or remotely, on different machines. For remote synchronization, this article focuses on using SSH to accomplish it.

FileRun is a self-hosted Google Drive alternative. It is a full featured web based file manager with an easy to use user interface.

FileBrowser provides a file managing interface within a specified directory and it can be used to upload, delete, preview, rename and edit your files. It allows the creation of multiple users and each user can have its own directory.

Rsync is a utility in the command line which enables users to transfer and synchronize files efficiently between a computer and an external hard drive in the entire connected network.

Warpinator is a free, open-source tool for sending and receiving files between computers that are on the same network.

FileZilla Client is a fast and reliable cross-platform FTP, FTPS and SFTP client with lots of useful features and an intuitive graphical user interface.

Dragit is an application for intuitive file sharing between devices. It's useful for when you want to send file from one computer to another with minimal effort. Dragit automatically detects devices in the local network with help of mDNS protocol and allows you to send file immediately.

WinFsp is a set of software components for Windows computers that allows the creation of user mode file systems. In this sense it is similar to FUSE (Filesystem in Userspace), which provides the same functionality on UNIX-like computers.

SSHFS-Win is a minimal port of SSHFS to Windows. Looking under the hood it uses Cygwin for the POSIX environment and WinFsp for the FUSE (Filesystem in Userspace) functionality.

RiftShare is a cross platform (Windows, MacOS, Linux) file sharing tool that supports fully encrypted transfers both on the local network and off network using a simple passphrase. RiftShare uses magic-wormhole under the hood and is compatible with other magic-wormhole clients. It is also fully open source and licensed under the GPLv3.

Usermode FTP Server is a tool that let's you start an FTP server as user and transfer files with any FTP client. Allowing you to access your files directly with many file browsers' builtin FTP support: Windows File Explorer, Thunar, Gnome Files, Dolphin and many more.

Listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary.

Encryption

Back to the Top

VeraCrypt is free open-source disk encryption software for Windows, Mac OS X and Linux. The file encryption, data encryption performed by VeraCrypt is real-time (on-the-fly), automatic, transparent, needs very little memory, and does not involve temporary unencrypted files.

AxCrypt is an inexpensive and effective encryption tool for Windows, macOS, iOS, and Android.

AESCrypt is an advanced file encryption utility that integrates with the Windows shell or runs from the Linux command prompt to provide a simple, yet powerful, tool for encrypting files using the Advanced Encryption Standard (AES). It is available for Windows, MacOS, and Linux.

Linux Unified Key Setup (LUKS) is a disk encryption specification created by Clemens Fruhwirth in 2004 and was originally intended for Linux. It uses device mapper crypt ( dm-crypt) as a kernel module to handle encryption on the block device level.

GNU Privacy Guard (GnuPG) is a complete and free implementation of the OpenPGP standard as defined by RFC4880 (also known as PGP ). It allows you to encrypt and sign your data and communications; it features a versatile key management system, along with access modules for all kinds of public key directories.

Pretty Good Privacy (PGP) is an encryption program that provides cryptographic privacy and authentication for data communication. It's used for signing, encrypting, and decrypting texts, e-mails, files, directories, and whole disk partitions and to increase the security of e-mail communications.

How Encryption Keys work

Symmetric is a data encryption method whereby the same private key is used to encode and decode information.
Asymmetric is a data encryption method that allows users to encrypt information using shared keys. For example, if you need to send a message across the internet, but you don't want anyone but the intended recipient to see what you've written.

Types of Encryption

Triple DES (Triple Data Encryption Algorithm) is a symmetric-key block cipher, which applies the DES cipher algorithm three times to each data block(contains 64 bits of data).
AES (Advanced Encryption Standard) is an algorithm that encrypts and decrypts data in blocks of 128 bits. It can do this using 128-bit, 192-bit, or 256-bit keys.
RSA (Rivest–Shamir–Adleman) is a type of public-key cryptography used for secure data transmission of e-mail and other digital transactions over the Internet.
Twofish is a symmetric key block cipher with a block size of 128 bits and key sizes up to 256 bits. It is an advanced version of Blowfish encryption.
Format Preserving Encryption (FPE) is a valid encryption algorithm to be used for compliance with NIST standards. It is mostly used in on-premise encryption and tokenization solutions.

Application Level Encryption

Hashes is a function that converts an input of letters and numbers into an encrypted output of a fixed length. For example, algorithms such as MD5 (Message Digest 5) or SHA (Secure Hash Algorithm).
Digital Certificates is a file that verifies the identity of a device or user and enables encrypted connections. A digital signature is a hashing approach that uses a numeric string to provide authenticity and validate identity. Digital certificates are typically issued by a certificate authority (CA), which is a trusted third-party entity that issues digital certificates for use by other parties.

Backups

Back to the Top

Proxmox Backup Server is an enterprise backup solution for backing up and restoring VMs, containers, and physical hosts. The open-source solution supports incremental backups, deduplication, Zstandard compression, and authenticated encryption.

Borgmatic is a simple, configuration-driven backup software for servers and workstations. It protects your files with client-side encryption. Backup your databases too. Monitor it all with integrated third-party services.

Vorta is a backup client for macOS and Linux desktops. It integrates the mighty Borg Backup with your favorite desktop environment to protect your data from disk failure, ransomware and theft.

UrBackup is an easy to setup Open Source client/server backup system, that through a combination of image and file backups accomplishes both data safety and a fast restoration time. File and image backups are made while the system is running without interrupting current processes. Available for Windows, macOS, and Linux.

Kopia is a user-friendly desktop app for Windows, macOS, and Linux which allows you to create snapshots, define policies, and restore files quickly with Fast and Encrypted Backups.

rsnapshot is a filesystem snapshot utility based on rsync. This makes it easy to make periodic snapshots of local machines, and remote machines over ssh.

Home Server

Back to the Top

Home Assistant is an open source home automation that puts local control and privacy first. Home Assistant is powered by a worldwide community of tinkerers and DIY enthusiasts that runs great on Raspberry Pi.

Homebridge is a software framework that allows you to integrate with smart home devices that do not natively support HomeKit. There are over 2,000 Homebridge plugins supporting thousands of different smart accessories.

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Shelly Cloud is a Smart home control tool that has been perfected and provides precise monitoring of your Shelly devices no matter where you are. Shelly devices are compatible with Alexa, Google Home, Android, and iOS.

Zigbee is the full-stack, secure, reliable, and market-proven solution used by a majority of large smart home ecosystem providers, such as Amazon's Echo Plus, Samsung SmartThings, Signify (Philips Hue), and more.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone's home smarter and safer.

Homey is an applciation to control, automate and monitor your entire smart home from your phone, tablet or desktop.

Caddy is the only web server to use HTTPS automatically and by default. Caddy obtains and renews TLS certificates for your sites automatically.

Bazarr is a companion application to Sonarr and Radarr. It can manage and download subtitles based on your requirements. You define your preferences by TV show or movie and Bazarr takes care of everything for you.

Sonarr is a PVR for Usenet and BitTorrent users. It can monitor multiple RSS feeds for new episodes of your favorite shows and will grab, sort and rename them.

Homarr is a customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr)

Rustdesk is an open source virtual/remote desktop infrastructure for everyone. Display and control your PC (Windows, macOS, and Linux) and Android devices.

TinyPilot is a tool that enables KVM over IP letting you control any computer remotely.

PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.

authentik is an open-source Identity Provider focused on flexibility and versatility. You can use authentik in an existing environment to add support for new protocols. authentik is also a great solution for implementing signup/recovery/etc in your application, so you don't have to deal with it.

ESPHome Remote IS a WI-FI smart home remote with display that runs on ESPHome. It uses Lilygo T-Display or M5Stack Fire.

Tdarr is a distributed transcode automation application using FFmpeg/HandBrake + Audio/Video library analytics + video health checking (Windows, macOS, Linux & Docker). A common use for Tdarr is to simply convert video files from h264 to h265 (hevc), saving 40%-50% in size.

AppFlowy is an open-source alternative to Notion where you're in charge of your data and customizations.

deemix is a barebone deezer downloader library built from the ashes of Deezloader Remix.

Neko is a self hosted virtual browser that runs in docker and uses WebRTC.

QNAP Switch System (QSS) is a configuration interface for QNAP's managed switch series. Enable management functions such as link aggregation, VLAN, and RSTP, to take care of your network topology with ease.

ASUSTOR is a subsidiary of ASUS and a leading provider of network attached storage (NAS). It specializes in the development and integration of storage, backup, multimedia, video surveillance and mobile applications for home and enterprise users.

Seafile is an open-source, cross-platform file-hosting software system. Seafile organize files into libraries stored on a central server. Each library can be synced into any desktop computer(Windows, Mac and Linux) and mobile devices through apps.

SnapRAID is a folder-based backup tool that behaves like a software or hardware RAID5/6 disk raid, but is not a disk raid itself. There is no realtime recovery, free space between disks cannot be combined and manual excution of backup is needed.

FreeNAS is an Open Source Storage Platform and supports sharing across Windows, Apple, and UNIX-like systems. This includes ZFS (high storage capacities and integrates file systems and volume management into a single piece of software). It supports UPS, CIFS/SMB, FTP, NFS, RSYNC, SSH, AFP, Unison, UPnP, Webserver, iSCSI protocols, local and MS AD authentication, and disk enctyption.

Gladys Assistant is a privacy-first, open-source home assistant and runs great on Raspberry Pi.

Audiobookshelf is a self-hosted audiobook and podcast server.

Mistborn is a secure platform for easily standing up and managing your own cloud services: including firewall, ad-blocking, and multi-factor WireGuard VPN access.

Media Server

Back to the Top

Overseerr is a free and open source software application for managing requests for your media library. It integrates with your existing services, such as Sonarr, Radarr, and Plex.

Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media. It is an alternative to the proprietary Emby and Plex, to provide media from a dedicated server to end-user devices via multiple apps.

Emby is a home media server built on top of other popular open source technologies such as Service Stack, jQuery, jQuery mobile, and Mono. It features a REST-based API with built-in documention to facilitate client development.

OpenMediaVault is a next generation network attached storage (NAS) solution based on Debian Linux. It contains services like SSH, (S)FTP, SMB/CIFS, AFS, UPnP media server, DAAP media server, RSync, BitTorrent client and many more.

Plex media server is a application that gives you the power to add, access and share all the entertainment that matters to you, on almost any device. With 50,000+ on demand titles and hundreds of channels of live TV, plus your own personal media collection, using one powerful app.

Infuse is a Video Player for iOS, Apple TV, and Mac. It plays every video file ever created to avoid wasting hours converting and transcoding files.

InfuseSync is a plugin for Emby and Jellyfin media servers that tracks all media changes to decrease sync times with Infuse clients.

Intro Skipper is a tool that analyzes the audio of television episodes to detect and skip over intro sequences in Jellyfin.

AirSonic is a free, web-based media streamer, providing ubiquitous access to your music.

TubeSync is a PVR (personal video recorder) for YouTube. Or, like Sonarr but for YouTube (with a built-in download client). It is designed to synchronize channels and playlists from YouTube to local directories and update your media server once media is downloaded.

PeerTube is an ActivityPub-federated video streaming platform using P2P directly in your web browser.

Ant Media Server is a streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency.

Castopod is an open-source hosting platform made for podcasters who want engage and interact with their audience.

RuneAudio is a free and open source software that turns embedded hardware into Hi-Fi music players.

Volumio (The Audiophile Music Player) is a very powerful and convenient music aggregator, now Volumio can also be used with great results in all sorts of different situations.

Snapcast is a multiroom client-server audio player, where all clients are time synchronized with the server to play perfectly synced audio. It's not a standalone player, but an extension that turns your existing audio player into a Sonos-like multiroom solution.

SonoBus is an easy to use application for streaming high-quality, low-latency peer-to-peer audio between devices over the internet or a local network.

Smart Home Automation

Back to the Top

Smart home is a process that allows homeowners to control appliances, thermostats, lights, and other smart devices remotely using a smartphone or tablet through an internet connection.

Most smart devices have their own Virtual Local Area Network (VLAN) with little to no internet access with broadcasts forwarding to LAN Subnet aka Subnetwork for discovery. Using software such as Home Assistant, Homebridge, ESPHome, etc. help simplify the process of controlling and automating all your smart devices.

Matter is an open standard for smart home technology that lets your device work with any Matter-certified ecosystem using a single protocol. Matter comes from the Connectivity Standards Alliance, an organization of hundreds of companies(Amazon, Apple, Google, Comcast, Zigbee Alliance, and Connectivity Standards Alliance (CSA) creating products for the smart home.

Proprietary Smart Devices

Amazon Alexa is a smart virtual assistant software to manage Alexa-enabled devices, control music playback, view shopping lists on the go, keep track of upcoming reminders, check on active timers and much more.
Google Assistant is a smart virtual assistant software on mobile and home automation devices.
Apple HomeKit is a software framework that enables your app to coordinate and control home automation accessories from multiple vendors to present a coherent, user-focused interface. Using HomeKit, your app can: Discover HomeKit-compatible automation accessories and add them to a persistent, cross-device home configuration database.
Samsung SmartThings is a sofwtare framework that you can connect, monitor and control multiple smart home devices quicker and easier. Connect your Samsung smart TVs, smart appliances, smart speakers and brands like Ring, Nest and Philips Hue all from one app.
Philips Hue is a smart lighting system. The smart lights, Hue Bridge, and smart controls will forever change the way you experience light.
Sonos is the wireless home sound system that fills as many rooms as you want with great-sounding music, movies, and TV.

------------------------------------------------------------------

Home Assistant is an open source home automation that puts local control and privacy first. Home Assistant is powered by a worldwide community of tinkerers and DIY enthusiasts that runs great on Raspberry Pi.

Add-ons are additional applications and services, that can be run alongside Home Assistant. The Home Assistant OS and Supervised installations types, provide the Supervisor, which is capable of running and managing these add-ons.

Home Assistant Official Add-ons

Addons created and maintained by the Home Assistant team.

DuckDNS - This updates your Duck DNS IP address and generate SSL using Let's Encrypt.
Almond - An Open, Privacy-Preserving Virtual Assistant.
HomeMatic - HomeMatic central based on OCCU.
Let's Encrypt - Get a free SSL certificate from Let's Encrypt; an open and automated certificate authority (CA).
MariaDB - An open source relational database (fork of MySQL).
File editor - Browser-based configuration file editor.
Mosquitto - Fast and reliable MQTT broker.
Terminal & SSH - Allows logging in remotely to using a web terminal or SSH client.
Samba - Access your configuration files using Windows network shares.
NGINX SSL proxy - Reverse proxy with SSL termination.
deCONZ - Control a ZigBee network using ConBee or RaspBee hardware by Dresden Elektronik.
TellStick - Run a TellStick and TellStick Duo service.
Ada - Ada is voice assistant powered by Almond which is open and privacy-preserving.

Home Assistant Third Party Add-ons

Add-ons created by the community.

Dasshio - Easily use your Amazon Dash Buttons.
InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
Grafana - Open platform for beautiful analytics and monitoring.
Tor - Protect your privacy and access your instance via Tor.
Spotify Connect - Spotify Connect client for playing music on your Home Assistant device.
SSH & Web Terminal - SSH and Web-based terminal with tons of pre-loaded useful tools.
UniFi Controller - The UniFi Controller allows you to manage your UniFi network using a web browser.
Node-RED - Flow-based programming for the Internet of Things.
Plex Media Server - Your recorded media beautifully organized and ready to stream.
IDE - Advanced web-based IDE, based on Cloud9 IDE.
zigbee2mqtt - Zigbee to MQTT bridge, get rid of your proprietary Zigbee bridges.
Matrix - A secure and decentralized communication platform.
AdGuard Home - A network-wide ad-and-tracker blocking DNS server with parental control.
Traccar - Traccar is modern GPS Tracking Platform.
Home Panel - A touch-compatible web frontend for controlling the home.
Hass.io Google Drive Backup - A complete and easy to configure solution for backing up your snapshots to Google Drive.
Grocy - A groceries & household management solution for your home.
EmonCMS - A powerful open-source web app for processing, logging, and visualizing energy, temperature, and other environmental data.
CrowdSec - A next-gen collaborative IPS/IDS to protect you from intrusion.
AppDaemon - Python Apps and HADashboard.
TasmoAdmin - Centrally manage all your Sonoff-Tasmota devices.
Aircast - AirPlay capabilities for your Chromecast players.
AirSonos - AirPlay capabilities for your Sonos players.
Log Viewer - Browser-based live log viewing utility.
Tautulli - Monitor and get statistics from your Plex server.
motionEye - Simple, elegant and feature-rich CCTV/NVR for your cameras.
JupyterLab - Create documents containing live code, equations, visualizations, and explanatory text.
Glances - A cross-platform system monitoring tool written in Python.

Home Assistant Custom Cards

The Home Assistant Dashboards allows people to build custom cards on top of it, which you can easily add to your instance.

Simple Thermostat - A simpler and more flexible thermostat card.
Card Modder - Style your Lovelace cards.
Bar Card - Customizable animated bar card.
forked-daapd Card - Control a forked daapd instance.
Dual Gauge Card - Shows two gauges in one.
Atomic Calendar Revive - Calendar card with advanced settings.
Simple Weather Card - A minimalistic weather card, inspired by Google Material Design.
Auto-Entities Card - Dynamically adds entities.
Canvas Gauge Card - Use awesome gauges from canvas-gauges.com.
Big Number Card - Display big numbers for sensors, including severity level as background.

Animated Weather Card - Nice looking card showing the weather, with subtle animations.
Thermostat Card - Thermostat control card that looks like a Nest Thermostat.

Raspberry Pi Status Card - Show status of your Raspberry Pis.
Mini Media Player - A minimalistic media player card.
Mini Graph Card - A minimalistic sensor graph card.
Button card - Button card for your entities.
Slider Entity Row - Add a slider to adjust, e.g., the brightness of lights in lovelace entity cards.
Power Wheel Card - An intuitive way to represent the power that your home is consuming or producing.
Home Card - A quick glance of the state of your home.
Banner Card - A fluffy linkable banner with interactive glances to spice up your home dashboards.
Spotify Card - List and select from current available devices and users top playlists on Spotify.
Battery Entity - Displaying battery levels for battery entities.
Multiple Entity Row - Show multiple entity states or attributes on entity rows.
Home Feed Card - Display a combination of persistent notifications, calendar events, and entities in the style of a feed.
Config Template Card - Allow using templates in Lovelace.
RGB Light Card - Colorful buttons to control your RGB Lights.
Restriction Card - A card to provide restrictions on Lovelace cards defined within.
Vacuum Map Card - This card provides a user-friendly way to fully control Xiaomi (Roborock/Viomi/Dreame/Roidmi) and Neato (+ possibly other) vacuums.
Vacuum Card - A card to card for controlling a vacuum cleaner robot.
Purifier Card - A card for controlling air purifiers.

Home Assistant Custom Integrations

Additional integrations for Home Assistant created by the community.

Lutron Caseta Pro - Integrates Lutron Caseta Smart Bridge PRO / RA2 Select.
SmartIR - Integrates devices using Broadlink IR.
Alexa Media Player - Allow control of Amazon Alexa devices.
Circadian Lighting - Circadian Lighting slowly synchronizes your color changing lights with the regular naturally occuring color temperature of the sky throughout the day.
Volkswagen Carnet - Integrates Volkswagen Carnet (requires valid Carnet subscription).
Untappd - Connects with your Untappd account.
Elasticsearch - Publishes events to Elasticsearch.
HASS Aarlo - Asynchronous Arlo integration. Similar to the Arlo web site; monitors events and states for all base stations, cameras and doorbells.
Xiaomi Cloud Map Extractor - Presents a live view of a map for Xiaomi (Roborock/Viomi/Roidmi/Dreame) vacuums without a need for rooting.
Xiaomi Hygrothermo - Sensor platform for Xiaomi Mijia BT Hygrothermo temperature and humidity sensor.
WebRTC Camera - View RTSP streams from IP Cameras in real-time through WebRTC or MSE with Pan/Zoom controls.
Sonoff LAN - Control Sonoff devices with eWeLink (original) firmware over LAN and/or Cloud.
Spotcast - Start Spotify playback on an idle Chromecast device as well as control Spotify connect devices.
The Watchman - Keep track of missing entities and services in your config files.

Home Assistant integrations. Credit: Home Assistant

Homebridge is a software framework that allows you to integrate with smart home devices that do not natively support HomeKit. There are over 2,000 Homebridge plugins supporting thousands of different smart accessories.

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

Homebridge Raspberry Pi Image is a free Raspbian based Raspberry Pi image with Homebridge and Homebridge Config UI X pre-installed.

Homebridge Config UI X is a web based management tool for Homebridge that allows you to manage all aspects of your Homebridge setup.

Homebridge webOS TV is a plugin for homebridge which allows you to control your LG webOS TV from your Home app! It should work with all TVs that support webOS2 and newer.

Homebridge Unifi Protect is a Homebridge plugin that provides HomeKit support to the UniFi Protect device ecosystem. UniFi Protect is Ubiquiti's next-generation video security platform, with rich camera, doorbell, and NVR controller hardware options for you to choose from, as well as an app which you can use to view, configure and manage your video camera and doorbells.

Homebridge Camera FFmpeg is a Homebridge Plugin Providing FFmpeg-based Camera Support.

Homebridge Mi Aqara is a homebridge plugin for XiaoMi Aqara plugin.

Homebridge Camera UI is a tool that allows you to expose cameras from camera.ui to HomeKit via Homebridge.

HOOBS is a tool that makes smart accessories(over 2000 options) compatible with your favorite ecosystem. Whether you prefer Apple Homekit, Google Home, or Amazon Alexa, you’re unlikely to find compatible accessories and services that all work together nicely under one roof.

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Shelly Cloud is a Smart home control tool that has been perfected and provides precise monitoring of your Shelly devices no matter where you are. Shelly devices are compatible with Alexa, Google Home, Android, and iOS.

Homey is an applciation to control, automate and monitor your entire smart home from your phone, tablet or desktop.

Ecobee is a home automation company in Canada that makes thermostats for residential and commercial use.

Lutron Caséta is a smart lighting control system that is a great solution for giving any client smart lighting control. It was purposely built to work in homes of all ages and it works with older wiring as well as new.

Insteon switches is a Hub for controlling and configuring your home's devices is quick, easy and fun. The setup takes a couple of minutes and a few moments per light switch, sensor, etc. It bridges your devices to 3rd party services like Amazon Alexa and Google Assistant, while being controlled from any smartphone or tablet.

Jeedom is an open source software; taht gives you complete access to the software that manages your home automation. It's compatible with various protocols such as, ZigBee, Z-Wave, EnOcean, KNX, LoRaWAN, BACnet, Modbus, etc..

Beestat is a tool that connects with your thermostat and provides you with useful charts and analytics so that you can make informed decisions and see how the changes you make lower your energy footprint.

MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth.

Zigbee is the full-stack, secure, reliable, and market-proven solution used by a majority of large smart home ecosystem providers, such as Amazon's Echo Plus, Samsung SmartThings, Signify (Philips Hue), and more.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone's home smarter and safer.

pfSense is a firewall/router computer software distribution based on FreeBSD.

Pi-hole is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software, intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations.

AdGuard Home is a DNS relay station with ad/tracker/other blocking, IP address redirections, and DNS-over-HTTPS.

OpenWRT is an open-source project for embedded operating systems based on Linux, primarily used on embedded devices to route network traffic.

ZoneMinder is a full-featured, open source, state-of-the-art video surveillance software system. Monitor your home, office, or wherever you want.

Plex media server is a application that gives you the power to add, access and share all the entertainment that matters to you, on almost any device. With 50,000+ on demand titles and hundreds of channels of live TV, plus your own personal media collection, using one powerful app.

Video Surveillance

Back to the Top

Frigate is an open source NVR built around real-time AI object detection. All processing is performed locally on your own hardware, and your camera feeds never leave your home.

Viseron is a Self-hosted, local only NVR and AI Computer Vision software.

zmninja is a high performance, cross platform ionic app for Home/Commerical Security Surveillance using ZoneMinder.

Moonfire NVR is a security camera network video recorder.

Shinobi Pro is a Next Generation in Open-Source Video Management Software with support for over 6000 IP and USB Cameras.

Text-To-Speech Synthesis (TTS)

Back to the Top

Coqui TTS is a deep learning toolkit for Text-to-Speech, battle-tested in research and production.

Mozilla TTS is a library for advanced Text-to-Speech generation. It's built on the latest research, was designed to achieve the best trade-off among ease-of-training, speed and quality.

NVIDIA NeMo is a conversational AI toolkit built for researchers working on automatic speech recognition (ASR), text-to-speech synthesis (TTS), large language models (LLMs), and natural language processing (NLP).

whisper.cpp is a high-performance inference of OpenAI's Whisper automatic speech recognition (ASR) model.

Video and Audio Processing

Back to the Top

Intel® Quick Sync Video is a tools that uses the dedicated media processing capabilities of Intel® Graphics Technology to decode and encode fast, enabling the processor to complete other tasks and improving system responsiveness.

Intel® QuickAssist Technology (Intel® QAT) is a scalable, flexible, and extendable way to accelerate data encryption/decryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database.

FFmpeg is a leading multimedia framework that can decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge ones on multiple platforms such as Windows, macOS, and Linux.

FFmpeg.guide is a simple GUI tool to create complex FFmpeg filtergraphs quickly and correctly, without having to mess with the cumbersome filter syntax.

HandBrake is a tool for transcoding video from almost any format with a selection of widely supported codecs. It is supported on Window, macOS, and Linux.

Tdarr is a cross-platform conditional based transcoding application for automating media library transcode/remux management in order to process your media files as required. It can set rules for the required codecs, containers, languages etc that your media should have which helps keeps things organized and can increase compatability with your devices. A common use for Tdarr is to simply convert video files from h264 to h265 (hevc), saving 40%-50% in size.

SRS is a simple, high efficiency and realtime video server, supports RTMP, WebRTC, HLS, HTTP-FLV, SRT and GB28181.

Video/Audio Standards

AAC(Advanced Audio Coding) is an audio coding standard for lossy digital audio compression. It's endorsed by ISO and IEC as MPEG-2 and MPEG-4 standards for video streams.

H.264(AVC) is a video compression standard based on block-oriented and motion-compensated integer-DCT coding that defines multiple profiles (tools) and levels (max bitrates and resolutions) with support up to 8K.

H.265(HEVC) is a video compression standard that is the successor to H.264(AVC). It offers a 25% to 50% better data compression at the same level of video quality, or improved video quality at the same bit-rate.

HTTP Live Streaming (HLS) is a communications protocol developed by Apple that sends live and on‐demand audio and video to iPhone, iPad, Mac, Apple Watch, Apple TV, and PC.

Dynamic Adaptive Streaming over HTTP (DASH) is an adaptive streaming protocol that allows for a video stream to switch between bit rates on the basis of network performance, in order to keep a video playing.

OpenMAX™ is a cross-platform API that provides comprehensive streaming media codec and application portability by enabling accelerated multimedia components to be developed, integrated and programmed across multiple operating systems and silicon platforms.

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing. Applications can take advantage of advances in codec and filter technology transparently.

Media Source Extensions (MSE) is a W3C specification that allows JavaScript to send byte streams to media codecs within Web browsers that support HTML5 video and audio. Also, this allows the implementation of client-side prefetching and buffering code for streaming media entirely in JavaScript.

WebRTC is an open-source project that adds real-time communication capabilities to your application that works on top of an open standard. It supports video, voice, and generic data to be sent between peers, allowing developers to build powerful voice- and video-communication solutions.

Podcasting

Back to the Top

Castopod is an open-source hosting platform made for podcasters who want engage and interact with their audience.

Audiobookshelf is a self-hosted audiobook and podcast server.

Podverse is creating an open source podcast app for iOS, Android, F-Droid, and Web.

Alby is a Bitcoin Lightning App for your Browser.

Alby wallet API is an implemented OAuth an open standard that apps use to provide client applications with secure delegated access. Thus Podverse users create an Alby wallet account or use their existing Alby account to grant specific access rights to their Alby wallet.

AntennaPod is a podcast player that is completely open. The app is open-source and you can subscribe to any RSS feed. AntennaPod is built by volunteers without commercial interest, so it respects your privacy while giving you full control.

Podgrab is a self-hosted podcast manager/downloader/archiver tool to download podcast episodes as soon as they become live with an integrated player.

Podify is a self-hosted service that allows you to download videos and audio from any source supported by youtube-dl, sort the downloads into feeds, and subscribe to these feeds using your favorite podcast app.

dir2cast is designed to turn a directory of MP3s into a podcast - automatically. Perfect for, say, radio shows - upload the MP3s to a folder, and use dir2cast.php as your PodCast URL.

Snipd is an AI-powered podcast player. Search for moments in the transcript, get summaries, share clips to social media, and export to your second brain & note taking apps. Discover, save, and share highlights from podcasts.

Wave Share is a serverless, peer-to-peer, local file sharing through sound.

KBD Audio is a collection of command-line and GUI tools for capturing and analyzing audio data.

AudioBooks

Back to the Top

Audioserve is a simple personal server to serve audio files from directories. Intended primarily for audio books, but anything with decent directories structure will do. Focus here is on simplicity and minimalist design.

Audiobookshelf is a self-hosted audiobook and podcast server.

Jellyfin Bookshelf Plugin

Maps

Back to the Top

MapTiler Server is a self-hosted Aerial and satellite imagery maps of the entire world from your own server or laptop.

KelperJs is a open source full-stack geosocial network platform.

OpenStreetMap(OSM) is a map of the world, created by people like you and free to use under an open license. Hosting is supported by UCL, Fastly, Bytemark Hosting, and other partners.

uMap is a tool that lets you create maps with OpenStreetMap layers in a minute and embed them in your site.

Bookmarks

Back to the Top

Linkding is a simple bookmark service that you can host yourself. It's designed be to be minimal, fast, and easy to set up using Docker.

LinkAce is a self-hosted bookmark archive to collect links of your favorite websites.

Eagle is a tool to collect, search and organize your design files in a logical way and all in one place.

Shlink is a self-hosted URL shortener that keeps control over all your shortened URLs, by serving them under your own domains, using this simple yet powerful tool.

Pinry is a tiling image board system for people who want to save, tag, and share images, videos and webpages in an easy to skim through format.

Shaark is a self-hosted platform to keep and share your content: web links, posts, passwords and pictures.

Photos

Back to the Top

PhotoPrism® is an AI-powered app for browsing, organizing & sharing your photo collection. It makes use of the latest technologies to tag and find pictures automatically without getting in your way. You can run it at home, on a private server, or in the cloud.

Immich is a high performance self-hosted photo and video backup solution directly from your mobile phone.

Piwigo is a full featured, self-hosted, and open source photo gallery application for the web. It comes with more than 200 templates, plugins, and configurations that let you personalize how your photos are presented. It allows users to upload photos from digiKam, Shotwell, Lightroom or mobile applications. Lastly, users can create galleries and give viewing permissions to their clients. They can download individual photos or whole albums, post comments, give ratings, mark photos as favorites, and perform searches.

Czkawka is a Multi-functional app to find duplicates, empty folders, similar images, etc.

Phockup is a Media sorting tool to organize photos and videos from your camera in folders by year, month and day.

PiGallery 2 is a fast directory-first photo gallery website, with rich UI, optimized for running on low resource servers (especially on Raspberry Pi).

Photoview is a simple self-hosted and user-friendly photo gallery that's made for photographers and aims to provide an easy and fast way to navigate directories, with thousands of high-resolution photos.

digiKam is a free and open-source Professional Photo Management tool.

ShareX is a free and open source program that lets you capture or record any area of your screen and share it with a single press of a key. It also allows uploading images, text or other types of files to many supported destinations you can choose from.

PhotoSync is a service to wirelessly transfer, backup & share photos/videos to your computer, NAS, other phones and popular cloud/photo services. It's available for Windows, MacOS, Linux, Android, and iOS.

Lychee is a great looking and easy-to-use photo-management-system you can run on your server, to manage and share photos.

Photoview is a simple and user-friendly photo gallery that's made for photographers and aims to provide an easy and fast way to navigate directories, with thousands of high-resolution photos.

Gimme-iPhotos is a tool uses pyicloud to synchronize photos and videos from iCloud to your local machine.

PyiCloud is a module which allows pythonistas to interact with iCloud webservices. It's powered by the fantastic requests HTTP library.

Pixelfed is a fresh take on photo sharing. It decentralized ActivityPub protocol so you can comment, follow, and interact with remote Pixelfed, Mastodon and Pleroma posts and profiles from your Pixelfed account as if you were both on the same website.

Chevereto is an image hosting software that allows you to create a beautiful and full-featured image hosting website on your own server.

Got Your Back (GYB) is a command line tool for backing up your Gmail messages to your computer using Gmail's API over HTTPS.

Note-Taking

Back to the Top

Joplin is an open source note-taking app that you can securely access from any device.

HedgeDoc is an open-source, web-based, self-hosted, collaborative markdown editor.

Turtl is a secure, collaborative notebook for bookmarks or passwords, files or shopping lists.

TiddlyWiki is a single-file mode wiki application for todo lists, effective project management tool and of course writing drafts and notes. It has extensions for all the major browsers.

Milanote is a tool for organizing creative projects into beautiful visual boards.

Laverna is a note taking application with Markdown editor and encryption support. Consider it like open source alternative to Evernote.

Notesnook is a fully open source & end-to-end encrypted note taking alternative to Evernote.

Zettlr is an open-source Markdown editor for the 21st century.

Carnet is a complete open source note taking app. It has extensions for all the major browsers.

Frog is a tool that quickly extract text from almost any source: youtube, screencasts, PDFs, webpages, photos, etc. Grab the image and get the text.

Gaming

Back to the Top

Cartridge is a self-hosted game library made with Laravel + Vue.js.

Cartridge Features

Scan for ROM files and match with IGDB game information
Serve ROM download links alongside game details
Manage access to library with user creation and permissions (WIP)
Allow users to request games (Planned)
Play select ROMs in-browser using JS emulators (Planned)
Track played and favorite games (even ones that aren't available for download) (Planned)

Moonlight Game Streaming is a program that let you stream from your PC games over the Internet with no configuration required. Stream from almost any device, whether you're in another room or miles away from your gaming rig. Sunshine is a Game stream host for Moonlight that is a self-hosted, low latency, cloud gaming solution with support for AMD, Intel, and NVIDIA GPUs. It is an open source implementation of NVIDIA's GameStream.

Chiaki is a Free and Open Source Software Client for PlayStation 4 and PlayStation 5 Remote Play for Linux, FreeBSD, OpenBSD, Android, macOS, Windows, Nintendo Switch and potentially even more platforms.

Game Emulators

EmuDeck is a tool that takes care of everything for your retrogaming needs from RetroArch Configuration, Bezels, Gamepad Configuration for GameCube, Wii, Citra, SNES, etc. EmuDeck will even install EmulationStation Desktop Edition and carry over all their custom configurations and no need to configure ROM paths or anything.

EmulationStation Desktop Edition (ES-DE) is a frontend application for browsing and launching games from your multi-platform game collection. It's available for Unix/Linux, macOS(M1 & Intel) and Windows.

RetroPie is a frontend for emulators that allows you to turn your Raspberry Pi, ODroid C1/C2, or PC into a retro-gaming machine. It builds upon Raspbian, EmulationStation, RetroArch and many other projects to enable you to play your favourite Arcade, home-console, and classic PC games with the minimum set-up.

RetroArch is a frontend for emulators, game engines and media players. It enables you to run classic games on a wide range of computers and consoles through its slick graphical interface. Settings are also unified so configuration is done once and for all. RetroArch Flatpak

Pterodactyl is a free, open-source game server management panel built with PHP, React, and Go. Designed with security in mind, Pterodactyl runs all game servers in isolated Docker containers while exposing a beautiful and intuitive UI to end users.

LinuxGSM (Linux Game Server Managers) is a command-line tool for quick, simple deployment and management of Linux dedicated game servers.

Cartridge is a self-hosted game library made with Laravel + Vue.js.

Cartridge Features

Scan for ROM files and match with IGDB game information
Serve ROM download links alongside game details
Manage access to library with user creation and permissions (WIP)
Allow users to request games (Planned)
Play select ROMs in-browser using JS emulators (Planned)
Track played and favorite games (even ones that aren't available for download) (Planned)

Cartridge UI

Dolphin is an emulator for two recent Nintendo video game consoles: the GameCube and the Wii. It allows PC gamers to enjoy games for these two consoles in full HD (1080p) with several enhancements: compatibility with all PC controllers, turbo speed, networked multiplayer, and even more.

Citra is an open-source emulator for the Nintendo 3DS capable of playing many of your favorite games.

yuzu is an experimental open-source emulator for the Nintendo Switch from the creators of Citra.[

m64p is a Nintendo 64 Emulator. It uses mupen64plus-gui, a brand new mupen64plus frontend written in Qt5. It supports all of the things you’d expect from a frontend (savestate management, pausing, screenshots).

DeSmuME is a Nintendo DS emulator.

Snes9x is a portable, freeware Super Nintendo Entertainment System (SNES) emulator.

bsnes is a Super Nintendo (SNES) emulator focused on performance, features, and ease of use.

mGBA is a new emulator for running Game Boy Advance games. It aims to be faster and more accurate than many existing Game Boy Advance emulators, as well as adding features that other emulators lack.

DOSBox is an open-source DOS emulator which primarily focuses on running DOS Games.

DOSBox Staging is a full x86 CPU emulator (independent of host architecture), capable of running DOS programs that require real or protected mode.

Flycast is a multi-platform Sega Dreamcast, Naomi and Atomiswave emulator derived from reicast.

PCSX2 is a PlayStation 2 'emulator', a free program that tries to replicate the PlayStation 2 console to enable you to play PS2 games on your PC.

RPCS3 is an experimental open-source Sony PlayStation 3 emulator and debugger written in C++ for Windows and Linux. RPCS3 started development in May of 2011 by its founders DH and Hykem. The emulator is currently capable of running over 1800 commercial titles powered by Vulkan and OpenGL.

MAME is a Arcade Machine Emulator.

xemu is an original Xbox emulator.

Xenia is an Xbox 360 Emulator.

Also checkout these subreddits for more great Game Emulators recommendations

Foundations/Projects

Back to the Top

Matter is an open standard for smart home technology that lets your device work with any Matter-certified ecosystem using a single protocol. Matter comes from the Connectivity Standards Alliance, an organization of hundreds of companies(Amazon, Apple, Google, Comcast, Zigbee Alliance, and Connectivity Standards Alliance (CSA) creating products for the smart home.

Open Source Hardware Association (OSHWA) is a non-profit organization that advocates for open-source hardware. It aims to act as a hub of open source hardware activity of all types while actively cooperating with other initiatives such as the TAPR Open Hardware License, open-source development groups at CERN, and the Open Source Initiative (OSI).

The Open Connectivity Foundation is dedicated to ensuring secure interoperability for consumers, businesses and industries by delivering a standard communications platform, a bridging specification, an open source implementation and a certification program allowing devices to communicate regardless of form factor, operating system, service provider, transport technology or ecosystem.

Raspberry Pi Foundation is a UK-based charity with the mission to enable young people to realise their full potential through the power of computing and digital technologies.

OpenSSF(Open Source Security Foundation) is a cross-industry forum for a collaborative effort to improve open source software security.

OpenJS Foundation is the premier home for critical open source JavaScript projects, including Appium, Dojo, jQuery, Node.js, and webpack, and 27 more.

EdgeX Foundry is a vendor-neutral project under the Linux Foundation. The initiative is aligned around a common goal: the simplification and standardization of the foundation for edge computing architectures in the Industrial IoT market, while still allowing the ecosystem to add significant value.

Eclipse Foundation provides our global community of individuals and organizations with a mature, scalable and commercially-friendly environment for open source software collaboration and innovation.

System Hardware

Back to the Top

CPUs

Intel Processors(x86)

Back to the Top

I recommend using Intel CPUs no older than the second generation of the Intel Core processors (Core i7, i5, i3) AKA Sandy Bridge(Jan. 2011) for those that want to utilize Intel® Quick Sync Video. Though, if you're concerned about power efficiency(~5W idle) I would recommend 7th Generation or newer.

Also, I recommend using Intel® QuickAssist Technology (Intel® QAT) a scalable, flexible, and extendable way to accelerate data encryption/decryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database. Available in 3rd Gen Intel® Xeon® Scalable Processors and Intel Atom® Processor C Series/P Series.

AMD Processors(x86)

Back to the Top

Devices

Back to the Top

Note: Will be adding more device soon!

Operating Systems

Back to the Top

Creating a bootable media device(USB/MicroSD card)

Rufus is a utility that helps format and create bootable USB flash drives.

Rufus

OR

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Etcher UI

A List of Operating Systems that are great for either settig up a personal Home Server or a Enterprise Server for your Organization/Company.

Home Assistant OS is a container-based system for managing your Home Assistant Core installation and related applications. The system is controlled via Home Assistant which communicates with the Supervisor. The Supervisor provides an API to manage the installation. This includes changing network settings or installing and updating software.

Home Assistant OS

Umbrel is an OS for running a personal server in your home. It can Self-host open source apps like Nextcloud, Bitcoin node, and more.

Umbrel

CasaOS is a simple, easy-to-use, elegant open-source Home Cloud system.

CasaOS

Ubuntu is a modern open source operating system on Linux for the enterprise Server, Desktop, Cloud, and IoT developed by Canonical.

Ubuntu Server
Ubuntu for ARM
Ubuntu for Raspberry Pi
Ubuntu Flavours is for those that prefer an alternative desktop environment such as KDE Plasma Desktop, MATE, Xfce, LXQt, Budgie, and UKUI you can download a Flavour for your preferred desktop environment and use that to install Ubuntu, pre-configured for the desktop environment of your choice.

Ubuntu

Debian is an operating system and a distribution of Free Software. It is maintained and updated through the work of many users who volunteer their time and effort.

Debian 11

Linux Mint is a modern, elegant, and comfortable open source operating system(based on Debian and Ubuntu), which is both powerful and easy to use for both new and advanced users. The flagsip version of Linux Mint uses the Cinnamon desktop environment similiar to Windows 7.

Linux Mint

Linux Mint Debian Edition (LMDE) uses Debian Bullseye as the base for a very stable and rock solid user experience with the Cinnamon desktop.

Linux Mint Debian Edition (LMDE)

Pop!_OS created by System76.

Pop!_OS

Fedora Linux is a polished, easy to use operating system for laptop & desktop computers, with a complete set of tools for developers and makers of all kinds. The OS serves as the foundation for which you can scale existing apps and roll out emerging technologies across bare-metal, virtual, container, and all types of cloud environments.

Fedora Spins is for those that prefer an alternative desktop environment such as KDE Plasma Desktop, MATE; or Xfce, you can download a spin for your preferred desktop environment and use that to install Fedora, pre-configured for the desktop environment of your choice.
Fedora Server is a powerful, flexible operating system that includes the best and latest datacenter technologies. It puts you in control of all your infrastructure and services.
Fedora ARM is an initiative to bring versions of Fedora tailored for running on ARM-based systems.
Fedora Silverblue is a variant of the Fedora Workstation that uses rpm-ostree to provide an immutable OS image with reliable updates and easy rollbacks.
Fedora Kinoite is an immutable desktop operating system. It aims to be extremely stable and reliable. It also aims to be an excellent platform for developers and for those using container-focused workflows. Kinoite is a variant of the Fedora KDE Spin.
Fedora CoreOS is an automatically-updating, minimal operating system for running containerized workloads securely and at scale.

Fedora Linux

CentOS Stream is a continuously delivered distro(uses the Fedora OS base) that tracks just ahead of Red Hat Enterprise Linux (RHEL) development, positioned as a midstream between Fedora Linux and RHEL.

CentOS Stream

Red Hat® Enterprise Linux® (RHEL) the world's leading enterprise Linux platform. The OS serves as the foundation for which you can scale existing apps and roll out emerging technologies across bare-metal, virtual, container, and all types of cloud environments.

Red Hat Enterprise Linux Desktop

AlmaLinux is an open source enterprise-ready Linux distribution forked from Red Hat Enterprise Linux(RHEL). It's a very stable/solid operating system especially for production envrionments such as servers, though, you can also install a GUI on AlmaLinux and use it as a desktop OS. It was founded by the team behind the CloudLinux OS.

AlmaLinux Desktop Setup and Install

Rocky Linux is a community enterprise operating system designed to be 100% bug-for-bug compatible and forked from Red Hat Enterprise Linux(RHEL) now that its downstream partner(Red Hat) has shifted direction.

Rocky Linux Desktop

SUSE is a leading Linux OS most adaptable Linux operating system and the only open Kubernetes management platform thanks to their acquistion of Rancher. They also developer of SUSE Linux Enterprise and the primary sponsor of the community-supported openSUSE Project, which develops the openSUSE Linux distribution.

openSUSE Leap is a brand new way of building openSUSE and is new type of a hybrid Linux distribution. Leap uses source from SUSE Linux Enterprise (SLE), which gives Leap a level of stability unmatched by other Linux distributions, and combines that with community developments to give users, developers and sysadmins the best stable Linux experience available.
openSUSE Tumbleweed is a pure rolling release version of openSUSE containing the latest "stable" versions of all software instead of relying on rigid periodic release cycles. The project does this for users that want the newest stable software.
openSUSE Kubic is a multi-purpose Standalone & Kubernetes Container Operating System based on openSUSE MicroOS. Kubic uses kubeadm to provide an easy way of configuring a Kubernetes cluster across multiple machines, while our MicroOS base keeps your operating system updated automatically, with fully atomic rollbacks if required.
openSUSE MicroOS is a M icro Service OS providing Transactional (Atomic) Updates upon a read-only btrfs root filesystem. It's designed to host container workloads with automated administration & patching.

SUSE Linux Enterprise 12

openSUSE

NixOS is a Linux distribution built on top of the Nix package manager. It has tools dedicated to DevOps and deployment tasks. NixOS Guide

Nix Tour is an interactive tour that uses the actual package manager to learn you the language by example, in the browser.
Nix is a package manager and build system that parses reproducible build instructions specified in the Nix Expression Language, is a pure functional language with lazy evaluation. Nix expressions are pure functions taking dependencies as arguments and producing derivation specifying a reproducible build environment for the package. Nix stores the results of the build in unique addresses specified by a hash of the complete dependency tree, creating an immutable package store that allows for atomic upgrades, rollbacks and concurrent installation of different versions of a package, essentially eliminating dependency hell.
Nix Expression Language is a pure, lazy, functional language. Purity means that operations in the language don't have side-effects (for instance, there is no variable assignment). The language is not a full-featured, general purpose language. Its main job is to describe packages, compositions of packages, and the variability within packages.
Nixpkgs is the largest repository of Nix packages(over 80,000 packages) and NixOS modules. The repository is hosted on GitHub and maintained by the community, with official backing from the NixOS Foundation. Additionally, checkout Language-specific package helpers and Alternative Package Sets.
NixOS Packages Search is a tool for searching through NixOS packages.

NixOS Packages Search

NixOS Desktop with the new Calamares Installer

NixOS with the Plasma Desktop

BSD

Back to the Top

FreeBSD is an Unix-like operating system used to power modern servers, desktops, and embedded platforms. A large community has continually developed it for more than thirty years. Its advanced networking, security, and storage features have made FreeBSD the platform of choice for many of the busiest web sites and most pervasive embedded networking and storage devices.

FreeBSD Software ports. Source: FreeBSD Software

OpenBSD is a security-focused, free and open-source, Unix-like operating system based on the Berkeley Software Distribution. It comes with a secure minimal firewall, webserver, mailserver, and an optional graphical desktop.

NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices.

NetBSD Documentation

DragonFly BSD is a free and open-source Unix-like operating system forked from FreeBSD 4.8.

DragonFly Documentation

The BSD Desktop for the average user

GhostBSD is a simple desktop-oriented operating system based on FreeBSD with MATE, OpenRC and OS packages for simplicity. GhostBSD has a selection of commonly used software preinstalled and required to start using it to its full potential.

GhostBSD Desktop. Source:GhostBSD

Storage

Back to the Top

Useful Tools for Storage Management

Scrutiny is a WebUI for smartd Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds.

DUA (Disk Usage Analyzer) is a tool to conveniently learn about the usage of disk space of a given directory. It's parallel by default and will max out your SSD, providing relevant information as fast as possible. Optionally delete superfluous data, and do so more quickly than rm.

Dirstat-rs is a fast, cross-platform disk usage CLI, similar to Windirstat.

Dutree is a tool to analyze file system usage written in Rust.

Shufflecake is a tool for Linux that allows to create multiple hidden volumes on a storage device in such a way that it is very difficult, even under forensic inspection, to prove the existence of such volumes.

File systems

Back to the Top

FSArchiver is a system tool that allows you to save the contents of a file system to a compressed archive file. The file system can be restored on a partition which has a different size and it can be restored on a different file system.

GlusterFS is a free and open source scalable network filesystem. Gluster is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks.

Ceph is a software-defined storage solution designed to address the object, block, and file storage needs of data centers adopting open source as the new norm for high-growth block storage, object stores and data lakes. Ceph provides enterprise scalable storage while keeping CAPEX and OPEX costs in line with underlying bulk commodity disk prices.

Hadoop Distributed File System (HDFS) is a distributed file system that handles large data sets running on commodity hardware. It is used to scale a single Apache Hadoop cluster to hundreds (and even thousands) of nodes. HDFS is one of the major components of Apache Hadoop, the others being MapReduce and YARN.

ZFS is an enterprise-ready open source file system and volume manager with unprecedented flexibility and an uncompromising commitment to data integrity.

ZFSBootMenu is a Linux bootloader that attempts to provide an experience similar to the FreeBSD bootloader. It takes advantage of ZFS features, it allows a user to have multiple “boot environments” (with different distros, for example), manipulate snapshots before booting, and even bootstrap a system installation via zfs recv.

OpenZFS is an open-source storage platform. It includes the functionality of both traditional file systems and volume manager. It has many advanced features including:

Protection against data corruption.
Integrity checking for both data and metadata.
Continuous integrity verification and automatic "self-healing" repair.

Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration. Its main features and benefits are:

Snapshots which do not make the full copy of files
RAID - support for software-based RAID 0, RAID 1, RAID 10
Self-healing - checksums for data and metadata, automatic detection of silent data corruptions

Composefs is a native Linux file system designed to help sharing filesystem contents, as well as ensuring said content is not modified. The initial target usecase are container images and ostree commits.

MergerFS is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfs, unionfs, and aufs.

MergerFS Features

Configurable behaviors / file placement
Ability to add or remove filesystems at will
Resistance to individual filesystem failure
Support for extended attributes (xattrs)
Support for file attributes (chattr)
Runtime configurable (via xattrs)
Works with heterogeneous filesystem types
Moving of file when filesystem runs out of space while writing
Ignore read-only filesystems when creating files
Turn read-only files into symlinks to underlying file
Hard link copy-on-write / CoW
Support for POSIX ACLs

Proxmox Cluster File System (PMXCFS) is a File System used to transparently distribute the cluster configuration to all cluster nodes.

UnionFS is a filesystem service for Linux, FreeBSD and NetBSD which implements a union mount for other file systems. It allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system.

OverlayFS is a modern union filesystem that is similar to AUFS, but faster and with a simpler implementation. It's typically used on systems running on embed devices, like OpenWRT, where is useful to preserve a basic set of configurations and at the same time allowing the user to perform modifications.

Bcachefs is an advanced new filesystem for Linux, with an emphasis on reliability and robustness and the complete set of features one would expect from a modern filesystem. Scalability has been tested to 50+ TB, will eventually scale far higher.

Squashfs is a compressed read-only filesystem for Linux. It uses zlib, lz4, lzo, or xz compression to compress files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimize data overhead.

Apple File System (APFS) is the default file system for Mac computers using macOS 10.13 or later, features strong encryption, space sharing, snapshots, fast directory sizing, and improved file system fundamentals.

NTFS(New Technology File System) is the primary file system for recent versions of Windows and Windows Server—provides a full set of features including security descriptors, encryption, disk quotas, and rich metadata, and can be used with Cluster Shared Volumes (CSV) to provide continuously available volumes that can be accessed simultaneously from multiple nodes of a failover cluster.

exFAT(Extended File Allocation Table ) is the file system that was the successor to FAT32 in the FAT family of file systems. It was optimized for flash memory such as USB flash drives and SD cards.

Books

Back to the Top

Self-hosted Cookbook is a cookbook, for docker-compose based recipes, for self-hosted applications and services.
Geek's Cookbook is a collection of guides for establishing your own highly-available "private cloud" and using it to run self-hosted services such as GitLab, Plex, NextCloud, etc.
Database Books(PDFs)

Podcasts

Back to the Top

Self-Hosted Podcast is a chat show between Chris and Alex two long-time "self-hosters" who share their lessons and take you along for the journey as they learn new ones.
Self-Hosted SRE(Site Reality Engineer) Podcast is a feed to say thank you to our Self-Hosted Site Reality Engineers!
Home Assistant Podcast is a biweekly podcast with the latest news and interesting guests.

WireGuard

Back to the Top

What is WireGuard?

Back to the Top

WireGuard® is a straight-forward, fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec while avoiding the massive headache. WireGuard is designed as a general-purpose VPN for running on embedded interfaces and super computers alike, fit for many circumstances. Initially released for the Linux kernel, it is now cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable.

What is Tailscale?

Back to the Top

Tailscale is a WireGuard-based app that makes secure, private networks easy for teams of any scale. It works like an overlay network between the computers of your networks using all kinds of NAT traversal sorcery.

How NAT Traversal works on a Home router. Credit: Tailscale.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

What is Netmaker?

Back to the Top

Netmaker is a tool that enables you to create relays, gateways, full VPN meshes, and even zero trust networks. It's fully configurable to let you maximize the power of Wireguard.

NetMaker Architecture. Credit: Netmaker.

WireGuard Tools

Back to the Top

Wiretrustee is a WireGuard®-based mesh network that connects your devices into a single private network.

Wireguard Manager is a tool that enables you to build your own vpn under a minute.

Tailscale is a WireGuard-based app that makes secure, private networks easy for teams of any scale. It works like an overlay network between the computers of your networks using all kinds of NAT traversal sorcery.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

Firezone is a self-hosted WireGuard®-based VPN server and Linux firewall.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Mistborn is a secure platform for easily standing up and managing your own cloud services: including firewall, ad-blocking, and multi-factor WireGuard VPN access.

Mistborn CLI is a Command-line interface for Mistborn.

BoringTun is an implementation of the WireGuard® protocol designed for portability and speed. It's successfully deployed on millions of iOS and Android consumer devices as well as thousands of Cloudflare Linux servers.

PiVPN is the simplest VPN installer, designed for Raspberry Pi.

Algo VPN is a set of Ansible scripts that simplify the setup of a personal WireGuard and IPsec VPN. It uses the most secure defaults available and works with common cloud providers.

Pro Custodibus is a tool for managing WireGuard with a variety of business VPN (Virtual Private Network) use cases, such as site-to-site connectivity, secure remote access from anywhere, secure access to the cloud (Amazon Web Services, Google Cloud Platform, Microsoft Azure, etc), and more.

Drago is a flexible configuration manager for WireGuard designed to make it simple to configure secure network overlays spanning heterogeneous nodes distributed across different clouds and physical locations. Drago is in active development, and we welcome contributions from the open-source community.

Netmaker is a tool that helps connect any computers together over a secure, fast, private network, and manage multiple networks from a central server.

Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes. Kilo connects nodes in a cluster by providing an encrypted layer 3 network that can span across data centers and public clouds. The Pod network created by Kilo is always fully connected, even when the nodes are in different networks or behind NAT. By allowing pools of nodes in different locations to communicate securely, Kilo enables the operation of multi-cloud clusters. Kilo's design allows clients to VPN to a cluster in order to securely access services running on the cluster.

Subspace is a simple WireGuard VPN server GUI.

WG UI is a basic, self-contained management service for WireGuard with a self-serve web UI.

WireHole is a combination of WireGuard, PiHole, and Unbound in a docker-compose project with the intent of enabling users to quickly and easily create and deploy a personally managed full or split-tunnel WireGuard VPN with ad blocking capabilities (via Pihole), and DNS caching with additional privacy options (via Unbound).

Gluetun is a lightwieght VPN client in a thin Docker container for multiple VPN providers, written in Go, and uses OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.

Ethr is a cross platform network performance measurement tool written in golang. The goal of this project is to provide a native tool for comprehensive network performance measurements of bandwidth, connections/s, packets/s, latency, loss & jitter, across multiple protocols such as TCP, UDP, HTTP, HTTPS, and across multiple platforms such as Windows, Linux and other Unix systems.

Setting up WireGuard with PiVPN

Back to the Top

Installing PiVPN:

sudo apt install curl -y

curl -L https://install.pivpn.io | bash

Setting up WireGuard on Unraid

Back to the Top

Select Apps, then search for WireGuard and install Wireguard-Easy.

VPN manager

Almost all of the settings can stay as default, however, there are a few that we will modify.

Set the WG_HOST variable to be the IP address of your Unraid server.
If you’d like to modify the WireGuard port (51820), you can do that here.
Change the default Web GUI password.

Setting up WireGuard on pfSense

Back to the Top

When looking at how to set up WireGuard on pfSense, the first thing that we need to do is install the package. Follow the instructions below to install the WireGuard package on pfSense.

Open the Package Manager and search for WireGuard, then Install the latest version of the package.

After the package has installed, select VPN then WireGuard and under the Tunnels section, select Add Tunnel.
In the Tunnel Configuration, set the Description as WireGuard, the Listen Port as 51820, then Generate private and public keys.
Copy the Public Key. We will need this for our client configuration.
Create the tunnel, then select Settings, and ensure that Enable WireGuard is selected. Then Save and Apply.

Setting up WireGuard on OpenWRT

Back to the Top

Quick Links:

In your router’s webUI, navigate to System - Software, click Update lists:

In the Filter field, type WireGuard, locate and install the wireguard, wireguard-tools, kmod-wireguard, and luci-app-wireguard packages. Note: The wireguard package is included in version 22.02.

Generate WireGuard keypair

SSH into your router as ‘root’ (OpenWrt Wiki):

ssh [email protected]

Generate WireGuard keys:

wg genkey | tee privatekey | wg pubkey > publickey

chmod 600 privatekey

Note your Private & Public keys, you will need them later:

cat privatekey

cat publickey

Creating an Interface

Navigate to Network - Interface,

Click the Add new interface... button and enter the following configuration:

Name - give it any name
Protocol - WireGuard VPN

Create interface

In the General Settings tab:

Bring up on boot - Checked
Private Key - copy and paste the generated previously Private key
IP Address - enter the WireGuard IP Address obtained in the Client Area ending with /32, e.g. 172.27.124.169/32

Add a Firewall zone

Navigate to Network - Firewall

Click the Add button and enter the following configuration:

Name - Give it any name
Input - Reject
Output - Accept
Forward - Reject
Masquerading - Checked
MSS clamping - Checked
Covered networks - select the previously created VPN tunnel interface
Allow forward to destination zones - Unspecified
Allow forward from source zones - lan

DNS

Navigate to Network - Interfaces

Click on the Edit button next to the WAN interface

In the Advanced Settings tab, uncheck the Use DNS servers advertised by peer and specify one of the following DNS servers in the Use custom DNS servers field:

172.16.0.1 = regular DNS with no blocking
10.0.254.2 = standard AntiTracker to block advertising and malware domains
10.0.254.3 = Hardcore Mode AntiTracker to also block Google and Facebook domains

Click the Save button.

Last Steps

A device reboot is not required, though it may be useful to confirm that everything behaves as expected.
Run a leak test at https://www.dnsleaktest.com via one of the internal network clients attached to your OpenWRT router.

Setting up WireGuard on Home Assistant

Back to the Top

Install Wireguard Add-on in Home Assistant

Next, open up Home Assistant. Go to Supervisor > Add-on store, and search for WireGuard.
Click the WireGuard addon, and the click Install.

Configure Wireguard Settings

After installing WireGuard, do not start it yet. We need to configure a few options first.

Click the Configuration tab at the very top.
There are two blocks of code here: server and peers. The server section is the WireGuard server info, and the peers section is where you’d add new devices that will connect to your VPN.

Server Configuration

Host: add the subdomain you just created. (vpn.mydomain.com)
Addresses: If your internal network is using the 192.168.x.x or 10.x.x.x range, you can leave the default IP addresses WireGuard has provided. (see note above)
DNS: Set to your router’s internal IP address (Open CMD > ipconfig /all > Under DNS servers) If you have Adguard or PiHole installed, you can use the IP address of those instead. This will allow you to block ads even when connected to the WireGuard VPN.

Peers Configuration

This is where you’ll create WireGuard configuration files for each of the devices you want to connect to WireGuard with. For this example, I’m using my phone and leaving allowed_ips and client_allowed_ips as is. If you adding multiple devices, then you’ll need to copy the entire block of code starting at name, give it a different name, and add the next available IP address (For example: 172.27.66.4)

Click Save once finished.

Then, go back to the Info tab and click Start.

Port Forward

The next step is to forward port 51820 from your Home Assistant server through your router. Unfortunately, there are so many different types of routers, each with different steps to port forward. The important thing to note is that you’ll be port forwarding 51820(wireguard port) from the internal IP of your Home Assistant instance (for example: 192.168.68.24) and choosing the UDP protocol only.

Download Wireguard app on mobile device

Download the WireGuard app from the Apple App Store or Google Play Store. You will need it for the next step.

If all goes well, you can click into the new tunnel connection from within the app. If you see data flowing under the Transfer section, that means you are good to go.

Improving Security

Once you have everything setup and working correctly, you should read through the WireGuard Addon docs to setup up allowed_ips and client_allowed_ips to further secure your VPN instance. There’s also some other helpful options you can configure such as log level, but these are all optional.

Nextcloud

Back to the Top

Nextcloud is an industry-leading, on-premises content collaboration platform for file sync & share and communication server. It is fully open source and you can host it yourself or pay a company to do it for you. Also checkout the following links below:

Nexcloud login screen

Nextcloud Hub is a tool that allows you to share and collaborate on documents, send and receive email, manage your calendar and have video chats without data leaks. As fully on-premises solution, Nextcloud Hub provides the benefits of online collaboration without the compliance and security risks.

Nexcloud Hub

Nextcloud AIO (All In One) is a tool that provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Features it includes:

Nextcloud
Nextcloud Office
High performance backend for Nextcloud Files
High performance backend for Nextcloud Talk
Backup solution (based on BorgBackup)
Imaginary
ClamAV
Fulltextsearch

Nextcloud Desktop Client is a tool to synchronize files from Nextcloud Server with your computer.

Nextcloud Deck is a kanban style organization tool aimed at personal planning and project organization for teams integrated with Nextcloud.

Nextcloud Files is a tool tool that allows your employees have easy access to their files, photos and documents to work and can share and collaborate with team members, customers and partners. So IT knows nobody besides those they shared with has access to those files.

Nextcloud Talk is a tool that protects your communication better than other team collaboration platforms like Microsoft Teams or Slack, making sure your data stays on your servers. It also goes further than other encrypted communication technologies by keeping even metadata from leaking.

Nextcloud Home is a tool that allows you store your documents, calendar, contacts and photos on your server at home, at one of at one Nextcloud's providers or in a data center you trust.

Nextcloud Enterprise is a service that gives professional organizations software optimized and tested for mission critical environments.

Nextcloud Outlook Integration is a tool that automatically upload files to replace large attachments or integrate Calendars and Contacts in Microsoft Outlook.

Collabora Online in Nextcloud is a powerful LibreOffice-based online office suite with collaborative editing, which supports all major document, spreadsheet and presentation file formats and works in all modern browsers.

ONLYOFFICE integration in Nextcloud is a service that empowers your users to collaborate on office documents with team members in real time. It has compatibility with Microsoft Office formats means perfect documents, every time.

Nextcloud VM(virtual machine appliance) is a set of carefully crafted family of *nix scripts, which interactively guide you through a quality-controlled installation of a Nextcloud instance for Home/SME Server and scripts for Raspberry Pi 4. It is Community developed and maintained.

Raspberry Pi

Back to the Top

Models of Raspberry Pi boards

Back to the Top

Raspberry Pi 4 Model B

Check out the Raspberry Pi 4

Raspberry Pi 4 Model B Hardware Specifications

Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
2GB, 4GB or 8GB LPDDR4-3200 SDRAM (depending on model)
2.4 GHz and 5.0 GHz IEEE 802.11ac wireless
Bluetooth 5.0, BLE
Gigabit Ethernet
2 USB 3.0 ports; 2 USB 2.0 ports.
Raspberry Pi standard 40 pin GPIO header (fully backwards compatible with previous Pi boards)
2 × micro-HDMI ports (up to 4kp60 supported)
OpenGL ES 3.0 graphics

Raspberry Pi 400 Personal Computer Kit

Check out the Raspberry Pi 400 Personal Computer Kit

Raspberry Pi 400 Hardware Specifications

Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.8GHz
4GB LPDDR4-3200 SDRAM
2.4 GHz and 5.0 GHz IEEE 802.11ac wireless
Bluetooth 5.0, BLE
Gigabit Ethernet
2 USB 3.0 ports; 2 USB 2.0 ports.
Raspberry Pi standard 40 pin GPIO header
2 × micro-HDMI ports (up to 4kp60 supported)
OpenGL ES 3.0 graphics

Raspberry Pi Pico Microcontroller

Check out the Raspberry Pi Pico

Raspberry Pi Pico Hardware Specifications

RP2040 microcontroller chip designed by Raspberry Pi in the UK
Dual-core Arm Cortex-M0+ processor, flexible clock running up to 133 MHz
264KB on-chip SRAM
2MB on-board QSPI Flash
26 multifunction GPIO pins, including 3 analogue inputs
2 × UART, 2 × SPI controllers, 2 × I2C controllers, 16 × PWM channels
1 × USB 1.1 controller and PHY, with host and device support
8 × Programmable I/O (PIO) state machines for custom peripheral support
Castellated module allows soldering direct to carrier boards
Drag-and-drop programming using mass storage over USB
Low-power sleep and dormant modes
Accurate on-chip clock
Temperature sensor
Accelerated integer and floating-point libraries on-chip

Raspberry Pi OS. The default Operating System for every Raspberry Pi device

Check out Raspberry Pi OS

Raspberry Pi Learning Resources

Back to the Top

Raspberry Pi is an ARM powered single board computer(SBC) that is the size of a credit card and costs around $35.

Raspberry Pi Foundation is a UK-based charity that works to put the power of computing and digital making into the hands of people all over the world.

Microsecond accurate NTP with a Raspberry Pi and PPS GPS

Getting Started with Raspberry Pi Projects

Online learning for the Raspberry Pi

Raspberry Pi Training Program

Raspberry Pi Online Courses on Udemy

Raspberry Pi Online Courses on Coursera

The Raspberry Pi Platform and Python Programming course on Coursera

Learning Raspberry Pi with Online Courses on edX

Raspberry Pi Online Training Courses on LinkedIn Learning

Getting Started with Raspberry Pi course on FutureLearn

Home Assistant on Raspberry Pi

PiSwitch: Build your own Nintendo Switch-style console

Raspberry Pi Operating Systems

Back to the Top

Raspberry Pi OS

Hass.io(Home Assistant OS)

OmniROM(Android 11) based on ASOP

Manjaro Linux ARM

Arch Linux ARM

Ubuntu MATE for Raspberry Pi

Ubuntu Desktop for Raspberry Pi

Ubuntu Core on a Raspberry Pi

Ubuntu Server for ARM

Fedora ARM

Kali Linux for the Raspberry Pi

Raspberry Pi Tools

Back to the Top

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Raspberry Pi Locator is a website to track Raspberry Pi 4 model B, Compute Module 4, Pi Zero 2 W, and Pico availability across multiple retailers in different countries.

Raspberry Pi Network Install (Beta) is a feature can be used to start the Raspberry Pi Imager application directly on a Raspberry Pi 4, or a Raspberry Pi 400, by downloading it from the internet using an Ethernet cable. The Raspberry Pi Imager application, which will run in memory on your Raspberry Pi, can then be used to flash the operating system onto a blank SD Card or USB disk, just like normal.

Raspberry Pi Bootloader is a feature, which is now available in beta, that utilize an EEPROM(Electrically Erasable Programmable Read-Only Memory) to store the system’s bootloader. This EEPROM is persistent storage that is located on the Pi’s mainboard. The advantage of using the EEPROM instead is that the Raspberry Pi 4 can perform tasks without needing any storage to be attached.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Home Assistant is an open source home automation that puts local control and privacy first. Home Assistant is powered by a worldwide community of tinkerers and DIY enthusiasts that runs great on Raspberry Pi.

Gladys Assistant is a privacy-first, open-source home assistant and runs great on Raspberry Pi.

Kodi for Raspberry Pi is a free and open source media player application developed by the XBMC/Kodi Foundation.

Pi-hole is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software, intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations.

PiKVM is a very simple and fully functional Raspberry Pi-based KVM over IP.

PiShrink is a bash script that automatically shrink a pi image that will then resize to the max size of the SD card on boot.

RPiPlay is an open-source implementation of an AirPlay mirroring server for the Raspberry Pi that supports iOS 9 and later.

Gpiozero is a simple interface to GPIO(General-Purpose Input/Output) devices with the Raspberry Pi.

Balena Sound is a single or multi-room streamer for an existing audio device using a Raspberry Pi! It supports Bluetooth, Airplay and Spotify Connect.

OpenBalena is a platform to deploy and manage connected devices.

Home Assistant

Back to the Top

Home Assistant is a container-based system for managing your Home Assistant Core installation and related applications. The system is controlled via Home Assistant which communicates with the Supervisor. The Supervisor provides an API to manage the installation. This includes changing network settings or installing and updating software.

Quick Links

Home Assistant Frontend is a frontend for Home Assistant.

Tools to write the HA image to your boot media(microSD card or USB device)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Home Assistant integrations

Home Assistant integrations. Credit: Home Assistant

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Shelly Cloud is a Smart home control tool that has been perfected and provides precise monitoring of your Shelly devices no matter where you are. Shelly devices are compatible with Alexa, Google Home, Android, and iOS.

Plex media server is a application that gives you the power to add, access and share all the entertainment that matters to you, on almost any device. With 50,000+ on demand titles and hundreds of channels of live TV, plus your own personal media collection, using one powerful app.

Amazon Alexa is a smart virtual assistant software to manage Alexa-enabled devices, control music playback, view shopping lists on the go, keep track of upcoming reminders, check on active timers and much more.

Google Assistant is a smart virtual assistant software on mobile and home automation devices.

Apple HomeKit is a software framework that enables your app to coordinate and control home automation accessories from multiple vendors to present a coherent, user-focused interface. Using HomeKit, your app can: Discover HomeKit-compatible automation accessories and add them to a persistent, cross-device home configuration database.

Samsung SmartThings is a sofwtare frmaeowrk that you can connect, monitor and control multiple smart home devices quicker and easier. Connect your Samsung smart TVs, smart appliances, smart speakers and brands like Ring, Nest and Philips Hue all from one app.

Ecobee is a home automation company in Canada that makes thermostats for residential and commercial use.

Lutron Caséta is a smart lighting control system that is a great solution for giving any client smart lighting control. It was purposely built to work in homes of all ages and it works with older wiring as well as new.

Philips Hue is a smart lighting system. The smart lights, Hue Bridge, and smart controls will forever change the way you experience light.

Sonos is the wireless home sound system that fills as many rooms as you want with great-sounding music, movies, and TV.

MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth.

Zigbee is the full-stack, secure, reliable, and market-proven solution used by a majority of large smart home ecosystem providers, such as Amazon's Echo Plus, Samsung SmartThings, Signify (Philips Hue), and more.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone's home smarter and safer.

Zwavejs2Mqtt is a fully configurable Zwave to MQTT Gateway and Control Panel Web UI.

Z-Wave JS Server is a small server wrapper around Z-Wave JS to access it via a WebSocket.

Z-Wave JS Config DB Browser is the official device configuration reference to find out if your device is supported. Currently supports 387 brands, spanning at least 2075 device configurations.

Homebridge

Back to the Top

Homebridge is a software frameowrk that allows you to integrate with smart home devices that do not natively support HomeKit. There are over 2,000 Homebridge plugins supporting thousands of different smart accessories.

Tools to write the Homebridge image to your boot media(microSD card or USB device)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

Homebridge UI

ESPHome

Back to the Top

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Quick Links

Install ESPHome using Home Assistant

In Home Assistant go to:

Configuration > Add-ons, Backups & Supervisor > Add-on Store (button in the lower right corner) or click on the My Home Assistant Link below:

Open your Home Assistant instance and show the Supervisor add-on store.

Next, search for ESPHome, click on the result and then click on the Install button.

When the installation is finished, the Install button will be replaced with Start button – click on it to start the ESPHome add-on.

Wait a few seconds for the ESPHome to start and then click on the Open Web UI button.

Install ESPHome using Docker

First thing is to pull the ESPHome Docker image from Docker Hub (Online).

docker pull esphome/esphome
Then, start the ESPHome wizard. This wizard will ask you about your device type, your device name, your WiFi credentials and finally will generate a yaml file containing all of the configurations for you.

docker run --rm -v "${PWD}":/config -it esphome/esphome wizard stl.yaml

Now, connect your ESP device to the device where Docker is running (either using an USB cable or Serial-To-USB adapter) and if you are on Linux type the following command :

dmesg | grep ttyUSB

Put your device in programming mode (if needed) and execute the next command to install the ESPHome on the device connected to the /dev/ttyUSB1 using the configuration stored in stl.yaml file

docker run --rm -v "${PWD}":/config --device=/dev/ttyUSB1 -it esphome/esphome run stl.yaml

Install ESPHome using Python

If you are on macOS or Linux check if Python 3.8 or later is installed by executing the command.

python3 --version

If you are on macOS, you need to install wheel and esphome packages by using the following command.

pip3 install wheel esphome

If you are on Linux, you have to install esphome package by using the following command.

pip3 install --user esphome

If you are on macOS or Linux you can start the ESPHome wizard using the following command.

esphome wizard stl-python.yaml

Finally, connect your ESP device to your Computer (using USB cable or Serial-To-usb adapter) and put it in programming mode (if needed). Then, Install ESPHome using the configuration in the stl-python.yaml file.

esphome run stl-python.yaml

Turning Raspberry Pi into a Router

Back to the Top

Software

OpenWrt Project is a Linux operating system targeting embedded devices. Instead of trying to create a single, static firmware, OpenWrt provides a fully writable filesystem with package management. It's primarily used on embedded devices to route network traffic.

OpenWrt Wiki - Raspberry Pi setup

Download the appropriate OpenWrt image for your Raspberry PI by going to the link above.

Tools to write the Operating System (OS) image to your boot media(microSD card)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Hardware

Raspberry Pi Router Board for CM4 module (Cost: $55 USD) is an expansion board based on the Raspberry Pi Compute Module 4. It brings Raspberry Pi CM4 two full-speed gigabit network ports and offers better performance, lower CPU usage, and higher stability for a long time work compared with a USB network card. It's compatible with Raspberry Pi OS, Ubuntu Server and other Raspberry Pi systems.

Raspberry Pi Router Board for CM4 module

Technical Specs:

Compatible Module: Raspberry Pi Compute Module 4 series.
BCM2711 4 core @ 1.5GHz Cortex-A72.
Support standard Raspberry Pi HAT interface.
Support POE HAT to supply power to the board.
Support POE HAT for external power supply.
Full-speed dual gigabit network interface.
Master-slave dual USB2.0 interface.
Micro SD card slot, used to support non-eMMC version of CM4.
Standard HDMI video output interface.
0.91 inch IIC OLED display.
5V DC fan interface(Support controlling via PWM signal).
Ethernet: high-performance Gigabit ethernet controller RTL8111E chip, JXD 2111x G2406s chip as isolation transformer.
- Port0: Compute Module 4 Built-In.
- Port1: PCI Express 1000BASE-T NIC.
GPIO: 40-Pin GPIO compatible with Raspberry Pi.

Setting Watchdog Timer (WDT) on Raspberry Pi

Back to the Top

Watchdog Timer (WDT) is a timer that monitors microcontroller (MCU) programs to see if they are out of control or have stopped operating.

Installing and enabling WDT service

To enable watchdog you have to change the boot parameters by adding dtparam=watchdog=on in /boot/config.txt using a text editor such as nano, vim, gedit, etc.. Also, install watchdog package and enable it to start at startup. Also, make sure to restart your Raspberry Pi for these settings to take effect.

pi@raspberrypi:~ $ sudo apt install watchdog

pi@raspberrypi:~ $sudo systemctl enable watchdog

Configure WDT service

Configuration file for watchdog can be found in /etc/watchdog.conf.

max-load-1 = 24
watchdog-device = /dev/watchdog
realtime = yes
priority = 1

To start the WTD service:

pi@raspberrypi:~ $ sudo systemctl start watchdog

Check watchdog status:

pi@raspberrypi:~ $ sudo systemctl status watchdog

To stop the service:

pi@raspberrypi:~ $ sudo systemctl stop watchdog

Raspberry Pi Upgrades

Back to the Top

Raspberry Pi Cases from Pi-Shop US

Raspberry Pi Cases from The Pi Hut

X825 expansion board provides a complete storage solution for newest Raspberry Pi 4 Model B, it supports up to 4TB 2.5-inch SATA hard disk drives (HDD) / solid-state drive (SSD).

Sabrent M.2 SSD [NGFF] to USB 3.0 / SATA III 2.5-Inch Aluminum Enclosure Adapter

Samsung 970 EVO 250GB - NVMe PCIe M.2 2280 SSD

Western Digital 1TB WD Blue SN550 NVMe Internal SSD

SAMSUNG T5 Portable SSD

Samsung SSD 860 EVO 250GB mSATA Internal SSD

Samsung 850 EVO 120GB SSD mSATA

Grafana

Back to the Top

Grafana Learning Resources

Grafana is an analytics platform that enables you to query and visualize data, then create and share dashboards based on your visualizations. Easily visualize metrics, logs, and traces from multiple sources such as Prometheus, Loki, Elasticsearch, InfluxDB, Postgres, Fluentd, Fluentbit, Logstash and many more.

Getting Started with Grafana

Grafana Community

Grafana Professional Services Training | Grafana Labs

Grafana Pro Training AWS | Grafana Labs

Grafana Tutorials

Top Grafana Courses on Udemy

Grafana Online Training Courses | LinkedIn Learning

Grafana Training Courses - NobleProg

Setting Up Grafana to Visualize Our Metrics Course on Coursera

Grafana Tools

Grafana Cloud is a composable observability platform, integrating metrics, traces and logs with Grafana. Leverage the best open source observability software – including Prometheus, Loki, and Tempo – without the overhead of installing, maintaining, and scaling your observability stack.

Grafana Cloud Integrations. Source: Grafana

Grafana Enterprise is a service that includes features that provide better scalability, collaboration, operations, and governance in a self-managed environment.

Grafana Enterprise Stack. Source: Grafana

Grafana Tempo is an open source high-scale distributed tarcing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Loki, and Prometheus.

Grafana MetricTank is a multi-tenant timeseries platform for Graphite developed by Grafana Labs. MetricTank provides high-availability(HA) and efficient long-term storage, retrieval, and processing for large-scale environments.

Grafana Tanka is a robust configuration utility for your Kubernetes cluster, powered by the Jsonnet language.

Grafana Loki is a horizontally-scalable, highly-available(HA), multi-tenant log aggregation system inspired by Prometheus.

Cortex is a project that lets users query metrics from many Prometheusservers in a single place, without any gaps in the grpahs due to server failture. Also, Cortex lets you store Prometheus metrics for long term capacity planning and performance analysis.

Graphite is an open source monitoring system.

Networking

Back to the Top

Networking Tools & Concepts

cURL is a computer software project providing a library and command-line tool for transferring data using various network protocols(HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP or SMTPS). cURL is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the Internet transfer engine for thousands of software applications in over ten billion installations.

cURL Fuzzer is a quality assurance testing for the curl project.

DoH is a stand-alone application for DoH (DNS-over-HTTPS) name resolves and lookups.

Authelia is an open-source highly-available authentication server providing single sign-on capability and two-factor authentication to applications running behind NGINX.

nginx(engine x) is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev.

Proxmox Virtual Environment(VE) is a complete open-source platform for enterprise virtualization. It inlcudes a built-in web interface that you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

Wireshark is a very popular network protocol analyzer that is commonly used for network troubleshooting, analysis, and communications protocol development. Learn more about the other useful Wireshark Tools available.

HTTPie is a command-line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. HTTPie is designed for testing, debugging, and generally interacting with APIs & HTTP servers.

HTTPStat is a tool that visualizes curl statistics in a simple layout.

Wuzz is an interactive cli tool for HTTP inspection. It can be used to inspect/modify requests copied from the browser's network inspector with the "copy as cURL" feature.

Websocat is a ommand-line client for WebSockets, like netcat (or curl) for ws:// with advanced socat-like functions.

• Connection: In networking, a connection refers to pieces of related information that are transferred through a network. This generally infers that a connection is built before the data transfer (by following the procedures laid out in a protocol) and then is deconstructed at the at the end of the data transfer.

• Packet: A packet is, generally speaking, the most basic unit that is transferred over a network. When communicating over a network, packets are the envelopes that carry your data (in pieces) from one end point to the other.

Packets have a header portion that contains information about the packet including the source and destination, timestamps, network hops. The main portion of a packet contains the actual data being transferred. It is sometimes called the body or the payload.

• Network Interface: A network interface can refer to any kind of software interface to networking hardware. For instance, if you have two network cards in your computer, you can control and configure each network interface associated with them individually.

A network interface may be associated with a physical device, or it may be a representation of a virtual interface. The "loop-back" device, which is a virtual interface to the local machine, is an example of this.

• LAN: LAN stands for "local area network". It refers to a network or a portion of a network that is not publicly accessible to the greater internet. A home or office network is an example of a LAN.

• WAN: WAN stands for "wide area network". It means a network that is much more extensive than a LAN. While WAN is the relevant term to use to describe large, dispersed networks in general, it is usually meant to mean the internet, as a whole.

If an interface is connected to the WAN, it is generally assumed that it is reachable through the internet.

• Protocol: A protocol is a set of rules and standards that basically define a language that devices can use to communicate. There are a great number of protocols in use extensively in networking, and they are often implemented in different layers.

Some low level protocols are TCP, UDP, IP, and ICMP. Some familiar examples of application layer protocols, built on these lower protocols, are HTTP (for accessing web content), SSH, TLS/SSL, and FTP.

• Port: A port is an address on a single machine that can be tied to a specific piece of software. It is not a physical interface or location, but it allows your server to be able to communicate using more than one application.

• Firewall: A firewall is a program that decides whether traffic coming into a server or going out should be allowed. A firewall usually works by creating rules for which type of traffic is acceptable on which ports. Generally, firewalls block ports that are not used by a specific application on a server.

• NAT: Network address translation is a way to translate requests that are incoming into a routing server to the relevant devices or servers that it knows about in the LAN. This is usually implemented in physical LANs as a way to route requests through one IP address to the necessary backend servers.

• VPN: Virtual private network is a means of connecting separate LANs through the internet, while maintaining privacy. This is used as a means of connecting remote systems as if they were on a local network, often for security reasons.

Network Layers

While networking is often discussed in terms of topology in a horizontal way, between hosts, its implementation is layered in a vertical fashion throughout a computer or network. This means is that there are multiple technologies and protocols that are built on top of each other in order for communication to function more easily. Each successive, higher layer abstracts the raw data a little bit more, and makes it simpler to use for applications and users. It also allows you to leverage lower layers in new ways without having to invest the time and energy to develop the protocols and applications that handle those types of traffic.

As data is sent out of one machine, it begins at the top of the stack and filters downwards. At the lowest level, actual transmission to another machine takes place. At this point, the data travels back up through the layers of the other computer. Each layer has the ability to add its own "wrapper" around the data that it receives from the adjacent layer, which will help the layers that come after decide what to do with the data when it is passed off.

One method of talking about the different layers of network communication is the OSI model. OSI stands for Open Systems Interconnect.This model defines seven separate layers. The layers in this model are:

• Application: The application layer is the layer that the users and user-applications most often interact with. Network communication is discussed in terms of availability of resources, partners to communicate with, and data synchronization.

• Presentation: The presentation layer is responsible for mapping resources and creating context. It is used to translate lower level networking data into data that applications expect to see.

• Session: The session layer is a connection handler. It creates, maintains, and destroys connections between nodes in a persistent way.

• Transport: The transport layer is responsible for handing the layers above it a reliable connection. In this context, reliable refers to the ability to verify that a piece of data was received intact at the other end of the connection. This layer can resend information that has been dropped or corrupted and can acknowledge the receipt of data to remote computers.

• Network: The network layer is used to route data between different nodes on the network. It uses addresses to be able to tell which computer to send information to. This layer can also break apart larger messages into smaller chunks to be reassembled on the opposite end.

• Data Link: This layer is implemented as a method of establishing and maintaining reliable links between different nodes or devices on a network using existing physical connections.

• Physical: The physical layer is responsible for handling the actual physical devices that are used to make a connection. This layer involves the bare software that manages physical connections as well as the hardware itself (like Ethernet).

The TCP/IP model, more commonly known as the Internet protocol suite, is another layering model that is simpler and has been widely adopted.It defines the four separate layers, some of which overlap with the OSI model:

• Application: In this model, the application layer is responsible for creating and transmitting user data between applications. The applications can be on remote systems, and should appear to operate as if locally to the end user.

The communication takes place between peers network.

• Transport: The transport layer is responsible for communication between processes. This level of networking utilizes ports to address different services. It can build up unreliable or reliable connections depending on the type of protocol used.

• Internet: The internet layer is used to transport data from node to node in a network. This layer is aware of the endpoints of the connections, but does not worry about the actual connection needed to get from one place to another. IP addresses are defined in this layer as a way of reaching remote systems in an addressable manner.

• Link: The link layer implements the actual topology of the local network that allows the internet layer to present an addressable interface. It establishes connections between neighboring nodes to send data.

Interfaces

Interfaces are networking communication points for your computer. Each interface is associated with a physical or virtual networking device. Typically, your server will have one configurable network interface for each Ethernet or wireless internet card you have. In addition, it will define a virtual network interface called the "loopback" or localhost interface. This is used as an interface to connect applications and processes on a single computer to other applications and processes. You can see this referenced as the "lo" interface in many tools.

Network Protocols

Networking works by piggybacks on a number of different protocols on top of each other. In this way, one piece of data can be transmitted using multiple protocols encapsulated within one another.

Media Access Control(MAC) is a communications protocol that is used to distinguish specific devices. Each device is supposed to get a unique MAC address during the manufacturing process that differentiates it from every other device on the internet. Addressing hardware by the MAC address allows you to reference a device by a unique value even when the software on top may change the name for that specific device during operation. Media access control is one of the only protocols from the link layer that you are likely to interact with on a regular basis.

The IP protocol is one of the fundamental protocols that allow the internet to work. IP addresses are unique on each network and they allow machines to address each other across a network. It is implemented on the internet layer in the IP/TCP model. Networks can be linked together, but traffic must be routed when crossing network boundaries. This protocol assumes an unreliable network and multiple paths to the same destination that it can dynamically change between. There are a number of different implementations of the protocol. The most common implementation today is IPv4, although IPv6 is growing in popularity as an alternative due to the scarcity of IPv4 addresses available and improvements in the protocols capabilities.

ICMP: internet control message protocol is used to send messages between devices to indicate the availability or error conditions. These packets are used in a variety of network diagnostic tools, such as ping and traceroute. Usually ICMP packets are transmitted when a packet of a different kind meets some kind of a problem. Basically, they are used as a feedback mechanism for network communications.

TCP: Transmission control protocol is implemented in the transport layer of the IP/TCP model and is used to establish reliable connections. TCP is one of the protocols that encapsulates data into packets. It then transfers these to the remote end of the connection using the methods available on the lower layers. On the other end, it can check for errors, request certain pieces to be resent, and reassemble the information into one logical piece to send to the application layer. The protocol builds up a connection prior to data transfer using a system called a three-way handshake. This is a way for the two ends of the communication to acknowledge the request and agree upon a method of ensuring data reliability. After the data has been sent, the connection is torn down using a similar four-way handshake. TCP is the protocol of choice for many of the most popular uses for the internet, including WWW, FTP, SSH, and email. It is safe to say that the internet we know today would not be here without TCP.

UDP: User datagram protocol is a popular companion protocol to TCP and is also implemented in the transport layer. The fundamental difference between UDP and TCP is that UDP offers unreliable data transfer. It does not verify that data has been received on the other end of the connection. This might sound like a bad thing, and for many purposes, it is. However, it is also extremely important for some functions. It’s not required to wait for confirmation that the data was received and forced to resend data, UDP is much faster than TCP. It does not establish a connection with the remote host, it simply fires off the data to that host and doesn't care if it is accepted or not. Since UDP is a simple transaction, it is useful for simple communications like querying for network resources. It also doesn't maintain a state, which makes it great for transmitting data from one machine to many real-time clients. This makes it ideal for VOIP, games, and other applications that cannot afford delays.

HTTP: Hypertext transfer protocol is a protocol defined in the application layer that forms the basis for communication on the web. HTTP defines a number of functions that tell the remote system what you are requesting. For instance, GET, POST, and DELETE all interact with the requested data in a different way.

FTP: File transfer protocol is in the application layer and provides a way of transferring complete files from one host to another. It is inherently insecure, so it is not recommended for any externally facing network unless it is implemented as a public, download-only resource.

DNS: Domain name system is an application layer protocol used to provide a human-friendly naming mechanism for internet resources. It is what ties a domain name to an IP address and allows you to access sites by name in your browser.

SSH: Secure shell is an encrypted protocol implemented in the application layer that can be used to communicate with a remote server in a secure way. Many additional technologies are built around this protocol because of its end-to-end encryption and ubiquity. There are many other protocols that we haven't covered that are equally important. However, this should give you a good overview of some of the fundamental technologies that make the internet and networking possible.

REST(REpresentational State Transfer) is an architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other.

JSON Web Token (JWT) is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).

OAuth 2.0 is an open source authorization framework that enables applications to obtain limited access to user accounts on an HTTP service, such as Amazon, Google, Facebook, Microsoft, Twitter GitHub, and DigitalOcean. It works by delegating user authentication to the service that hosts the user account, and authorizing third-party applications to access the user account.

Docker

Back to the Top

Container Architecture. Source: Containerd.io

Docker Learning Resources

Docker Training Program

Docker Certified Associate (DCA) certification

Docker Documentation | Docker Documentation

The Docker Workshop

Docker Courses on Udemy

Docker Courses on Coursera

Docker Courses on edX

Docker Courses on Linkedin Learning

Docker Tools

Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly working in collaboration with cloud, Linux, and Windows vendors, including Microsoft.

Docker Enterprise is a subscription including software, supported and certified container platform for CentOS, Red Hat Enterprise Linux (RHEL), Ubuntu, SUSE Linux Enterprise Server (SLES), Oracle Linux, and Windows Server 2016, as well as for cloud providers AWS and Azure. In November 2019 Docker's Enterprise Platform business was acquired by Mirantis.

Docker Desktop is an application for MacOS and Windows machines for the building and sharing of containerized applications and microservices. Docker Desktop delivers the speed, choice and security you need for designing and delivering containerized applications on your desktop. Docker Desktop includes Docker App, developer tools, Kubernetes and version synchronization to production Docker Engines.

Docker Hub is the world's largest library and community for container images Browse over 100,000 container images from software vendors, open-source projects, and the community.

Docker Compose is a tool that was developed to help define and share multi-container applications. With Docker Compose, you can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

Docker Swarm is a Docker-native clustering system swarm is a simple tool which controls a cluster of Docker hosts and exposes it as a single "virtual" host.

Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession.

Docker Containers is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

Docker Engine is a container runtime that runs on various Linux (CentOS, Debian, Fedora, Oracle Linux, RHEL, SUSE, and Ubuntu) and Windows Server operating systems. Docker creates simple tooling and a universal packaging approach that bundles up all application dependencies inside a container which is then run on Docker Engine.

Docker Images is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings. Images have intermediate layers that increase reusability, decrease disk usage, and speed up docker build by allowing each step to be cached. These intermediate layers are not shown by default. The SIZE is the cumulative space taken up by the image and all its parent images.

Docker Network is a that displays detailed information on one or more networks.

Docker Daemon is a service started by a system utility, not manually by a user. This makes it easier to automatically start Docker when the machine reboots. The command to start Docker depends on your operating system. Currently, it only runs on Linux because it depends on a number of Linux kernel features, but there are a few ways to run Docker on MacOS and Windows as well by configuring the operating system utilities.

Docker Storage is a driver controls how images and containers are stored and managed on your Docker host.

Kitematic is a simple application for managing Docker containers on Mac, Linux and Windows letting you control your app containers from a graphical user interface (GUI).

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images. Podman provides a command line interface (CLI) familiar to anyone who has used the Docker Container Engine.

Containerd is a daemon that manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond. It is available for Linux and Windows.

Kubernetes

Back to the Top

Kubernetes Learning Resources

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

Getting Kubernetes Certifications

Getting started with Kubernetes on AWS

Kubernetes on Microsoft Azure

Intro to Azure Kubernetes Service

Azure Red Hat OpenShift

Getting started with Google Cloud

Getting started with Kubernetes on Red Hat

Getting started with Kubernetes on IBM

Red Hat OpenShift on IBM Cloud

Enable OpenShift Virtualization on Red Hat OpenShift

YAML basics in Kubernetes

Elastic Cloud on Kubernetes

Docker and Kubernetes

Running Apache Spark on Kubernetes

Kubernetes Across VMware vRealize Automation

VMware Tanzu Kubernetes Grid

All the Ways VMware Tanzu Works with AWS

VMware Tanzu Education

Using Ansible in a Cloud-Native Kubernetes Environment

Managing Kubernetes (K8s) objects with Ansible

Setting up a Kubernetes cluster using Vagrant and Ansible

Running MongoDB with Kubernetes

Kubernetes Fluentd

Understanding the new GitLab Kubernetes Agent

Intro Local Process with Kubernetes for Visual Studio 2019

Kubernetes Contributors

KubeAcademy from VMware

Kubernetes Tutorials from Pulumi

Kubernetes Playground by Katacoda

Scalable Microservices with Kubernetes course from Udacity

Kubernetes Tools, Frameworks, and Projects

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images. Podman provides a command line interface (CLI) familiar to anyone who has used the Docker Container Engine.

Containerd is a daemon that manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond. It is available for Linux and Windows.

Google Kubernetes Engine (GKE) is a managed, production-ready environment for running containerized applications.

Azure Kubernetes Service (AKS) is serverless Kubernetes, with a integrated continuous integration and continuous delivery (CI/CD) experience, and enterprise-grade security and governance. Unite your development and operations teams on a single platform to rapidly build, deliver, and scale applications with confidence.

Amazon EKS is a tool that runs Kubernetes control plane instances across multiple Availability Zones to ensure high availability.

AWS Controllers for Kubernetes (ACK) is a new tool that lets you directly manage AWS services from Kubernetes. ACK makes it simple to build scalable and highly-available Kubernetes applications that utilize AWS services.

Container Engine for Kubernetes (OKE) is an Oracle-managed container orchestration service that can reduce the time and cost to build modern cloud native applications. Unlike most other vendors, Oracle Cloud Infrastructure provides Container Engine for Kubernetes as a free service that runs on higher-performance, lower-cost compute.

Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.

Red Hat Openshift is a fully managed Kubernetes platform that provides a foundation for on-premises, hybrid, and multicloud deployments.

OKD is a community distribution of Kubernetes optimized for continuous application development and multi-tenant deployment. OKD adds developer and operations-centric tools on top of Kubernetes to enable rapid application development, easy deployment and scaling, and long-term lifecycle maintenance for small and large teams.

Odo is a fast, iterative, and straightforward CLI tool for developers who write, build, and deploy applications on Kubernetes and OpenShift.

Kata Operator is an operator to perform lifecycle management (install/upgrade/uninstall) of Kata Runtime on Openshift as well as Kubernetes cluster.

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.

OpenShift Hive is an operator which runs as a service on top of Kubernetes/OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift 4 clusters.

Rook is a tool that turns distributed storage systems into self-managing, self-scaling, self-healing storage services. It automates the tasks of a storage administrator: deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management.

VMware Tanzu is a centralized management platform for consistently operating and securing your Kubernetes infrastructure and modern applications across multiple teams and private/public clouds.

Kubespray is a tool that combines Kubernetes and Ansible to easily install Kubernetes clusters that can be deployed on AWS, GCE, Azure, OpenStack, vSphere, Packet (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal.

KubeInit provides Ansible playbooks and roles for the deployment and configuration of multiple Kubernetes distributions.

Rancher is a complete software stack for teams adopting containers. It addresses the operational and security challenges of managing multiple Kubernetes clusters, while providing DevOps teams with integrated tools for running containerized workloads.

K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.

Helm is a Kubernetes Package Manager tool that makes it easier to install and manage Kubernetes applications.

Knative is a Kubernetes-based platform to build, deploy, and manage modern serverless workloads. Knative takes care of the operational overhead details of networking, autoscaling (even to zero), and revision tracking.

KubeFlow is a tool dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable.

Etcd is a distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. Etcd is used as the backend for service discovery and stores cluster state and configuration for Kubernetes.

OpenEBS is a Kubernetes-based tool to create stateful applications using Container Attached Storage.

Container Storage Interface (CSI) is an API that lets container orchestration platforms like Kubernetes seamlessly communicate with stored data via a plug-in.

MicroK8s is a tool that delivers the full Kubernetes experience. In a Fully containerized deployment with compressed over-the-air updates for ultra-reliable operations. It is supported on Linux, Windows, and MacOS.

Charmed Kubernetes is a well integrated, turn-key, conformant Kubernetes platform, optimized for your multi-cloud environments developed by Canonical.

Grafana Kubernetes App is a toll that allows you to monitor your Kubernetes cluster's performance. It includes 4 dashboards, Cluster, Node, Pod/Container and Deployment. It allows for the automatic deployment of the required Prometheus exporters and a default scrape config to use with your in cluster Prometheus deployment.

KubeEdge is an open source system for extending native containerized application orchestration capabilities to hosts at Edge.It is built upon kubernetes and provides fundamental infrastructure support for network, app. deployment and metadata synchronization between cloud and edge.

Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily basis. It has support for MacOS, Windows and Linux operating systems.

Flux CD is a tool that automatically ensures that the state of your Kubernetes cluster matches the configuration you've supplied in Git. It uses an operator in the cluster to trigger deployments inside Kubernetes, which means that you don't need a separate continuous delivery tool.

Platform9 Managed Kubernetes (PMK) is a Kubernetes as a service that ensures fully automated Day-2 operations with 99.9% SLA on any environment, whether in data-centers, public clouds, or at the edge.

Ansible

Back to the Top

Mac Development Ansible Playbook by Jeff Geerling

Ansible Learning Resources

Ansible is a simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs. It uses a very simple language (YAML, in the form of Ansible Playbooks) that allows you to describe your automation jobs in a way that approaches plain English. Anisble works on Linux (Red Hat EnterPrise Linux(RHEL) and Ubuntu) and Microsoft Windows.

Red Hat Training for Ansible

Top Ansible Courses Online from Udemy

Introduction to Ansible: The Fundamentals on Coursera

Learning Ansible Fundamentals on Pluralsight

Introducing Red Hat Ansible Automation Platform 2.1

Ansible Documentation

Ansible Galaxy User Guide

Ansible Use Cases

Ansible Integrations

Ansible Collections Overview

Working with playbooks

Ansible for DevOps Examples by Jeff Geerling

Getting Started: Writing Your First Playbook - Ansible

Working With Modules in Ansible

Ansible Best Practices: Roles & Modules

Working with command line tools for Ansible

Encrypting content with Ansible Vault

Using vault in playbooks with Ansible

Using Ansible With Azure

Configuring Ansible on an Azure VM

How to Use Ansible: An Ansible Cheat Sheet Guide from DigitalOcean

Intro to Ansible on Linode | Spatial Labs

Ansible DevOps Tools Integration

Ansible Automation Hub is the official location to discover and download supported collections, included as part of an Ansible Automation Platform subscription. These content collections contain modules, plugins, roles, and playbooks in a downloadable package.

Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. As modules move from the core Ansible repository into collections, the module documentation will move to the collections pages.

Ansible Lint is a command-line tool for linting playbooks, roles and collections aimed towards any Ansible users. Its main goal is to promote proven practices, patterns and behaviors while avoiding common pitfalls that can easily lead to bugs or make code harder to maintain.

Ansible cmdb is a tool that takes the output of Ansible’s fact gathering and converts it into a static HTML overview page containing system configuration information.

Ansible Inventory Grapher visually displays inventory inheritance hierarchies and at what level a variable is defined in inventory.

Ansible Playbook Grapher is a command line tool to create a graph representing your Ansible playbook tasks and roles.

Ansible Shell is an interactive shell for Ansible with built-in tab completion for all the modules.

Ansible Silo is a self-contained Ansible environment by Docker.

Ansigenome is a command line tool designed to help you manage your Ansible roles.

ARA is a records Ansible playbook runs and makes the recorded data available and intuitive for users and systems by integrating with Ansible as a callback plugin.

Capistrano is a remote server automation tool. It supports the scripting and execution of arbitrary tasks, and includes a set of sane-default deployment workflows.

Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. It builds on top of Invoke (subprocess command execution and command-line features) and Paramiko (SSH protocol implementation), extending their APIs to complement one another and provide additional functionality.

ansible-role-wireguard is an Ansible role for installing WireGuard VPN. Supports Ubuntu, Debian, Archlinx, Fedora and CentOS Stream.

wireguard_cloud_gateway is an Ansible role for setting up Wireguard as a gateway VPN server for cloud networks.

Red Hat OpenShift is focused on security at every level of the container stack and throughout the application lifecycle. It includes long-term, enterprise support from one of the leading Kubernetes contributors and open source software companies.

OpenShift Hive is an operator which runs as a service on top of Kubernetes/OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift 4 clusters.

Databases

Back to the Top

SQL/NoSQL Learning Resources

SQL is a standard language for storing, manipulating and retrieving data in relational databases.

NoSQL is a database that is interchangeably referred to as "nonrelational, or "non-SQL" to highlight that the database can handle huge volumes of rapidly changing, unstructured data in different ways than a relational (SQL-based) database with rows and tables.

Transact-SQL(T-SQL) is a Microsoft extension of SQL with all of the tools and applications communicating to a SQL database by sending T-SQL commands.

Introduction to Transact-SQL

SQL Tutorial by W3Schools

Learn SQL Skills Online from Coursera

SQL Courses Online from Udemy

SQL Online Training Courses from LinkedIn Learning

Learn SQL For Free from Codecademy

GitLab's SQL Style Guide

OracleDB SQL Style Guide Basics

Tableau CRM: BI Software and Tools

Databases on AWS

Best Practices and Recommendations for SQL Server Clustering in AWS EC2.

Connecting from Google Kubernetes Engine to a Cloud SQL instance.

Educational Microsoft Azure SQL resources

MySQL Certifications

SQL vs. NoSQL Databases: What's the Difference?

What is NoSQL?

SQL/NoSQL Tools and Databases

Netdata is high-fidelity infrastructure monitoring and troubleshooting, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

Azure Data Studio is an open source data management tool that enables working with SQL Server, Azure SQL DB and SQL DW from Windows, macOS and Linux.

RStudio is an integrated development environment for R and Python, with a console, syntax-highlighting editor that supports direct code execution, and tools for plotting, history, debugging and workspace management.

MySQL is a fully managed database service to deploy cloud-native applications using the world's most popular open source database.

PostgreSQL is a powerful, open source object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It is a fully managed, multiregion, multimaster, durable database with built-in security, backup and restore, and in-memory caching for internet-scale applications.

Apache Cassandra™ is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Cassandra provides linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

Apache HBase™ is an open-source, NoSQL, distributed big data store. It enables random, strictly consistent, real-time access to petabytes of data. HBase is very effective for handling large, sparse datasets. HBase serves as a direct input and output to the Apache MapReduce framework for Hadoop, and works with Apache Phoenix to enable SQL-like queries over HBase tables.

Hadoop Distributed File System (HDFS) is a distributed file system that handles large data sets running on commodity hardware. It is used to scale a single Apache Hadoop cluster to hundreds (and even thousands) of nodes. HDFS is one of the major components of Apache Hadoop, the others being MapReduce and YARN.

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other frameworks on a dynamically shared pool of nodes.

Apache Spark is a unified analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing.

ElasticSearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java.

Logstash is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

Trino is a Distributed SQL query engine for big data. It is able to tremendously speed up ETL processes, allow them all to use standard SQL statement, and work with numerous data sources and targets all in the same system.

Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources, transform the data according to business rules, and load it into a destination data store.

Redis(REmote DIctionary Server) is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. It provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.

FoundationDB is an open source distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads. FoundationDB was acquired by Apple in 2015.

IBM DB2 is a collection of hybrid data management products offering a complete suite of AI-empowered capabilities designed to help you manage both structured and unstructured data on premises as well as in private and public cloud environments. Db2 is built on an intelligent common SQL engine designed for scalability and flexibility.

MongoDB is a document database meaning it stores data in JSON-like documents.

OracleDB is a powerful fully managed database helps developers manage business-critical data with the highest availability, reliability, and security.

MariaDB is an enterprise open source database solution for modern, mission-critical applications.

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine.SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day.

SQLite Database Browser is an open source SQL tool that allows users to create, design and edits SQLite database files. It lets users show a log of all the SQL commands that have been issued by them and by the application itself.

InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or monitoring and alerting purposes, user dashboards, Internet of Things sensor data, and visualizing and exploring the data and more. It also has support for processing data from Graphite.

Atlas is an in-memory dimensional time series database.

CouchbaseDB is an open source distributed multi-model NoSQL document-oriented database. It creates a key-value store with managed cache for sub-millisecond data operations, with purpose-built indexers for efficient queries and a powerful query engine for executing SQL queries.

dbWatch is a complete database monitoring/management solution for SQL Server, Oracle, PostgreSQL, Sybase, MySQL and Azure. Designed for proactive management and automation of routine maintenance in large scale on-premise, hybrid/cloud database environments.

Cosmos DB Profiler is a real-time visual debugger allowing a development team to gain valuable insight and perspective into their usage of Cosmos DB database. It identifies over a dozen suspicious behaviors from your application’s interaction with Cosmos DB.

Adminer is an SQL management client tool for managing databases, tables, relations, indexes, users. Adminer has support for all the popular database management systems such as MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch and MongoDB.

DBeaver is an open source database tool for developers and database administrators. It offers supports for JDBC compliant databases such as MySQL, Oracle, IBM DB2, SQL Server, Firebird, SQLite, Sybase, Teradata, Firebird, Apache Hive, Phoenix, and Presto.

DbVisualizer is a SQL management tool that allows users to manage a wide range of databases such as Oracle, Sybase, SQL Server, MySQL, H3, and SQLite.

AppDynamics Database is a management product for Microsoft SQL Server. With AppDynamics you can monitor and trend key performance metrics such as resource consumption, database objects, schema statistics and more, allowing you to proactively tune and fix issues in a High-Volume Production Environment.

Toad is a SQL Server DBMS toolset developed by Quest. It increases productivity by using extensive automation, intuitive workflows, and built-in expertise. This SQL management tool resolve issues, manage change and promote the highest levels of code quality for both relational and non-relational databases.

Lepide SQL Server is an open source storage manager utility to analyse the performance of SQL Servers. It provides a complete overview of all configuration and permission changes being made to your SQL Server environment through an easy-to-use, graphical user interface.

Sequel Pro is a fast MacOS database management tool for working with MySQL. This SQL management tool helpful for interacting with your database by easily to adding new databases, new tables, and new rows.

Telco 5G

Back to the Top

VMware Cloud First Approach. Source: VMware.

VMware Telco Cloud Automation Components. Source: VMware.

Telco Learning Resources

HPE(Hewlett Packard Enterprise) Telco Blueprints overview

Network Functions Virtualization Infrastructure (NFVI) by Cisco

Introduction to vCloud NFV Telco Edge from VMware

VMware Telco Cloud Automation(TCA) Architecture Overview

5G Telco Cloud from VMware

Maturing OpenStack Together To Solve Telco Needs from Red Hat

Red Hat telco ecosystem program

OpenStack for Telcos by Canonical

Open source NFV platform for 5G from Ubuntu

Understanding 5G Technology from Verizon

Verizon and Unity partner to enable 5G & MEC gaming and enterprise applications

Understanding 5G Technology from Intel

Understanding 5G Technology from Qualcomm

Telco Acceleration with Xilinx

VIMs on OSM Public Wiki

Amazon EC2 Overview and Networking Introduction for Telecom Companies

Citrix Certified Associate – Networking(CCA-N)

Citrix Certified Professional – Virtualization(CCP-V)

CCNP Routing and Switching

Certified Information Security Manager(CISM)

Wireshark Certified Network Analyst (WCNA)

Juniper Networks Certification Program Enterprise (JNCP)

Cloud Native Computing Foundation Training and Certification Program

Tools

Open Stack is an open source cloud platform, deployed as infrastructure-as-a-service (IaaS) to orchestrate data center operations on bare metal, private cloud hardware, public cloud resources, or both (hybrid/multi-cloud architecture). OpenStack includes advance use of virtualization & SDN for network traffic optimization to handle the core cloud-computing services of compute, networking, storage, identity, and image services.

StarlingX is a complete cloud infrastructure software stack for the edge used by the most demanding applications in industrial IOT, telecom, video delivery and other ultra-low latency use cases.

Airship is a collection of open source tools for automating cloud provisioning and management. Airship provides a declarative framework for defining and managing the life cycle of open infrastructure tools and the underlying hardware.