Install Recurve agent

Data Plane of Recurve consists of multiple worker hosts to run the data jobs orchestrated from the Control Plane.

Each worker contains a service called Recurve Agent that communicates with Control Plane and manages execution environments on the host. Some rules for setting workers are as follows:

• Multiple workers can be set up for horizontal scalability.

• Workers can be installed in different places across the Internet.

• Workers that are meant for a single environment must be able to communicate with each other. Therefore, it is recommended that workers for the same environment be located in the same local network, for maximum connection stability and minimum latency.

This guide shows you how to set up Recurve agent.

Requirements

Your worker hosts should meet these requirements.

Hardware:

Depending on the usage, your setup can follow these configuration ranges to optimize the performance with Recurve.

	Number of workers	CPU (cores / worker)	Memory (GB / worker)	Disk (GBs/worker)
Minimum	1	2	4	20
Recommended	2+	4	16	100

Software:

• Operating System: Linux (RHEL and Ubuntu are tested).

• Python: 3.9+, we recommend using a newer version and virtual environment if feasible.

• Docker: 20.10.0+, with Docker Compose installed.

Make sure the user runs the recurve agent has access to docker cli. This command might be helpful. You may need to logout and re-login to the terminal session after modifying the user's group.

sudo usermod -aG docker $USER 

Set up Recurve Agent

Follow these steps:

1. On the left sidebar of Recurve, click on the Organization icon and select Environment & Workers.

2. Click Deploy Agent and follow the on-screen instructions to deploy the agent.

   Alternatively, follow the steps below.

Install Recurve agent in CLI

1. On the worker host: Install the agent package.

pip install -U recurve-agent

You can verify the installation by running the recurve version.

1. On the worker host: Register the host to the Control Plane

recurve login

1. On the Environments & workers page, click Deploy Agent, copy the API key, and paste it to the host.

2. On the worker host: Start the agent; it will keep sending heartbeats to the Control Plane.

recurve agent start

1. Refresh the page to see the newly deployed host. The status should be Online.

2. Allocate the agent to an environment.

   Once the agent is running with the Online status, click action > Join Environment and select the environment to allocate. When the agent successfully joins the environment, you should see the Active status.

Note:

• The recurve agent start command starts the agent in the foreground, and will terminate after the terminal session closed. For production deployment, you should let the agent run in the background. For more details, refer to Run Agent in Background.

• Both dev and prod must have at least one agent running.

Actions on an agent

Once the agent is up and running, there are some key actions that you can perform with the service. Click on the action button of an agent:

Here you can:

• Restart Service: restart the agent service; the agent will still join the environment after restarting.

• Stop Service: stop the agent service; the agent will become inactive.

• Leave Environment: remove the agent service from the current environment.

Run the agent in background

There are multiple ways to run an agent in the background, for example, using systemd or Supervisor.

systemd

1. Create a configuration file.

Create/etc/systemd/system/recurve-agent.service with the following content:

[Unit]
Description=Recurve Agent Service
After=network.target

[Service]
User=<os_user>
Group=<os_user>
ExecStart=/path/to/python/bin/recurve agent start
Restart=on-failure
Environment=PYTHONUNBUFFERED=1
StandardOutput=append:/var/log/recurve/agent.log
StandardError=append:/var/log/recurve/agent.log

[Install]
WantedBy=multi-user.target

1. Prepare log folder and permission.

sudo mkdir -p /var/log/recurve
sudo chown <os_user>:<os_user> /var/log/recurve

1. Start the service.

sudo systemctl daemon-reload
sudo systemctl start recurve-agent
sudo systemctl enable recurve-agent

Supervisor

Install supervisor

1. Install supervisor.

# Install on Ubuntu 
sudo apt install supervisor

# Install on CentOS (RHEL)
sudo dnf install supervisor

# or
sudo yum install supervisor

1. Create groups and users.

sudo groupadd supervisor
sudo usermod -aG supervisor $USER

1. Create configuration file.

After the installation, there should already be a configuration file in /etc/supervisord.conf. Do some modifications to allow users within supervisor to run access supervisord.

[unix_http_server]
file=/run/supervisor/supervisor.sock   ; (the path to the socket file)
chmod=770
chown=root:supervisor

1. Enable and start supervisord

sudo systemctl enable supervisord
sudo systemctl start supervisord

Now you should be able to run supervisorctl (the control command line interface to supervisord)

Add recurve-agent

1. Add a configuration file to /etc/supervisord.d/recurve-agent.ini, the content seems like

[program:recurve-agent]
command = /path/to/python/bin/recurve agent start
autorestart = true
stopasgroup = true
killasgroup = true
umask = 000
user = <os_user>
redirect_stderr = true
stdout_logfile = /var/log/recurve/agent.log

Please note:

• Replace the command with the absolute path of the recurve binary. It's typically the bin folder of the Python environment (or virtual environment).

• Replace the user.

• Replace the log path stdout_logfile, the user should have write access to the folder.

1. Manage the process by supervisorctl

# Reload config and add/remove as necessary, and will restart affected programs
supervisorctl update

# start, stop, and restart recurve-agent
supervisorctl start recurve-agent
supervisorctl stop recurve-agent
supervisorctl restart recurve-agent

Upgrade the agent

Keeping the Recurve agent updated is always recommended.

To upgrade the agent , follow these steps:

1. Stop the agent service.

   Depending on the way you manage the service, you may need to run one of these commands to stop the process:

# if managed by Systemd
sudo systemctl stop recurve-agent

# if managed by Supervisor
supervisorctl stop recurve-agent

1. Upgrade and re-login.

# Upgrade to the latest version
pip install -U recurve-agent

recurve login

1. Start the agent.

# if managed by Systemd
sudo systemctl start recurve-agent

# if managed by Supervisor
supervisorctl start recurve-agent