Skip to content
Snippets Groups Projects
Commit 4d9819c9 authored by Sybren A. Stüvel's avatar Sybren A. Stüvel
Browse files

Removed Flamenco Server and Manager, and moved Worker to top level

This is a clone of the Flamenco repository from back in the days when
Server, Manager and Worker shared the same Git repository. This is the
commit where that ended, and they went their separate ways.
parent 51bbd2d7
No related branches found
No related tags found
No related merge requests found
Showing
with 87 additions and 1347 deletions
*.pyc *.pyc
*.bak *.bak
*.blend[1-9] *.blend[1-9]
/temp/
/docs/site/
.cache/ .cache/
.coverage .coverage
*.css.map
/packages/flamenco-manager-go/bin/ /flamenco_worker.egg-info/
/packages/flamenco-manager-go/pkg/ /flamenco-worker.db
/packages/flamenco-manager-go/src/[a-eg-z]* /build/
/packages/flamenco-manager-go/src/flamenco-manager/flamenco-manager /dist/
/packages/flamenco-manager-go/src/flamenco-manager/flamenco-manager.yaml
/packages/flamenco-manager-go/src/flamenco-manager/test_*.json
/packages/flamenco-manager-go/src/flamenco-manager/docker/flamenco-manager
/packages/flamenco-manager-go/src/flamenco-manager/docker/flamenco-manager.yaml
/packages/flamenco-manager-go/src/flamenco-manager/docker/flamenco-manager-install-*.sh
/packages/flamenco-manager-go/src/flamenco-manager/docker/flamenco-manager-*.docker.tgz
/packages/flamenco-worker-python/flamenco_worker.egg-info/
/packages/flamenco-worker-python/flamenco-worker.db
/packages/flamenco-worker-python/build/
/packages/flamenco-worker-python/dist/
# Flamenco 2.0 # Flamenco Worker
Development repo for Flamenco 2.0 (originally known as brender). Flamenco is a Free and Open Source This is the Flamenco Worker implemented in Python 3.
Job distribution system for render farms.
Warning: currently Flamenco is in beta stage, and can still change in major ways.
This project contains the 3 main components of the system: Server, Manager(s) and Worker(s) ## Configuration
## Server Configuration is read from three locations:
Flamenco Server is a [Pillar](https://pillarframework.org) extension, therefore it is implemented
in Python and shares all the existing Pillar requirements. Flamenco Server relies on Pillar to
provide all the authentication, authorization, project and user management functionality.
## Manager - A hard-coded default in the Python source code.
The Manager is written in [Go](https://golang.org/). The Manager is documented in its own - `flamenco-worker.cfg` in the current working directory.
[README](./packages/flamenco-manager-go/README.md). - `$HOME/.flamenco-worker.cfg`.
## Worker When those files do not exist, they are skipped (i.e. this is not an error). They
The Flamenco worker is a very simple standalone component implemented in should be in INI format, as specified by the
[Python](https://www.python.org/). The Worker is documented [configparser documentation](https://docs.python.org/3/library/configparser.html)
in its own [README](./packages/flamenco-worker-python/README.md).
## User and Developer documentation ### Configuration contents:
The documentation is built with [MkDocs](http://www.mkdocs.org/) and uses the
[mkdocs-material](http://squidfunk.github.io/mkdocs-material/) theme, so make sure you have it
installed. You can build the docs by running.
``` All configuration keys should be placed in the `[flamenco-worker]` section of the
pip install mkdocs-material config files.
cd docs
mkdocs serve - `manager_url`: Flamenco Manager URL.
``` - `worker_id`: ID of the worker, handed out by the Manager upon registration (see
Registration below) and used for authentication with the Manager.
- `worker_secret`: Secret key of the worker, given to the Manager upon registration
and authentication.
- `job_types`: Space-separated list of job types this worker may execute.
- `task_update_queue_db`: filename of the SQLite3 database holding the queue of task
updates to be sent to the Master.
### TODO
- Certain settings are currently only settable by editing constants in the Python source code.
It might be nice to read them from the config file too, at some point.
- Update worker address in MongoDB when communicating with it.
## Invocation
Install using `pip install -e .` for development, or `setup.py install` for production.
This creates a command `flamenco-worker`, which can be run with `--help` to obtain
a list of possible CLI arguments.
## Registration
If the configuration file does not contain both a `worker_id` and `worker_secret`, at startup
the worker will attempt to register itself at the Master.
Once registered via a POST to the manager's `/register-worker` endpoint, the `worker_id` and
`worker_secret` will be written to `$HOME/.flamenco-worker.cfg`
## Task fetch & execution
1. A task is obtained by the FlamencoWorker from the manager via a POST to its `/task` endpoint.
If this fails (for example due to a connection error), the worker will retry every few seconds
until a task fetch is succesful.
2. The task is given to a TaskRunner object.
3. The TaskRunner object iterates over the commands and executes them.
4. At any time, the FlamencoWorker can be called upon to register activities and log lines,
and forward them to the Manager. These updates are queued in a SQLite3 database, such that
task execution isn't interrupted when the Manager cannot be reached.
5. A separate coroutine of TaskUpdateQueue fetches updates from the queue, and forwards them to
the Master, using a POST to its `/tasks/{task-id}/update` endpoint.
**TODO:** the response to this endpoint may indicate a request to abort the currently running
task. This should be implemented.
## Shutdown
Pressing [CTRL]+[C] will cause a clean shutdown of the worker.
If there is a task currently running, it will be aborted without changing its status. Any pending task updates are sent to the Manager, and then the Manager's `/sign-off` URL is
POSTed to, to indicate a clean shutdown of the worker. Any active task that is still
assigned to the worker is given status "claimed-by-manager" so that it can be re-activated
by another worker.
## Systemd integration
To run Flamenco Worker as a systemd-managed service, copy `flamenco-worker.service` to
`/etc/systemd/system/flamenco-worker.service`, then run `systemctl daemon-reload`.
After installation of this service, `systemctl {start,stop,status,restart} flamenco-worker`
commands can be used to manage it. To ensure that the Flamenco Worker starts at system boot,
use `systemctl enable flamenco-worker`.
## Signals
Flamenco Worker responds to the following POSIX signals:
- `SIGINT`, `SIGTERM`: performs a clean shutdown, as described in the Shutdown section above.
- `SIGUSR1`: logs the currently scheduled asyncio tasks.
This diff is collapsed.
#!/usr/bin/env python
from __future__ import print_function
"""CLI command for scheduling Blender renders on Flamenco Server.
Assumes that the workers already have access to the required files.
The user_config_dir and user_data_dir functions come from
https://github.com/ActiveState/appdirs/blob/master/appdirs.py and
are licensed under the MIT license.
"""
import argparse
import json
import os.path
import pprint
import sys
import requests
PY2 = sys.version_info[0] == 2
if PY2:
str = unicode
input = raw_input
from urlparse import urljoin
else:
from urllib.parse import urljoin
if sys.platform.startswith('java'):
import platform
os_name = platform.java_ver()[3][0]
if os_name.startswith('Windows'): # "Windows XP", "Windows 7", etc.
system = 'win32'
elif os_name.startswith('Mac'): # "Mac OS X", etc.
system = 'darwin'
else: # "Linux", "SunOS", "FreeBSD", etc.
# Setting this to "linux2" is not ideal, but only Windows or Mac
# are actually checked for and the rest of the module expects
# *sys.platform* style strings.
system = 'linux2'
else:
system = sys.platform
if system == 'win32':
# Win32 is not supported because it requires too much fluff, and we don't use it in the studio.
raise RuntimeError("Sorry, Windows is not supported for now.")
def find_user_id(server_url, auth_token):
"""Returns the current user ID."""
print(15 * '=', 'User info', 15 * '=')
url = urljoin(server_url, '/api/users/me')
resp = requests.get(url, auth=(auth_token, ''))
resp.raise_for_status()
user_info = resp.json()
print('You are logged in as %(full_name)s (%(_id)s)' % user_info)
print()
return user_info['_id']
def find_manager_id(server_url, auth_token):
"""Returns the manager ID.
If the user has more than one Manager available, offers a choice.
"""
print(15 * '=', 'Manager selection', 15 * '=')
url = urljoin(server_url, '/api/flamenco/managers')
resp = requests.get(url, auth=(auth_token, ''))
resp.raise_for_status()
managers = resp.json()['_items']
if not managers:
raise SystemExit('No Flamenco Managers available to your account.')
if len(managers) == 1:
print('One Flamenco Manager is available to you:')
manager = managers[0]
else:
print('Please choose which Flamenco Manager should handle this job:')
for idx, manager in enumerate(managers):
print(' [%i] %s (%s)' % (idx + 1, manager['name'], manager['_id']))
choice = input('Give index, or ENTER for the first one: ')
if choice:
manager = managers[int(choice) - 1]
else:
manager = managers[0]
print('Using manager "%s" (%s)' % (manager['name'], manager['_id']))
print()
return manager['_id']
def find_project_id(server_url, auth_token):
"""Returns the project ID.
If the user has more than one Flamenco-enabled project, offers a choice.
"""
import json
print(15 * '=', 'Project selection', 15 * '=')
url = urljoin(server_url, '/api/projects')
resp = requests.get(url,
params={
'where': json.dumps({'extension_props.flamenco': {'$exists': 1}}),
'projection': json.dumps({'_id': 1, 'name': 1, 'permissions': 1})
},
auth=(auth_token, ''))
resp.raise_for_status()
projects = resp.json()['_items']
if not projects:
raise SystemExit('No Flamenco Projects available to your account.')
if len(projects) == 1:
print('One Flamenco Project is available to you:')
project = projects[0]
else:
print('Please choose which Flamenco Project this job belongs to:')
for idx, project in enumerate(projects):
print(' [%i] %s (%s)' % (idx + 1, project['name'], project['_id']))
choice = input('Give index, or ENTER for the first one: ')
if choice:
project = projects[int(choice) - 1]
else:
project = projects[0]
print('Using project "%s" (%s)' % (project['name'], project['_id']))
print()
return project['_id']
def create_render_job(server_url, auth_token, settings, args):
user_id = find_user_id(server_url, auth_token)
project_id = find_project_id(server_url, auth_token)
manager_id = find_manager_id(server_url, auth_token)
filename = os.path.basename(settings['filepath'])
job = {
u'status': u'queued',
u'priority': 50,
u'name': args.name or u'render %s' % filename,
u'settings': settings,
u'job_type': args.jobtype,
u'user': user_id,
u'manager': manager_id,
u'project': project_id,
}
if args.description:
job[u'description'] = args.description
print()
print('The job:')
json.dump(job, sys.stdout, indent=4, sort_keys=True)
url = urljoin(server_url, '/api/flamenco/jobs')
print()
print('Press [ENTER] to POST to %s' % url)
input()
resp = requests.post(url, json=job, auth=(auth_token, ''))
if resp.status_code == 204:
print('Job created.')
print(resp.headers)
else:
print('Response:')
if resp.headers['content-type'] == 'application/json':
pprint.pprint(resp.json())
else:
print(resp.text)
def find_credentials():
"""Finds BlenderID credentials.
:rtype: str
:returns: the authentication token to use.
"""
import glob
# Find BlenderID profile file.
configpath = user_config_dir('blender', 'Blender Foundation', roaming=True)
found = glob.glob(os.path.join(configpath, '*'))
for confpath in reversed(sorted(found)):
profiles_path = os.path.join(confpath, 'config', 'blender_id', 'profiles.json')
if not os.path.exists(profiles_path):
continue
print('Reading credentials from %s' % profiles_path)
with open(profiles_path) as infile:
profiles = json.load(infile, encoding='utf8')
if profiles:
break
else:
print('Unable to find Blender ID credentials. Log in with the Blender ID add-on in '
'Blender first.')
raise SystemExit()
active_profile = profiles[u'active_profile']
profile = profiles[u'profiles'][active_profile]
print('Logging in as %s' % profile[u'username'])
return profile[u'token']
def main():
parser = argparse.ArgumentParser()
parser.add_argument('blendfile', help='The blendfile to render. It is assumed that a worker '
'can find this file, since we do not yet have upload/'
'download/copy commands.')
parser.add_argument('frames',
help='Frame range to render, in "printer range" notation (1-4,15-30). '
'Should not contain any spaces.')
parser.add_argument('-r', '--render-output',
help='Where to render to, defaults to whatever is defined in the blendfile.')
parser.add_argument('-o', '--format',
help='The file format to output to, defaults to whatever is defined '
'in the blendfile.')
parser.add_argument('-c', '--chunk-size',
help='The maximum number of frames to render on a single worker.')
parser.add_argument('-u', '--server-url', default='https://cloud.blender.org/',
help='URL of the Flamenco server.')
parser.add_argument('-t', '--token',
help='Authentication token to use. If not given, your token from the '
'Blender ID add-on is used.')
parser.add_argument('-b', '--blender-cmd', default='{blender}',
help='Blender command, defaults to "{blender}".')
parser.add_argument('-n', '--name', help='Optional job name, defaults to "render {filename}".')
parser.add_argument('-d', '--description', help='Optional job description.')
parser.add_argument('-p', '--progressive',
help='Progressive render information, '
'in the format "sample_count:num_chunks"')
parser.add_argument('--jobtype', default='blender-render',
help='Sets the job type; set automatically when using --progressive')
args = parser.parse_args()
settings = {
'filepath': args.blendfile,
'frames': args.frames,
'blender_cmd': args.blender_cmd,
}
if args.render_output:
settings['render_output'] = args.render_output
if args.format:
settings['format'] = args.format
if args.chunk_size:
settings['chunk_size'] = int(args.chunk_size)
if args.progressive:
scount, nchunks = args.progressive.split(':')
settings['cycles_sample_count'] = int(scount)
settings['cycles_num_chunks'] = int(nchunks)
args.jobtype = 'blender-render-progressive'
if not args.token:
args.token = find_credentials()
create_render_job(args.server_url, args.token, settings, args)
def user_config_dir(appname=None, appauthor=None, version=None, roaming=False):
r"""Return full path to the user-specific config dir for this application.
"appname" is the name of application.
If None, just the system directory is returned.
"appauthor" (only used on Windows) is the name of the
appauthor or distributing body for this application. Typically
it is the owning company name. This falls back to appname. You may
pass False to disable it.
"version" is an optional version path element to append to the
path. You might want to use this if you want multiple versions
of your app to be able to run independently. If used, this
would typically be "<major>.<minor>".
Only applied when appname is present.
"roaming" (boolean, default False) can be set True to use the Windows
roaming appdata directory. That means that for users on a Windows
network setup for roaming profiles, this user data will be
sync'd on login. See
<http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
for a discussion of issues.
Typical user config directories are:
Mac OS X: same as user_data_dir
Unix: ~/.config/<AppName> # or in $XDG_CONFIG_HOME, if defined
Win *: same as user_data_dir
For Unix, we follow the XDG spec and support $XDG_CONFIG_HOME.
That means, by default "~/.config/<AppName>".
"""
if system in {"win32", "darwin"}:
path = user_data_dir(appname, appauthor, None, roaming)
else:
path = os.getenv('XDG_CONFIG_HOME', os.path.expanduser("~/.config"))
if appname:
path = os.path.join(path, appname)
if appname and version:
path = os.path.join(path, version)
return path
def user_data_dir(appname=None, appauthor=None, version=None, roaming=False):
r"""Return full path to the user-specific data dir for this application.
"appname" is the name of application.
If None, just the system directory is returned.
"appauthor" (only used on Windows) is the name of the
appauthor or distributing body for this application. Typically
it is the owning company name. This falls back to appname. You may
pass False to disable it.
"version" is an optional version path element to append to the
path. You might want to use this if you want multiple versions
of your app to be able to run independently. If used, this
would typically be "<major>.<minor>".
Only applied when appname is present.
"roaming" (boolean, default False) can be set True to use the Windows
roaming appdata directory. That means that for users on a Windows
network setup for roaming profiles, this user data will be
sync'd on login. See
<http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
for a discussion of issues.
Typical user data directories are:
Mac OS X: ~/Library/Application Support/<AppName>
Unix: ~/.local/share/<AppName> # or in $XDG_DATA_HOME, if defined
Win XP (not roaming): C:\Documents and Settings\<username>\Application Data\<AppAuthor>\<AppName>
Win XP (roaming): C:\Documents and Settings\<username>\Local Settings\Application Data\<AppAuthor>\<AppName>
Win 7 (not roaming): C:\Users\<username>\AppData\Local\<AppAuthor>\<AppName>
Win 7 (roaming): C:\Users\<username>\AppData\Roaming\<AppAuthor>\<AppName>
For Unix, we follow the XDG spec and support $XDG_DATA_HOME.
That means, by default "~/.local/share/<AppName>".
"""
if system == "win32":
raise RuntimeError("Sorry, Windows is not supported for now.")
elif system == 'darwin':
path = os.path.expanduser('~/Library/Application Support/')
if appname:
path = os.path.join(path, appname)
else:
path = os.getenv('XDG_DATA_HOME', os.path.expanduser("~/.local/share"))
if appname:
path = os.path.join(path, appname)
if appname and version:
path = os.path.join(path, version)
return path
if __name__ == '__main__':
main()
#!/bin/bash
cd docs
command -v mkdocs 2>/dev/null 2>&1 || { echo >&2 "Command mkdocs not found. Are you in the right venv?"; exit 1; }
mkdocs build
rsync -auv ./site/* armadillica@flamenco.io:/home/armadillica/flamenco.io/docs
# System architecture
The architecture of Flamenco is simple, hierarchical.
In a production setup we have one Server and one or more Managers which control one or
more Workers each.
![Architecture diagram](img/architecture_diagram.svg)
With this configuration it is possible to have a very generic and simple API on the Server, and
develop/maintain different type of front-ends for it.
Communication between components happens via HTTP. In particular, Server and Manager use a simple
and well-defined REST API. This allows third parties to easily implement their Manager, and
integrate it into Flamenco. Flamenco ships with its own Free and Open Source Manager and Worker
implementations.
The whole system is designed with bottom-up communication in mind. For example:
- Worker has a loop that sends requests to the Manager
- Worker sends request a task to the Manager
- Manager checks Tasks availability with Server
- Server replies to Manager with available Tasks
- Manager replies to Worker with a Task to execute
- Worker executes the Commands of a Task and reports progress to the Manager
By using update buffers on Managers and Worker components we can keep the overall
infrastructure as resilient, responsive and available a possible.
## Server
In a Flamenco network, there can be only one server. The functionality of the server consists in:
- storing a list of Managers
- storing Jobs
- generating and storing Tasks (starting from a Job)
- dispatch Tasks to a Manager
- serving entry points to inspect the status of:
+ Jobs
+ Tasks
+ Workers
The Server software is based on [Pillar](https://pillarframework.org/), the Free and Open Source
CMS that provides agnostic user authentication, resource and project management. It requires:
- Linux, macOS or Windows
- Python 2.7 (soon to become Python 3)
- MongoDB
## Manager
The goal of the Manager, as the name suggests, is to handle the workload provided by the Server
and leverage the computing resources available (Workers) to complete the Tasks as soon as possible.
Because the communication between Server and Manager happens via a semi-RESTful API, a Manager
can be implemented in many ways. At Blender we implemented a Manager in Go, which is available
as Free and Open Source software. It requires:
- Linux, macOS or Windows
- Golang
- MongoDB
## Worker
The lowest-level component of the Flamenco infrastructure, a Worker is directly responsible for
the execution of a Task, which is composed by an ordered series of Commands. Similarly to the
Manager, Blender provides a Free and Open Source implementation, with the following requirements:
- Linux, macOS or Windows
- Python 3.5 or greater
## Jobs, Tasks and Commands
Flamenco is designed to handle several types of jobs, mostly serving computer animated film
production, for example:
- 3D animation rendering
- simulation baking
- distributed still image rendering
- video encoding
A Job is the highest level structure, containing all the necessary information on how to process
the Job itself.
In order to distribute the execution of a Job, we split it into Tasks, according to the instructions
provided. This process is called *Job compilation*.
A task is essentially a list of Commands. Each worker can claim one (or more tasks) to execute,
which means it will sequentially run all the Commands contained in it.
- keeping a log of operations related to Jobs (task logging happens on the manager)
- collecting and storing all the data needed to complete a Job
## Components and processes documentation
In the rest of the documentation we cover more in depth the different components of the
architecture, as well as the various processes (Job creation, Manager and Worker management, etc.).
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="851px" height="371px" version="1.1" content="&lt;mxfile userAgent=&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36&quot; version=&quot;6.0.3.3&quot; editor=&quot;www.draw.io&quot; type=&quot;google&quot;&gt;&lt;diagram name=&quot;Page-1&quot;&gt;7VnLcpswFP0aZtpFPAgBdpaNm6aLZKYznjbNUgEFqGWuRwg/+vUVRrwscEmaYvrwxuhwka7OOb4Xg4Hnq90NJ+vwDnzKDMv0dwZ+b1gWstBUfmXIPkdmjpkDAY98FVQBi+g7VWARlkY+TRqBAoCJaN0EPYhj6okGRjiHbTPsCVhz1TUJqAYsPMJ09D7yRXi0iwz/SKMgLFZGpjrzSLxlwCGN1XqGhZ8On/z0ihRzqfgkJD5saxC+NvCcA4j8aLWbU5ZxW9CWX/eh4+zUoo9T7BD86PvoyUIXWXC2zoawVO3tcywiwagv0Vuyp1wlIvbF5hl4S5pNiAx8VWyY01i8LANby0BbcRtGgi7WxMvGW2kpuXAoVkzlkAgOSzoHBlwiMcS0TGxDuaC7I9GqXDv4UKnfUFhRwffyIjXFpZJF2RbZarxtMUFYMwCeKpAo4wXl1BVX8kDR1U5dS/7P5Br/nGrqS6OrIXARQgAxYdcVenVwb6l/TYZvVIi9+q2SVICEqhluAdYqju4i8VUemxNHjR4aZspS6NyxghJIuUc7tllUBMID2il0u86cMiKiTXP9NtEOl77jnOxrAWuIYpHUZv6UAZV9bMtp+MdyjwQ8iseueSpeHuQZVPYpt9LLURor9gv8UHNAt7K9vRHL5GtTZMNsDlMz3oBecV7DK7oZnHYzlFPkOamrKtV/t+kcc2DTIfc1q1C3B095q68/B3Qd+lXb9e0rXdnV2sSC8k3LnUBThGd1aXnjYx4+fVs1OtmYsds0OZrpnRlZLZ3ZfoXGrBE4HcbQA5vW6mFa91yevfznakgfOWbnkkP/W3FHYvmvSNYQ800UX4SQJvTtmAvKURcsx2cpKAhpVP0NFcXp0wbNdp0GaIMvuRv+s2tKL0HwuQSx/wvSKsjZqrxzqsrPGaT+mEu8bY+pxLsal1eM+HTE/JXcKP5sdE7+Zhp/98CXiZA/AYhHzKLljonFYvEajV/uRsyeg0bFnv6AddTsuc6o2NPbyWQyGTF9zmxMDQTpFXDc9E0vh3OfHFYvzfKHltWbSXz9Aw==&lt;/diagram&gt;&lt;/mxfile&gt;" style="background-color: rgb(255, 255, 255);"><defs/><g transform="translate(0.5,0.5)"><rect x="0" y="0" width="850" height="370" fill="#ffffff" stroke="none" pointer-events="none"/><path d="M 335 80 L 335 110 Q 335 120 325 120 L 280 120 Q 270 120 270 130 L 270 153.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 270 158.88 L 266.5 151.88 L 270 153.63 L 273.5 151.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 335 80 L 335 110 Q 335 120 345 120 L 400 120 Q 410 120 410 130 L 410 153.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 410 158.88 L 406.5 151.88 L 410 153.63 L 413.5 151.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 335 80 L 335 110 Q 335 120 345 120 L 540 120 Q 550 120 550 130 L 550 153.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 550 158.88 L 546.5 151.88 L 550 153.63 L 553.5 151.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><rect x="275" y="40" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(316.5,53.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="36" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 36px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Server</div></div></foreignObject><text x="18" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">Server</text></switch></g><path d="M 270 200 L 270 225 Q 270 235 260 235 L 100 235 Q 90 235 90 245 L 90 263.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 90 268.88 L 86.5 261.88 L 90 263.63 L 93.5 261.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 270 200 L 270 225 Q 270 235 260 235 L 240 235 Q 230 235 230 245 L 230 263.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 230 268.88 L 226.5 261.88 L 230 263.63 L 233.5 261.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><rect x="210" y="160" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(217.5,173.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="104" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 106px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Manager (in-house)</div></div></foreignObject><text x="52" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">Manager (in-house)</text></switch></g><path d="M 410 200 L 410 225 Q 410 235 420 235 L 470 235 Q 480 235 480 245 L 480 263.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 480 268.88 L 476.5 261.88 L 480 263.63 L 483.5 261.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 410 200 L 410 225 Q 410 235 420 235 L 610 235 Q 620 235 620 245 L 620 263.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 620 268.88 L 616.5 261.88 L 620 263.63 L 623.5 261.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 410 200 L 410 225 Q 410 235 420 235 L 750 235 Q 760 235 760 245 L 760 263.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 760 268.88 L 756.5 261.88 L 760 263.63 L 763.5 261.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><rect x="350" y="160" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(364.5,173.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="90" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 92px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Manager (Cloud)</div></div></foreignObject><text x="45" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">Manager (Cloud)</text></switch></g><rect x="30" y="270" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(74.5,283.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="30" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 32px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Blade</div></div></foreignObject><text x="15" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">Blade</text></switch></g><rect x="170" y="270" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(197.5,283.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="64" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 64px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Workstation</div></div></foreignObject><text x="32" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">Workstation</text></switch></g><rect x="420" y="270" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(470.5,283.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="18" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 19px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">VM</div></div></foreignObject><text x="9" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">VM</text></switch></g><rect x="560" y="270" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(610.5,283.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="18" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 19px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">VM</div></div></foreignObject><text x="9" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">VM</text></switch></g><rect x="490" y="160" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(544.5,173.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="10" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 12px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">...</div></div></foreignObject><text x="5" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">...</text></switch></g><rect x="700" y="270" width="120" height="40" rx="6" ry="6" fill="#ffffff" stroke="#000000" pointer-events="none"/><g transform="translate(754.5,283.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="10" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 12px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">...</div></div></foreignObject><text x="5" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">...</text></switch></g></g></svg>
\ No newline at end of file
This diff is collapsed.
# Roadmap
The day-to-day planning for development is available on
[developer.blender.org](https://developer.blender.org/project/board/58/). In this section we summarize
the high level goals for the projects.
## Self-provisionable Server
Make it possible for developers to run the full stack (Server, Manager and Worker) in a local
environment. While this is easy to do for Manager and Worker, the challenge is to get the
Server (and its Pillar core) disconnected from Blender Cloud.
## Pre-built packages
Provide pre-built and easily installable components for regular user. Packaging will vary depending
on the component:
- Server: Python package (pip installable)
- Manager: Standalone platform-dependent binary
- Worker: Python package (possibly standalone)
## Complete Server API docs
To make it easy to implement own manager.
## More features
Resumable jobs, job callbacks, resource allocation, filtering.
\ No newline at end of file
# Scheduling
The scheduling system is supposed to hand out Tasks from the Server to each Worker, through a
Manager. By design, the communication between Server and Workers is *always* mediated by the
Manager, to allow completely customizeable resource management on the computing infrastructure
available.
At the core of the scheduling system lays the *Dependency Graph*. The Dependency Graph is a DAG
(Directed Acyclic Graph) where each node represents a Task. Tasks are initially stored in the
Server database, in a dedicated collection and are passed on to the Manager upon request.
The DG is generated with a database query to the Tasks collection and, depending on the query,
can return hundred-thousands of Tasks, which are then stored by the Manager in its own
database, so that they can be served to the Workers.
[![Architecture diagram](img/scheduling_diagram.svg)](img/scheduling_diagram.svg)
## Priority rules
The priority for the execution of a Task is determined by three factors:
- position in the DG
- job priority
- task priority
Therefore, the Task with no parent Task (or all its parent Tasks completed), with the highest
Job priority, with the highest Task priority will be dispatched first.
## Task requirements and resource allocation
**Note: This feature is not available yet.**
When a Worker queries the Manager for a Task, we use the *services* offered by it as a query
parameter to find the highest priority Task that can be executed. For example, a Worker
might offer `blender_render`, but not `ffmpeg`. This also extends to hardware settings,
so that we can specify a minimum amount of RAM or CPU cores required by a Task.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="512"
height="512"
id="svg2"
version="1.1"
inkscape:version="0.48.4 r9939"
sodipodi:docname="logo_flamenco.svg"
inkscape:export-filename="/shared/software/attract3/logo/attract_logo_bw_1080.png"
inkscape:export-xdpi="90"
inkscape:export-ydpi="90">
<defs
id="defs4" />
<sodipodi:namedview
id="base"
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageopacity="0.0"
inkscape:pageshadow="2"
inkscape:zoom="0.49497475"
inkscape:cx="153.60622"
inkscape:cy="196.16365"
inkscape:document-units="px"
inkscape:current-layer="text2984"
showgrid="false"
fit-margin-top="10"
fit-margin-left="10"
fit-margin-right="10"
fit-margin-bottom="10"
inkscape:showpageshadow="false"
inkscape:window-width="1281"
inkscape:window-height="708"
inkscape:window-x="327"
inkscape:window-y="612"
inkscape:window-maximized="0" />
<metadata
id="metadata7">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(-81.656231,-11.7557)">
<g
id="g2995"
transform="matrix(0.66925035,0,0,0.66925035,371.3471,-72.927844)"
style="fill:black;fill-opacity:1;stroke:none;fill-rule:nonzero">
<g
style="font-size:708.35473633px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:black;font-family:Open Sans;-inkscape-font-specification:Open Sans"
id="text2984">
<path
d="m -48.041991,633.10581 c -5.667221,-0.47221 -11.570172,-2.36115 -17.708869,-5.66684 -6.139439,-3.77785 -11.334035,-7.79186 -15.583804,-12.04203 -5.194952,-6.61126 -8.97284,-14.40315 -11.333676,-23.37571 -2.361524,-8.97241 -2.597642,-17.70878 -0.708354,-26.20912 2.360836,-9.91686 7.319314,-18.65323 14.875449,-26.20913 8.027654,-8.02788 17.000138,-12.98636 26.91748,-14.87545 3.777503,-0.94433 8.972099,-1.18045 15.583804,-0.70835 7.08314,1.4e-4 12.513854,0.70849 16.292159,2.12506 8.499831,3.30579 15.5833716,8.26427 21.2506423,14.87545 6.1386298,6.13919 10.3887537,13.69497 12.7503857,22.66735 1.416256,4.25023 1.888492,9.91706 1.416709,17.00052 -4.53e-4,7.08362 -0.708807,12.51434 -2.125064,16.29216 -2.361631,6.61137 -6.1395189,12.75044 -11.3336759,18.41722 -4.7227994,5.19464 -10.3896316,9.44477 -17.0005141,12.75038 -10.86185,5.19464 -21.959397,6.84746 -33.292672,4.95849 M -386.63556,599.81313 c -0.47228,-0.9444 -0.23616,-11.09748 0.70836,-30.45925 -0.94452,23.6119 0.23607,-10.86133 3.54177,-103.41979 -5e-5,-6.13887 0.23607,-17.47253 0.70836,-34.00103 l 1.41671,-41.08457 c 0.94441,-29.7506 2.36112,-51.70957 4.25013,-65.87699 1.88888,-14.63898 4.7223,-25.26429 8.50025,-31.87597 1.41664,-2.83304 5.66677,-9.44434 12.75039,-19.83393 7.55569,-10.3888 14.167,-19.12517 19.83393,-26.20912 7.55567,-10.38878 15.11145,-19.59738 22.66735,-27.62584 8.02789,-8.02757 19.83379,-18.41676 35.41774,-31.16761 12.75021,-10.86095 21.48658,-17.70837 26.20912,-20.54229 4.72218,-3.30515 9.91677,-5.43021 15.58381,-6.37519 14.16688,-2.83291 40.37598,3.07004 78.62737,17.70887 38.72308,14.6398 92.321865,38.95996 160.7965293,72.96054 73.1961417,37.30706 131.9895307,69.41911 176.3803307,96.33624 13.69423,8.50057 23.13895,15.11187 28.33419,19.83394 5.66618,4.72265 9.91631,9.91725 12.75038,15.5838 0.94382,2.36146 1.41605,4.25041 1.41671,5.66684 0.47158,0.94475 0.47158,2.36145 0,4.25013 l -2.12506,5.66683 c -1.41737,2.36145 -9.6815,11.57005 -24.79242,27.62584 -15.11218,15.58403 -27.62644,27.86216 -37.5428,36.83444 -12.27873,11.33387 -20.77898,18.65353 -25.50077,21.959 -4.25069,3.30583 -8.73694,4.95866 -13.45874,4.95848 -6.61186,1.8e-4 -13.93152,-1.65265 -21.958996,-4.95848 -7.556311,-3.30547 -20.778919,-10.62513 -39.667865,-21.959 -39.66832,-24.08383 -89.253102,-51.00128 -148.754494,-80.75244 -41.557115,-21.25033 -80.516585,-39.90366 -116.878535,-55.96002 -35.41793,-16.05569 -53.3629,-22.90311 -53.83496,-20.54229 l -3.54177,148.7545 c -0.47241,12.27834 -1.41688,20.77859 -2.83342,25.50077 -0.4724,2.36136 -2.36135,6.61149 -5.66684,12.75038 -9.91712,20.77856 -25.97314,42.2653 -48.16812,64.46028 -13.22272,12.75048 -25.73698,23.3758 -37.5428,31.87597 -11.80598,8.50032 -20.07011,12.75045 -24.79242,12.75038 -1.41676,7e-5 -2.36123,-0.94441 -2.83342,-2.83342"
style="fill:white;fill-opacity:1"
id="path2986" />
</g>
</g>
</g>
</svg>
# Flamenco Docs
Welcome to the Flamenco Documentation pages! Here we collect both development and user docs.
Flamenco is primarily a distributed render management solution, but it is designed to
support any kind of job that can be split across multiple machines.
Currently Flamenco is used for Render Management at Blender Institute, so it's heavily
tested for Blender jobs, but it is quite trivial to add more.
## Main features
* Every component runs on Linux, macOS and Windows
* Automatic detection of clients
* Powerful user and project management tools
* Dependency-graph-based priority system
* Resilient design (every individual component can fail without consequences)
* Extensible Jobs, Tasks and Commands
## Status of the documentation
We are currently updating the documentation after a large rewrite of Flamenco, therefore
the user manual is not available, and the developer docs are work in progress. We have
removed any legacy content though, so everything that can be found here is up to date,
or has been highlighted as upcoming functionality.
# Installation
Flamenco is still under development, therefore this installation guide is rather technical.
In the future it will be a straightforward process, in two variants.
## Using the cloud.blender.org Server
**Note: This feature is not available yet.**
In this case, only two components of the stack are controlled by the user: Manager and
Workers. The setup:
- On Blender Cloud: create and initialize a Project to be used with Flamenco
- On Blender Cloud: Create a Manager (and collect the identification token)
- Download the Manager binary, add the identification token to the configuration, start
the Manager (it will automatically connect with Blender Cloud)
- Download the Worker bundle and start it (it will automatically connect with the Manager)
## Using a self-provisioned Server
**Note: This feature is not available yet.**
- Download and start the server (using Docker)
- Follow the steps for setting up with Blender Cloud, updating the configuration to point
to the self-provisioned Server, rather than cloud.blender.org
# Project information
site_name: 'Flamenco'
site_description: 'Free and Open Source render management'
site_author: 'The Flamenco Team'
site_url: 'https://flamenco.io/'
# Repository
repo_name: 'armadillica/flamenco'
repo_url: 'https://github.com/armadillica/flamenco'
# Copyright
copyright: 'Copyright &copy; 2016 Blender Institute - CC-BY-SA v4.0.'
theme: 'material'
# Options
extra:
logo: 'img/logo_flamenco_white.svg'
palette:
primary: 'red'
accent: 'red'
social:
- type: 'github'
link: 'https://github.com/armadillica'
- type: 'twitter'
link: 'https://twitter.com/flamenco_io'
# Google Analytics
google_analytics:
- 'UA-1418081-11'
- 'auto'
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment