ZenCrepes configuration takes the shape of a file that can be shared (but can also be different) between zqueue, zindexer and zapi. It's full version is available in zindexer/src/components/config/defaultConfig.ts
This part of the documentation will break the configuration by type, for simplicity (and compactness) the documentation will show JSON snippets but the file is composed of its YML translation.
Connecting to Elasticsearch
You can connect to a local Elasticsearch instance or an instance running in Elastic Cloud (by providing your
cloudID and credentials). The details on how the connection is established are available here
ZenCrepes uses system indices to store some of the data it needs to operate.
- sources: Used by
zindexerto determine what data elements to fetch. For example you could scan an entire organization but only enable data fetching for some of its repositories. This index will contain the list of sources and which are enabled or not.
- datasets: Deprecated
- config: Holds ZenCrepes configuration (which datasets are available, which facets to show, how to display and export the list view). This index is used by
One Index per Source
Although this feature is currently available and operational, its use it still a bit uncertain. The original idea was to store data in one index per source, later allowing index level permissions to be configured. Using this feature on an organization with a lot of repositories will result in a lot of indices generated.
These are the indices used to store data from the different datasets. These must be identical between
Use this configuration option to setup the Redis host, used by zqueue to handle its queue mechanism.
This section of configures handling of GitHub.
Zindexer and Zqueue must abide by GitHub rate limits, and performance implication of some of the calls.
- maxNodes: Number of nodes to fetch at a time during bulk fetch operations. For example when fetching issues, the system will by default limit itself to 30 issues at a time. GitHub supports a maximum of 100, but for some of the most complex GraphQL queries (or largest repos), 100 could result in errors. So it's all about finding the sweet spot between number of queries and chances for errors. The sweet spot is usually in the 30-50 region.
- maxParrallel: Used by zqueue to limit the number of parallel threads processing the queue to 1.
- delayBetweenCalls: Should there be a minimum delay between each call to GitHub API ?
storyPointsLabels is an array of label and points, when parsing an issue
zqueue would look for one of these labels and assign the corresponding points to the issue.
The webhook section has three elements to define how the payload should be processed based on received events.
- includeGithubEvents: Allow for specifying which events should be processed by the queue. Using
'*'indicates all events should be processed.
- excludeGithubEvents: Allow for specifying which events should not be processed by the queue.
'*' in includeGithubEvents and
'push' in excludeGithubEvents, means the queue will process all but push events.
Circleci configuration is pretty straight forward, just specify the token you generated from CircleCI
Jira configuration takes an array of Jira servers.