eris

- Elasticsearch Recon Ingestion Scripts (ERIS) 🔎
git clone git://git.acid.vegas/eris.git
README.md (5579B)
      1 # Elasticsearch Recon Ingestion Scripts (ERIS)
      2 > A utility for ingesting various large scale reconnaissance data logs into Elasticsearch
      3 
      4 The is a suite of tools to aid in the ingestion of recon data from various sources *(httpx, masscan, zonefiles, etc)* into an [Elasticsearch](https://www.elastic.co/elasticsearch) cluster. The entire codebase is designed with asynconous processing, aswell as load balancing ingestion across all of the nodes in your cluster. Additionally, live data ingestion is supported from many of the sources supported. This means data can be directly processed and ingested into your Elasticsearch cluster instantly. The structure allows for the developement of "modules" or "plugins" if you will, to quickly create custom ingestion helpers for anything!
      5 
      6 ## Prerequisites
      7 - [python](https://www.python.org/)
      8     - [elasticsearch](https://pypi.org/project/elasticsearch/) *(`pip install elasticsearch`)*
      9     - [ecs-logging](https://pypi.org/project/ecs-logging) *(`pip install ecs-logging`)*
     10     - [aiofiles](https://pypi.org/project/aiofiles) *(`pip install aiofiles`)*
     11     - [aiohttp](https://pypi.org/projects/aiohttp) *(`pip install aiohttp`)*
     12     - [websockets](https://pypi.org/project/websockets/) *(`pip install websockets`) (only required for `--certs` ingestion)*
     13 
     14 ## Usage
     15 ```shell
     16 python eris.py [options] <input>
     17 ```
     18 **Note:** The `<input>` can be a file or a directory of files, depending on the ingestion script.
     19 
     20 ### Options
     21 ###### General arguments
     22 | Argument     | Description                                                      |
     23 |--------------|------------------------------------------------------------------|
     24 | `input_path` | Path to the input file or directory                              |
     25 | `--watch`    | Create or watch a FIFO for real-time indexing                    |
     26 | `--log`      | Logging level for file *(debug, info, warning, error, critical)* |
     27 
     28 **Note:** File logging is disabled by default. When enabled, it will log using the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs-logging/python/current/intro.html) *(ECS)*.
     29 
     30 ###### Elasticsearch arguments
     31 | Argument        | Description                                             | Default            |
     32 |-----------------|---------------------------------------------------------|--------------------|
     33 | `--host`        | Elasticsearch host                                      | `http://localhost` |
     34 | `--port`        | Elasticsearch port                                      | `9200`             |
     35 | `--user`        | Elasticsearch username                                  | `elastic`          |
     36 | `--password`    | Elasticsearch password                                  | `$ES_PASSWORD`     |
     37 | `--api-key`     | Elasticsearch API Key for authentication                | `$ES_APIKEY`       |
     38 | `--self-signed` | Elasticsearch connection with a self-signed certificate |                    |
     39 
     40 ###### Elasticsearch indexing arguments
     41 | Argument     | Description                          | Default             |
     42 |--------------|--------------------------------------|---------------------|
     43 | `--index`    | Elasticsearch index name             | Depends on ingestor |
     44 | `--pipeline` | Use an ingest pipeline for the index |                     |
     45 | `--replicas` | Number of replicas for the index     | `1`                 |
     46 | `--shards`   | Number of shards for the index       | `1`                 |
     47 
     48 ###### Performance arguments
     49 | Argument       | Description                                              | Default |
     50 |----------------|----------------------------------------------------------|---------|
     51 | `--chunk-max`  | Maximum size in MB of a chunk                            | `100`   |
     52 | `--chunk-size` | Number of records to index in a chunk                    | `50000` |
     53 | `--retries`    | Number of times to retry indexing a chunk before failing | `100`   |
     54 | `--timeout`    | Number of seconds to wait before retrying a chunk        | `60`    |
     55 
     56 ###### Ingestion arguments
     57 | Argument      | Description              |
     58 |---------------|--------------------------|
     59 | `--certstrem` | Index Certstream records |
     60 | `--httpx`     | Index HTTPX records      |
     61 | `--masscan`   | Index Masscan records    |
     62 | `--massdns`   | Index massdns records    |
     63 | `--zone`      | Index zone DNS records   |
     64 
     65 ~~This ingestion suite will use the built in node sniffer, so by connecting to a single node, you can load balance across the entire cluster.~~
     66 
     67 **Note:** The sniffer has been disabled for now due an [issue](https://github.com/elastic/elasticsearch-py/issues/2005#issuecomment-1645641960) with the 8.x elasticsearch client. The auth headers are not properly sent when enabling the sniffer. A working [patch](https://github.com/elastic/elasticsearch-py/issues/2005#issuecomment-1645641960) was shared and has been *mostly* converted in [helpers/sniff_patch.py](./helpers/sniff_patch.py) for the async client.
     68 
     69 ## Roadmap
     70 - Create a module for RIR database ingestion *(WHOIS, delegations, transfer, ASN mapping, peering, etc)*
     71 - Dynamically update the batch metrics when the sniffer adds or removes nodes.
     72 - Fix issue with leftover FIFO files *(catch SIGTERM / SIGINT signals)*
     73 - Create a working patch for the async client to properly send auth headers.
     74 
     75 ___
     76 
     77 ###### Mirrors for this repository: [acid.vegas](https://git.acid.vegas/eris) • [SuperNETs](https://git.supernets.org/acidvegas/eris) • [GitHub](https://github.com/acidvegas/eris) • [GitLab](https://gitlab.com/acidvegas/eris) • [Codeberg](https://codeberg.org/acidvegas/eris)