eris

- Elasticsearch Recon Ingestion Scripts (ERIS) 🔎
git clone git://git.acid.vegas/-c.git
Log | Files | Refs | Archive | README | LICENSE

DateCommit messageFiles+-
2024-03-24ECS formatting added to eris, certstream ingestor now caches to prevent duplication, elastic connections properly closed now5+60-21
2024-03-23Fixed _doc to doc in record1+7-7
2024-03-23Added more comments and thoughts1+2-1
2024-03-23Added a comment about the methodology of determining the zone the data is from.1+1-0
2024-03-23Updated example record comment under source to reflect on the new record structure1+4-1
2024-03-23AXFR detection in zone file input, to determine if the data is form ICANN or an AXFR against a dns (for source tracking)1+10-2
2024-03-23Added zone field too better querying on specific tlds/zones1+5-0
2024-03-23Records stored as a nested type for better querying, added source field so we can identify where the zone data derived (icann, axfr, breach, etc)1+9-6
2024-03-19Added IXP ingestor, updated all other ingestors7+476-326
2024-03-15Typo in pypi title1+1-1
2024-03-15Added elastic common schema (ecs) logging to file for ingesting eris logs straight into ES2+61-54
2024-03-14Certstream ingestor now only logs sub-domains since we already ingested zone files. Ignores www. and wildcard domains.1+14-4
2024-03-14add ingest_rir_transfers.py1+7-13
2024-03-14RIR Transfers ingestor added1+193-0
2024-03-13Thats how you fucking do it. Delegations stored in ranges now for querying.1+34-22
2024-03-12Fixed import error saying aiofiles instead of aiohttp1+2-2
2024-03-12Added anomaly detection to RIR delegations ingestor2+58-41
2024-03-12Created an ingestor for RIR delegations1+167-0
2024-03-12Lowered reconnect time on certstream timeout1+1-1
2024-03-12Renamed default_index to eris-certstream1+1-1
2024-03-12Fixed logger again2+12-9
2024-03-12Fixed missing logging.handlers import1+1-0
2024-03-12Fixed logger level1+1-1
2024-03-12Removed _id from certstream yield and renamed ingest_certs.py to ingest_certstream.py6+492-465
2024-03-12Code cleanup3+397-468
2024-03-11Full source commenting, uniformity in testing function, records stored as a list by default incase an IP address yields multiple PTR records1+75-71
2024-03-08Many bugs fixed in sniffer and async model.3+52-49
2024-03-08Overall code cleanup4+50-45
2024-03-08Fixed issue with ingest_certs and the ingestion function signature. Simple placeholder argument (un-used) added to maintain function uniformity1+1-3
2024-03-08Testing function added to every ingestor to debug directly. No more --dry-run needed.6+274-194
2024-03-08Changed yield to return when sentinal value is recieved through the FIFO1+1-1
2024-03-08Added a testing function for calling the ingest_massdns.py file directly for debugging1+47-26
2024-03-08MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7=1+0-4
2024-03-08MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7=12+133-917
2024-03-06OCD about formatting again4+83-60
2024-03-06Certstream ingestion improved, still need to test stripping nulls and refactor the index mapping so we only store whats needed1+6-4
2024-03-06Updated massdns ingestion script with sentinal value checking and using the ip address as the document id1+32-25
2024-03-06Added a note about NSEC3 hash cracking for zone file data1+5-0
2024-03-06Zone file ingestion script now uses the same sentinal value as masscans ingestion, set document id as the domain name to allow updating records if they exist1+27-15
2024-03-06Masscan ingestion script updated to use ip:port as the document id to allow updating records that already exist. Added a sentinal value to trigger an EOF when using --watch with FIFO's1+60-57
2024-03-06Updated cause I am OCD about spaces and formatting1+6-6
2024-03-06Introduction paragraph descriving the project overview added, updated roadmap1+5-3
2024-03-06Asyncronous refactorization pushed as main version 💯20+155-997
2024-03-06Added a recursive JSON explorer to remove empty or nulled keys from a dictionary2+31-2
2024-03-06Asyncronous refactorization of the codebase is complete, testing & metrics and then it will be production ready6+381-43
2024-03-05Asyncronous developed mon ERIS is complete, need to refactor ingestion helpers before pushing this as the main version5+209-222
2024-03-05Added certificate transparency monitoring via websockets for ingestion4+3-1
2024-03-05Added certificate transparency monitoring via websockets for ingestion4+191-3
2024-03-04Started asyncronous implementation of bulk streaming data, altered ERIS defaults, etc11+626-200
2024-02-02Added an asyncronous sniffer patch in helpers1+96-0
2024-02-02Ingestion agents are now modular, FIFO live ingestion added12+1059-1506
2024-01-27Updated README, copied over consistencies across the ingestors, docstring updates to reflect on new arguments6+455-187
2024-01-27Added parallel bulk uploading, error handling, sniffing nodes for discovery, dynamic batch sizes, and more1+0-0
2024-01-27Added parallel bulk uploading, error handling, sniffing nodes for discovery, dynamic batch sizes, and more7+527-163
2024-01-21Updated mirrors2+3-4
2024-01-20Updated README, fixed issue using the wrong domain in records for zone file ingestion (woops)6+75-21
2024-01-20Initial commit6+968-0