diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ad9ec5..e95b7cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,22 @@ # Change Log +## [v1.6](https://github.com/Boerderij/Varken/tree/v1.6) (2019-01-04) +[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.5...v1.6) + +**Implemented enhancements:** + +- \[Feature Request\] docker-compose stack install option [\#84](https://github.com/Boerderij/Varken/issues/84) +- Fix missing variables in varken.ini automatically [\#81](https://github.com/Boerderij/Varken/issues/81) +- Create Wiki for FAQ and help docs [\#80](https://github.com/Boerderij/Varken/issues/80) + +**Fixed bugs:** + +- \[BUG\] url:port does not filter [\#82](https://github.com/Boerderij/Varken/issues/82) + +**Merged pull requests:** + +- v1.6 Merge [\#75](https://github.com/Boerderij/Varken/pull/85) ([DirtyCajunRice](https://github.com/DirtyCajunRice)) + ## [v1.5](https://github.com/Boerderij/Varken/tree/v1.5) (2018-12-30) [Full Changelog](https://github.com/Boerderij/Varken/compare/v1.4...v1.5) @@ -20,43 +37,28 @@ - Add Ombi Issues [\#74](https://github.com/Boerderij/Varken/pull/74) ([anderssonoscar0](https://github.com/anderssonoscar0)) ## [v1.4](https://github.com/Boerderij/Varken/tree/v1.4) (2018-12-19) -[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.3-nightly...v1.4) +[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.1...v1.4) **Implemented enhancements:** - \[Feature Request\] Add tautulli request for library stats [\#64](https://github.com/Boerderij/Varken/issues/64) +- Create randomized 12-24 hour checks to update GeoLite DB after the first wednesday of the month [\#60](https://github.com/Boerderij/Varken/issues/60) +- \[Feature Request\]: Pull list of requests \(instead of just counts\) [\#58](https://github.com/Boerderij/Varken/issues/58) +- Feature Request , Add Sickchill [\#48](https://github.com/Boerderij/Varken/issues/48) **Fixed bugs:** - \[BUG\] Ombi all requests missing half of "pending" option [\#63](https://github.com/Boerderij/Varken/issues/63) - \[BUG\] asa bug with checking for apikey [\#62](https://github.com/Boerderij/Varken/issues/62) - -**Merged pull requests:** - -- v1.4 Merge [\#65](https://github.com/Boerderij/Varken/pull/65) ([DirtyCajunRice](https://github.com/DirtyCajunRice)) - -## [v1.3-nightly](https://github.com/Boerderij/Varken/tree/v1.3-nightly) (2018-12-18) -[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.2-nightly...v1.3-nightly) - -**Implemented enhancements:** - -- Create randomized 12-24 hour checks to update GeoLite DB after the first wednesday of the month [\#60](https://github.com/Boerderij/Varken/issues/60) - -**Fixed bugs:** - - \[BUG\] Add Catchall to ombi requests [\#59](https://github.com/Boerderij/Varken/issues/59) **Closed issues:** - Unify naming and cleanup duplication in iniparser [\#61](https://github.com/Boerderij/Varken/issues/61) -## [v1.2-nightly](https://github.com/Boerderij/Varken/tree/v1.2-nightly) (2018-12-16) -[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.1...v1.2-nightly) +**Merged pull requests:** -**Implemented enhancements:** - -- \[Feature Request\]: Pull list of requests \(instead of just counts\) [\#58](https://github.com/Boerderij/Varken/issues/58) -- Feature Request , Add Sickchill [\#48](https://github.com/Boerderij/Varken/issues/48) +- v1.4 Merge [\#65](https://github.com/Boerderij/Varken/pull/65) ([DirtyCajunRice](https://github.com/DirtyCajunRice)) ## [v1.1](https://github.com/Boerderij/Varken/tree/v1.1) (2018-12-11) [Full Changelog](https://github.com/Boerderij/Varken/compare/v1.0...v1.1) @@ -84,22 +86,12 @@ - Update issue templates [\#55](https://github.com/Boerderij/Varken/pull/55) ([DirtyCajunRice](https://github.com/DirtyCajunRice)) ## [v1.0](https://github.com/Boerderij/Varken/tree/v1.0) (2018-12-10) -[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.3-nightly...v1.0) +[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.1...v1.0) **Implemented enhancements:** - Add cisco asa from legacy [\#44](https://github.com/Boerderij/Varken/issues/44) - Add server ID to ombi to differenciate [\#43](https://github.com/Boerderij/Varken/issues/43) - -**Merged pull requests:** - -- v1.0 Merge [\#45](https://github.com/Boerderij/Varken/pull/45) ([DirtyCajunRice](https://github.com/DirtyCajunRice)) - -## [v0.3-nightly](https://github.com/Boerderij/Varken/tree/v0.3-nightly) (2018-12-07) -[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.2-nightly...v0.3-nightly) - -**Implemented enhancements:** - - Create Changelog for nightly release [\#39](https://github.com/Boerderij/Varken/issues/39) - Create proper logging [\#34](https://github.com/Boerderij/Varken/issues/34) @@ -107,12 +99,6 @@ - Remove "dashboard" folder and subfolders [\#42](https://github.com/Boerderij/Varken/issues/42) - Remove "Legacy" folder [\#41](https://github.com/Boerderij/Varken/issues/41) - -## [v0.2-nightly](https://github.com/Boerderij/Varken/tree/v0.2-nightly) (2018-12-06) -[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.1...v0.2-nightly) - -**Closed issues:** - - Create the DB if it does not exist. [\#38](https://github.com/Boerderij/Varken/issues/38) - create systemd examples [\#37](https://github.com/Boerderij/Varken/issues/37) - Create a GeoIP db downloader and refresher [\#36](https://github.com/Boerderij/Varken/issues/36) @@ -130,6 +116,7 @@ **Merged pull requests:** +- v1.0 Merge [\#45](https://github.com/Boerderij/Varken/pull/45) ([DirtyCajunRice](https://github.com/DirtyCajunRice)) - varken to nightly [\#40](https://github.com/Boerderij/Varken/pull/40) ([DirtyCajunRice](https://github.com/DirtyCajunRice)) ## [v0.1](https://github.com/Boerderij/Varken/tree/v0.1) (2018-10-20) diff --git a/Dockerfile b/Dockerfile index 33fe8b1..f64f619 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,6 +12,8 @@ RUN \ /config \ /app +WORKDIR /app + CMD cp /app/data/varken.example.ini /config/varken.example.ini && python3 /app/Varken.py --data-folder /config VOLUME /config diff --git a/README.md b/README.md index 544f703..6953942 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,11 @@ # Varken [![Build Status](https://travis-ci.org/Boerderij/Varken.svg?branch=master)](https://travis-ci.org/Boerderij/Varken) -[![Discord](https://img.shields.io/badge/Discord-Varken-7289DA.svg?logo=discord&style=flat-square)](https://discord.gg/AGTG44H) +[![Discord](https://img.shields.io/badge/Discord-Varken-7289DA.svg?logo=discord&style=flat-square)](https://discord.gg/VjZ6qSM) [![BuyMeACoffee](https://img.shields.io/badge/BuyMeACoffee-Donate-ff813f.svg?logo=CoffeeScript&style=flat-square)](https://www.buymeacoffee.com/varken) +[![Docker-Layers](https://images.microbadger.com/badges/image/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken) +[![Docker-Version](https://images.microbadger.com/badges/version/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken) +[![Docker Pulls](https://img.shields.io/docker/pulls/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/) +[![Docker Stars](https://img.shields.io/docker/stars/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/) Dutch for PIG. PIG is an Acronym for Plex/InfluxDB/Grafana @@ -10,14 +14,14 @@ from the Plex ecosystem into InfluxDB. Examples use Grafana for a frontend Requirements: -* Python3.6.7+ +* [Python 3.6.7+](https://www.python.org/downloads/release/python-367/) * Python3-pip * [InfluxDB](https://www.influxdata.com/)

Example Dashboard - +

Supported Modules: @@ -34,65 +38,8 @@ Key features: * Grafana [Worldmap Panel](https://grafana.com/plugins/grafana-worldmap-panel/installation) support -## Quick Setup (Git Clone) -``` -# Clone the repository -git clone https://github.com/Boerderij/Varken.git /opt/Varken - -# Follow the systemd install instructions located in varken.systemd -cp /opt/Varken/varken.systemd /etc/systemd/system/varken.service -nano /etc/systemd/system/varken.service - -# Create venv in project -/usr/bin/python3 -m venv /opt/Varken/varken-venv - -# Install requirements -/opt/Varken/varken-venv/bin/python -m pip install -r requirements.txt - -# Make a copy of varken.example.ini to varken.ini in the data folder -cp /opt/Varken/data/varken.example.ini /opt/Varken/data/varken.ini - -# Make the appropriate changes to varken.ini -nano /opt/Varken/data/varken.ini - -# Make sure all the files have the appropriate permissions -chown $USER:$USER -R /opt/Varken - -# Start the service and enable it -systemctl start varken -systemctl enable varken -``` -### Docker - -[![Docker-Layers](https://images.microbadger.com/badges/image/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken) -[![Docker-Version](https://images.microbadger.com/badges/version/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken) -[![Docker Pulls](https://img.shields.io/docker/pulls/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/) -[![Docker Stars](https://img.shields.io/docker/stars/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/) -
Example -

- -``` -docker run -d \ - --name=varken \ - -v :/config \ - -e PGID= -e PUID= \ - -e TZ=America/Chicago \ - boerderij/varken -``` -

-
- -#### Tags -* **latest** -* **nightly** -* **release-tag** e.g. v1.0 - -#### Upgrading with docker -```sh -docker stop varken -docker rm varken -# Run deploy command above -``` +## Installation Guides +Varken Installation guides can be found in the [wiki](https://github.com/Boerderij/Varken/wiki/Installation). ### InfluxDB [InfluxDB Installation Documentation](https://docs.influxdata.com/influxdb/v1.7/introduction/installation/) @@ -104,15 +51,4 @@ named `varken` ### Grafana [Grafana Installation Documentation](http://docs.grafana.org/installation/) -[Official Example Dashboards](https://grafana.com/dashboards?search=Varken%20%5BOfficial%5D) - -Grafana is used in our examples but not required, nor packaged as part of -Varken. Panel examples now exist in both nightly and tagged releases hosted -on grafana.com (link above). - -1. Use the link above, then click on your desired dashboard version -2. Click `Copy ID to Clipboard` -3. In grafana, click your dashboards menu dropdown, and then click `Import dashboard` -4. Paste the ID into the `Grafana.com Dashboard` field and then click into empty space on the screen. (This should change the screen to show `Importing Dashboard from Grafana.com` -5. Select your varken datasource name in the dropdown labeled `Varken` -6. Click Import! \ No newline at end of file +Official dashboard installation instructions can be found in the [wiki](https://github.com/Boerderij/Varken/wiki/Installation#grafana) diff --git a/Varken.py b/Varken.py index c157646..1951088 100644 --- a/Varken.py +++ b/Varken.py @@ -77,7 +77,7 @@ if __name__ == "__main__": vl.logger.info('Data folder is "%s"', DATA_FOLDER) vl.logger.info(u"%s %s (%s%s)", platform.system(), platform.release(), platform.version(), - f' - {PLATFORM_LINUX_DISTRO}' if PLATFORM_LINUX_DISTRO else '') + ' - ' + PLATFORM_LINUX_DISTRO if PLATFORM_LINUX_DISTRO else '') vl.logger.info(u"Python %s", version) diff --git a/data/varken.example.ini b/data/varken.example.ini index bbb2ceb..6a8f67a 100644 --- a/data/varken.example.ini +++ b/data/varken.example.ini @@ -1,12 +1,3 @@ -# Notes: -# - Sonarr + Radarr scripts support multiple servers. You can remove the second -# server by putting a # in front of the lines and section name, and removing -# that number from your server_ids list -# - fallback_ip, This is used when there is no IP listed in Tautulli. -# This can happen when you are streaming locally. Set this to your public IP. -# You do not need to change this value if your IP changes. This is only for -# location lookups when there is a failure. - [global] sonarr_server_ids = 1,2 radarr_server_ids = 1,2 diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..d4ea334 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,46 @@ +version: '3' +networks: + internal: + driver: bridge +services: + influxdb: + hostname: influxdb + user: auser + image: influxdb + networks: + - internal + volumes: + - /path/to/docker-influxdb/config-folder:/var/lib/influxdb + restart: unless-stopped + varken: + hostname: varken + image: boerderij/varken + networks: + - internal + volumes: + - /path/to/docker-varken/config-folder:/config + environment: + - PGID=911 + - PUID=911 + - TZ=America/Chicago + depends_on: + - influxdb + restart: unless-stopped + grafana: + hostname: grafana + user: auser + image: grafana/grafana + networks: + - internal + ports: + - 3000:3000 + volumes: + - /path/to/docker-grafana/config-folder:/config + environment: + - GF_PATHS_DATA=/config/data + - GF_PATHS_LOGS=/config/logs + - GF_PATHS_PLUGINS=/config/plugins + - GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel + depends_on: + - influxdb + restart: unless-stopped \ No newline at end of file diff --git a/varken/__init__.py b/varken/__init__.py index fba0206..90e1972 100644 --- a/varken/__init__.py +++ b/varken/__init__.py @@ -1,2 +1,2 @@ -VERSION = 1.5 -BRANCH = 'master' +VERSION = 1.7 +BRANCH = 'pre-nightly' diff --git a/varken/helpers.py b/varken/helpers.py index 648eb5a..5d693c1 100644 --- a/varken/helpers.py +++ b/varken/helpers.py @@ -11,6 +11,7 @@ from json.decoder import JSONDecodeError from os.path import abspath, join, basename, isdir from urllib3.exceptions import InsecureRequestWarning from requests.exceptions import InvalidSchema, SSLError, ConnectionError +from ipaddress import IPv4Address logger = getLogger() @@ -28,7 +29,7 @@ class GeoIPHandler(object): def lookup(self, ipaddress): ip = ipaddress self.logger.debug('Getting lat/long for Tautulli stream using ip with last octet ending in %s', - ip.split('.')[-1:]) + ip.split('.')[-1:][0]) return self.reader.city(ip) def update(self): @@ -83,6 +84,12 @@ def hashit(string): return hashed +def rfc1918_ip_check(ip): + rfc1918_ip = IPv4Address(ip).is_private + + return rfc1918_ip + + def connection_handler(session, request, verify): s = session r = request diff --git a/varken/iniparser.py b/varken/iniparser.py index 3569503..4ad040b 100644 --- a/varken/iniparser.py +++ b/varken/iniparser.py @@ -1,9 +1,10 @@ +from shutil import copyfile from logging import getLogger from os.path import join, exists from re import match, compile, IGNORECASE -from configparser import ConfigParser, NoOptionError +from configparser import ConfigParser, NoOptionError, NoSectionError -from varken.helpers import clean_sid_check +from varken.helpers import clean_sid_check, rfc1918_ip_check from varken.structures import SickChillServer from varken.varkenlogger import BlacklistFilter from varken.structures import SonarrServer, RadarrServer, OmbiServer, TautulliServer, InfluxServer, CiscoASAFirewall @@ -11,20 +12,19 @@ from varken.structures import SonarrServer, RadarrServer, OmbiServer, TautulliSe class INIParser(object): def __init__(self, data_folder): - self.config = ConfigParser(interpolation=None) + self.config = None self.data_folder = data_folder - + self.filtered_strings = None self.services = ['sonarr', 'radarr', 'ombi', 'tautulli', 'sickchill', 'ciscoasa'] - for service in self.services: - setattr(self, f'{service}_servers', []) self.logger = getLogger() - self.influx_server = InfluxServer() - self.parse_opts() - - self.filtered_strings = None + try: + self.parse_opts(read_file=True) + except NoSectionError as e: + self.logger.error('Missing section in (varken.ini): %s', e) + self.rectify_ini() def config_blacklist(self): filtered_strings = [section.get(k) for key, section in self.config.items() @@ -33,30 +33,53 @@ class INIParser(object): # Added matching for domains that use /locations. ConnectionPool ignores the location in logs domains_only = [string.split('/')[0] for string in filtered_strings if '/' in string] self.filtered_strings.extend(domains_only) + # Added matching for domains that use :port. ConnectionPool splits the domain/ip from the port + without_port = [string.split(':')[0] for string in filtered_strings if ':' in string] + self.filtered_strings.extend(without_port) for handler in self.logger.handlers: handler.addFilter(BlacklistFilter(set(self.filtered_strings))) def enable_check(self, server_type=None): t = server_type - try: - global_server_ids = self.config.get('global', t) - if global_server_ids.lower() in ['false', 'no', '0']: - self.logger.info('%s disabled.', t.upper()) - else: - sids = clean_sid_check(global_server_ids, t) - return sids - except NoOptionError as e: - self.logger.error(e) - - def read_file(self): - file_path = join(self.data_folder, 'varken.ini') - if exists(file_path): - with open(file_path) as config_ini: - self.config.read_file(config_ini) - self.config_blacklist() + global_server_ids = self.config.get('global', t) + if global_server_ids.lower() in ['false', 'no', '0']: + self.logger.info('%s disabled.', t.upper()) else: - self.logger.error('Config file missing (varken.ini) in %s', self.data_folder) + sids = clean_sid_check(global_server_ids, t) + return sids + + def read_file(self, inifile): + config = ConfigParser(interpolation=None) + ini = inifile + file_path = join(self.data_folder, ini) + + if not exists(file_path): + self.logger.error('File missing (%s) in %s', ini, self.data_folder) + if inifile == 'varken.ini': + try: + self.logger.debug('Creating varken.ini from varken.example.ini') + copyfile(join(self.data_folder, 'varken.example.ini'), file_path) + except IOError as e: + self.logger.error("Varken does not have permission to write to %s. Error: %s - Exiting.", e, + self.data_folder) + exit(1) + + self.logger.debug('Reading from %s', inifile) + with open(file_path) as config_ini: + config.read_file(config_ini) + + return config + + def write_file(self, inifile): + ini = inifile + file_path = join(self.data_folder, ini) + if exists(file_path): + self.logger.debug('Writing to %s', inifile) + with open(file_path, 'w') as config_ini: + self.config.write(config_ini) + else: + self.logger.error('File missing (%s) in %s', ini, self.data_folder) exit(1) def url_check(self, url=None, include_port=True, section=None): @@ -91,8 +114,31 @@ class INIParser(object): self.logger.debug('%s is a valid URL in module [%s].', url_check, module) return url_check - def parse_opts(self): - self.read_file() + def rectify_ini(self): + self.logger.debug('Rectifying varken.ini with varken.example.ini') + current_ini = self.config + example_ini = self.read_file('varken.example.ini') + + for name, section in example_ini.items(): + if name not in current_ini: + self.logger.debug('Section %s missing. Adding...', name) + current_ini[name] = {} + for key, value in section.items(): + if not current_ini[name].get(key): + self.logger.debug('%s is missing in %s. Adding defaults...', key, name) + current_ini[name][key] = value + + self.config = current_ini + self.write_file('varken.ini') + self.parse_opts() + + def parse_opts(self, read_file=False): + for service in self.services: + setattr(self, f'{service}_servers', []) + + if read_file: + self.config = self.read_file('varken.ini') + self.config_blacklist() # Parse InfluxDB options url = self.url_check(self.config.get('influxdb', 'url'), include_port=False, section='influxdb') @@ -105,7 +151,12 @@ class INIParser(object): # Check for all enabled services for service in self.services: - setattr(self, f'{service}_enabled', self.enable_check(f'{service}_server_ids')) + try: + setattr(self, f'{service}_enabled', self.enable_check(f'{service}_server_ids')) + except NoOptionError as e: + self.logger.error('Missing global %s. Error: %s', f'{service}_server_ids', e) + self.rectify_ini() + return service_enabled = getattr(self, f'{service}_enabled') if service_enabled: @@ -169,6 +220,12 @@ class INIParser(object): get_stats_run_seconds = self.config.getint(section, 'get_stats_run_seconds') + invalid_wan_ip = rfc1918_ip_check(fallback_ip) + + if invalid_wan_ip: + self.logger.error('Invalid failback_ip [%s] set for %s-%s!', fallback_ip, service, server_id) + exit(1) + server = TautulliServer(id=server_id, url=scheme + url, api_key=apikey, verify_ssl=verify_ssl, get_activity=get_activity, fallback_ip=fallback_ip, get_stats=get_stats, @@ -222,5 +279,6 @@ class INIParser(object): getattr(self, f'{service}_servers').append(server) except NoOptionError as e: - setattr(self, f'{service}_enabled', False) - self.logger.error('%s disabled. Error: %s', section, e) + self.logger.error('Missing key in %s. Error: %s', section, e) + self.rectify_ini() + return diff --git a/varken/tautulli.py b/varken/tautulli.py index 336bba9..7d5ec78 100644 --- a/varken/tautulli.py +++ b/varken/tautulli.py @@ -1,3 +1,4 @@ +from os import _exit from logging import getLogger from requests import Session, Request from datetime import datetime, timezone @@ -45,13 +46,19 @@ class TautulliAPI(object): getattr(session, 'ip_address_public') except AttributeError: self.logger.error('Public IP attribute missing!!! Do you have an old version of Tautulli (v1)?') - exit(1) + _exit(1) try: geodata = self.geoiphandler.lookup(session.ip_address_public) except (ValueError, AddressNotFoundError): if self.server.fallback_ip: - geodata = self.geoiphandler.lookup(self.server.fallback_ip) + # Try the failback ip in the config file + try: + geodata = self.geoiphandler.lookup(self.server.fallback_ip) + except AddressNotFoundError as e: + self.logger.error('%s', e) + _exit(1) + else: my_ip = self.session.get('http://ip.42.pl/raw').text geodata = self.geoiphandler.lookup(my_ip)