v1.6 Merge

v1.6 Merge
This commit is contained in:
Nicholas St. Germain 2019-01-04 00:27:22 -06:00 committed by GitHub
commit bca3e50808
10 changed files with 193 additions and 159 deletions

View file

@ -1,5 +1,22 @@
# Change Log
## [v1.6](https://github.com/Boerderij/Varken/tree/v1.6) (2019-01-04)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.5...v1.6)
**Implemented enhancements:**
- \[Feature Request\] docker-compose stack install option [\#84](https://github.com/Boerderij/Varken/issues/84)
- Fix missing variables in varken.ini automatically [\#81](https://github.com/Boerderij/Varken/issues/81)
- Create Wiki for FAQ and help docs [\#80](https://github.com/Boerderij/Varken/issues/80)
**Fixed bugs:**
- \[BUG\] url:port does not filter [\#82](https://github.com/Boerderij/Varken/issues/82)
**Merged pull requests:**
- v1.6 Merge [\#75](https://github.com/Boerderij/Varken/pull/85) ([DirtyCajunRice](https://github.com/DirtyCajunRice))
## [v1.5](https://github.com/Boerderij/Varken/tree/v1.5) (2018-12-30)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.4...v1.5)
@ -20,43 +37,28 @@
- Add Ombi Issues [\#74](https://github.com/Boerderij/Varken/pull/74) ([anderssonoscar0](https://github.com/anderssonoscar0))
## [v1.4](https://github.com/Boerderij/Varken/tree/v1.4) (2018-12-19)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.3-nightly...v1.4)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.1...v1.4)
**Implemented enhancements:**
- \[Feature Request\] Add tautulli request for library stats [\#64](https://github.com/Boerderij/Varken/issues/64)
- Create randomized 12-24 hour checks to update GeoLite DB after the first wednesday of the month [\#60](https://github.com/Boerderij/Varken/issues/60)
- \[Feature Request\]: Pull list of requests \(instead of just counts\) [\#58](https://github.com/Boerderij/Varken/issues/58)
- Feature Request , Add Sickchill [\#48](https://github.com/Boerderij/Varken/issues/48)
**Fixed bugs:**
- \[BUG\] Ombi all requests missing half of "pending" option [\#63](https://github.com/Boerderij/Varken/issues/63)
- \[BUG\] asa bug with checking for apikey [\#62](https://github.com/Boerderij/Varken/issues/62)
**Merged pull requests:**
- v1.4 Merge [\#65](https://github.com/Boerderij/Varken/pull/65) ([DirtyCajunRice](https://github.com/DirtyCajunRice))
## [v1.3-nightly](https://github.com/Boerderij/Varken/tree/v1.3-nightly) (2018-12-18)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.2-nightly...v1.3-nightly)
**Implemented enhancements:**
- Create randomized 12-24 hour checks to update GeoLite DB after the first wednesday of the month [\#60](https://github.com/Boerderij/Varken/issues/60)
**Fixed bugs:**
- \[BUG\] Add Catchall to ombi requests [\#59](https://github.com/Boerderij/Varken/issues/59)
**Closed issues:**
- Unify naming and cleanup duplication in iniparser [\#61](https://github.com/Boerderij/Varken/issues/61)
## [v1.2-nightly](https://github.com/Boerderij/Varken/tree/v1.2-nightly) (2018-12-16)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.1...v1.2-nightly)
**Merged pull requests:**
**Implemented enhancements:**
- \[Feature Request\]: Pull list of requests \(instead of just counts\) [\#58](https://github.com/Boerderij/Varken/issues/58)
- Feature Request , Add Sickchill [\#48](https://github.com/Boerderij/Varken/issues/48)
- v1.4 Merge [\#65](https://github.com/Boerderij/Varken/pull/65) ([DirtyCajunRice](https://github.com/DirtyCajunRice))
## [v1.1](https://github.com/Boerderij/Varken/tree/v1.1) (2018-12-11)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v1.0...v1.1)
@ -84,22 +86,12 @@
- Update issue templates [\#55](https://github.com/Boerderij/Varken/pull/55) ([DirtyCajunRice](https://github.com/DirtyCajunRice))
## [v1.0](https://github.com/Boerderij/Varken/tree/v1.0) (2018-12-10)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.3-nightly...v1.0)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.1...v1.0)
**Implemented enhancements:**
- Add cisco asa from legacy [\#44](https://github.com/Boerderij/Varken/issues/44)
- Add server ID to ombi to differenciate [\#43](https://github.com/Boerderij/Varken/issues/43)
**Merged pull requests:**
- v1.0 Merge [\#45](https://github.com/Boerderij/Varken/pull/45) ([DirtyCajunRice](https://github.com/DirtyCajunRice))
## [v0.3-nightly](https://github.com/Boerderij/Varken/tree/v0.3-nightly) (2018-12-07)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.2-nightly...v0.3-nightly)
**Implemented enhancements:**
- Create Changelog for nightly release [\#39](https://github.com/Boerderij/Varken/issues/39)
- Create proper logging [\#34](https://github.com/Boerderij/Varken/issues/34)
@ -107,12 +99,6 @@
- Remove "dashboard" folder and subfolders [\#42](https://github.com/Boerderij/Varken/issues/42)
- Remove "Legacy" folder [\#41](https://github.com/Boerderij/Varken/issues/41)
## [v0.2-nightly](https://github.com/Boerderij/Varken/tree/v0.2-nightly) (2018-12-06)
[Full Changelog](https://github.com/Boerderij/Varken/compare/v0.1...v0.2-nightly)
**Closed issues:**
- Create the DB if it does not exist. [\#38](https://github.com/Boerderij/Varken/issues/38)
- create systemd examples [\#37](https://github.com/Boerderij/Varken/issues/37)
- Create a GeoIP db downloader and refresher [\#36](https://github.com/Boerderij/Varken/issues/36)
@ -130,6 +116,7 @@
**Merged pull requests:**
- v1.0 Merge [\#45](https://github.com/Boerderij/Varken/pull/45) ([DirtyCajunRice](https://github.com/DirtyCajunRice))
- varken to nightly [\#40](https://github.com/Boerderij/Varken/pull/40) ([DirtyCajunRice](https://github.com/DirtyCajunRice))
## [v0.1](https://github.com/Boerderij/Varken/tree/v0.1) (2018-10-20)

View file

@ -12,6 +12,8 @@ RUN \
/config \
/app
WORKDIR /app
CMD cp /app/data/varken.example.ini /config/varken.example.ini && python3 /app/Varken.py --data-folder /config
VOLUME /config

View file

@ -1,7 +1,11 @@
# Varken
[![Build Status](https://travis-ci.org/Boerderij/Varken.svg?branch=master)](https://travis-ci.org/Boerderij/Varken)
[![Discord](https://img.shields.io/badge/Discord-Varken-7289DA.svg?logo=discord&style=flat-square)](https://discord.gg/AGTG44H)
[![Discord](https://img.shields.io/badge/Discord-Varken-7289DA.svg?logo=discord&style=flat-square)](https://discord.gg/VjZ6qSM)
[![BuyMeACoffee](https://img.shields.io/badge/BuyMeACoffee-Donate-ff813f.svg?logo=CoffeeScript&style=flat-square)](https://www.buymeacoffee.com/varken)
[![Docker-Layers](https://images.microbadger.com/badges/image/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken)
[![Docker-Version](https://images.microbadger.com/badges/version/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken)
[![Docker Pulls](https://img.shields.io/docker/pulls/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/)
[![Docker Stars](https://img.shields.io/docker/stars/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/)
Dutch for PIG. PIG is an Acronym for Plex/InfluxDB/Grafana
@ -10,14 +14,14 @@ from the Plex ecosystem into InfluxDB. Examples use Grafana for a
frontend
Requirements:
* Python3.6.7+
* [Python 3.6.7+](https://www.python.org/downloads/release/python-367/)
* Python3-pip
* [InfluxDB](https://www.influxdata.com/)
<p align="center">
Example Dashboard
<img width="800" src="https://i.imgur.com/G5bnpjs.png">
<img width="800" src="https://i.imgur.com/3hNZTkC.png">
</p>
Supported Modules:
@ -34,65 +38,8 @@ Key features:
* Grafana [Worldmap Panel](https://grafana.com/plugins/grafana-worldmap-panel/installation) support
## Quick Setup (Git Clone)
```
# Clone the repository
git clone https://github.com/Boerderij/Varken.git /opt/Varken
# Follow the systemd install instructions located in varken.systemd
cp /opt/Varken/varken.systemd /etc/systemd/system/varken.service
nano /etc/systemd/system/varken.service
# Create venv in project
/usr/bin/python3 -m venv /opt/Varken/varken-venv
# Install requirements
/opt/Varken/varken-venv/bin/python -m pip install -r requirements.txt
# Make a copy of varken.example.ini to varken.ini in the data folder
cp /opt/Varken/data/varken.example.ini /opt/Varken/data/varken.ini
# Make the appropriate changes to varken.ini
nano /opt/Varken/data/varken.ini
# Make sure all the files have the appropriate permissions
chown $USER:$USER -R /opt/Varken
# Start the service and enable it
systemctl start varken
systemctl enable varken
```
### Docker
[![Docker-Layers](https://images.microbadger.com/badges/image/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken)
[![Docker-Version](https://images.microbadger.com/badges/version/boerderij/varken.svg)](https://microbadger.com/images/boerderij/varken)
[![Docker Pulls](https://img.shields.io/docker/pulls/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/)
[![Docker Stars](https://img.shields.io/docker/stars/boerderij/varken.svg)](https://hub.docker.com/r/boerderij/varken/)
<details><summary>Example</summary>
<p>
```
docker run -d \
--name=varken \
-v <path to data>:/config \
-e PGID=<gid> -e PUID=<uid> \
-e TZ=America/Chicago \
boerderij/varken
```
</p>
</details>
#### Tags
* **latest**
* **nightly**
* **release-tag** e.g. v1.0
#### Upgrading with docker
```sh
docker stop varken
docker rm varken
# Run deploy command above
```
## Installation Guides
Varken Installation guides can be found in the [wiki](https://github.com/Boerderij/Varken/wiki/Installation).
### InfluxDB
[InfluxDB Installation Documentation](https://docs.influxdata.com/influxdb/v1.7/introduction/installation/)
@ -104,15 +51,4 @@ named `varken`
### Grafana
[Grafana Installation Documentation](http://docs.grafana.org/installation/)
[Official Example Dashboards](https://grafana.com/dashboards?search=Varken%20%5BOfficial%5D)
Grafana is used in our examples but not required, nor packaged as part of
Varken. Panel examples now exist in both nightly and tagged releases hosted
on grafana.com (link above).
1. Use the link above, then click on your desired dashboard version
2. Click `Copy ID to Clipboard`
3. In grafana, click your dashboards menu dropdown, and then click `Import dashboard`
4. Paste the ID into the `Grafana.com Dashboard` field and then click into empty space on the screen. (This should change the screen to show `Importing Dashboard from Grafana.com`
5. Select your varken datasource name in the dropdown labeled `Varken`
6. Click Import!
Official dashboard installation instructions can be found in the [wiki](https://github.com/Boerderij/Varken/wiki/Installation#grafana)

View file

@ -77,7 +77,7 @@ if __name__ == "__main__":
vl.logger.info('Data folder is "%s"', DATA_FOLDER)
vl.logger.info(u"%s %s (%s%s)", platform.system(), platform.release(), platform.version(),
f' - {PLATFORM_LINUX_DISTRO}' if PLATFORM_LINUX_DISTRO else '')
' - ' + PLATFORM_LINUX_DISTRO if PLATFORM_LINUX_DISTRO else '')
vl.logger.info(u"Python %s", version)

View file

@ -1,12 +1,3 @@
# Notes:
# - Sonarr + Radarr scripts support multiple servers. You can remove the second
# server by putting a # in front of the lines and section name, and removing
# that number from your server_ids list
# - fallback_ip, This is used when there is no IP listed in Tautulli.
# This can happen when you are streaming locally. Set this to your public IP.
# You do not need to change this value if your IP changes. This is only for
# location lookups when there is a failure.
[global]
sonarr_server_ids = 1,2
radarr_server_ids = 1,2

46
docker-compose.yml Normal file
View file

@ -0,0 +1,46 @@
version: '3'
networks:
internal:
driver: bridge
services:
influxdb:
hostname: influxdb
user: auser
image: influxdb
networks:
- internal
volumes:
- /path/to/docker-influxdb/config-folder:/var/lib/influxdb
restart: unless-stopped
varken:
hostname: varken
image: boerderij/varken
networks:
- internal
volumes:
- /path/to/docker-varken/config-folder:/config
environment:
- PGID=911
- PUID=911
- TZ=America/Chicago
depends_on:
- influxdb
restart: unless-stopped
grafana:
hostname: grafana
user: auser
image: grafana/grafana
networks:
- internal
ports:
- 3000:3000
volumes:
- /path/to/docker-grafana/config-folder:/config
environment:
- GF_PATHS_DATA=/config/data
- GF_PATHS_LOGS=/config/logs
- GF_PATHS_PLUGINS=/config/plugins
- GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel
depends_on:
- influxdb
restart: unless-stopped

View file

@ -1,2 +1,2 @@
VERSION = 1.5
BRANCH = 'master'
VERSION = 1.7
BRANCH = 'pre-nightly'

View file

@ -11,6 +11,7 @@ from json.decoder import JSONDecodeError
from os.path import abspath, join, basename, isdir
from urllib3.exceptions import InsecureRequestWarning
from requests.exceptions import InvalidSchema, SSLError, ConnectionError
from ipaddress import IPv4Address
logger = getLogger()
@ -28,7 +29,7 @@ class GeoIPHandler(object):
def lookup(self, ipaddress):
ip = ipaddress
self.logger.debug('Getting lat/long for Tautulli stream using ip with last octet ending in %s',
ip.split('.')[-1:])
ip.split('.')[-1:][0])
return self.reader.city(ip)
def update(self):
@ -83,6 +84,12 @@ def hashit(string):
return hashed
def rfc1918_ip_check(ip):
rfc1918_ip = IPv4Address(ip).is_private
return rfc1918_ip
def connection_handler(session, request, verify):
s = session
r = request

View file

@ -1,9 +1,10 @@
from shutil import copyfile
from logging import getLogger
from os.path import join, exists
from re import match, compile, IGNORECASE
from configparser import ConfigParser, NoOptionError
from configparser import ConfigParser, NoOptionError, NoSectionError
from varken.helpers import clean_sid_check
from varken.helpers import clean_sid_check, rfc1918_ip_check
from varken.structures import SickChillServer
from varken.varkenlogger import BlacklistFilter
from varken.structures import SonarrServer, RadarrServer, OmbiServer, TautulliServer, InfluxServer, CiscoASAFirewall
@ -11,20 +12,19 @@ from varken.structures import SonarrServer, RadarrServer, OmbiServer, TautulliSe
class INIParser(object):
def __init__(self, data_folder):
self.config = ConfigParser(interpolation=None)
self.config = None
self.data_folder = data_folder
self.filtered_strings = None
self.services = ['sonarr', 'radarr', 'ombi', 'tautulli', 'sickchill', 'ciscoasa']
for service in self.services:
setattr(self, f'{service}_servers', [])
self.logger = getLogger()
self.influx_server = InfluxServer()
self.parse_opts()
self.filtered_strings = None
try:
self.parse_opts(read_file=True)
except NoSectionError as e:
self.logger.error('Missing section in (varken.ini): %s', e)
self.rectify_ini()
def config_blacklist(self):
filtered_strings = [section.get(k) for key, section in self.config.items()
@ -33,30 +33,53 @@ class INIParser(object):
# Added matching for domains that use /locations. ConnectionPool ignores the location in logs
domains_only = [string.split('/')[0] for string in filtered_strings if '/' in string]
self.filtered_strings.extend(domains_only)
# Added matching for domains that use :port. ConnectionPool splits the domain/ip from the port
without_port = [string.split(':')[0] for string in filtered_strings if ':' in string]
self.filtered_strings.extend(without_port)
for handler in self.logger.handlers:
handler.addFilter(BlacklistFilter(set(self.filtered_strings)))
def enable_check(self, server_type=None):
t = server_type
try:
global_server_ids = self.config.get('global', t)
if global_server_ids.lower() in ['false', 'no', '0']:
self.logger.info('%s disabled.', t.upper())
else:
sids = clean_sid_check(global_server_ids, t)
return sids
except NoOptionError as e:
self.logger.error(e)
def read_file(self):
file_path = join(self.data_folder, 'varken.ini')
if exists(file_path):
with open(file_path) as config_ini:
self.config.read_file(config_ini)
self.config_blacklist()
global_server_ids = self.config.get('global', t)
if global_server_ids.lower() in ['false', 'no', '0']:
self.logger.info('%s disabled.', t.upper())
else:
self.logger.error('Config file missing (varken.ini) in %s', self.data_folder)
sids = clean_sid_check(global_server_ids, t)
return sids
def read_file(self, inifile):
config = ConfigParser(interpolation=None)
ini = inifile
file_path = join(self.data_folder, ini)
if not exists(file_path):
self.logger.error('File missing (%s) in %s', ini, self.data_folder)
if inifile == 'varken.ini':
try:
self.logger.debug('Creating varken.ini from varken.example.ini')
copyfile(join(self.data_folder, 'varken.example.ini'), file_path)
except IOError as e:
self.logger.error("Varken does not have permission to write to %s. Error: %s - Exiting.", e,
self.data_folder)
exit(1)
self.logger.debug('Reading from %s', inifile)
with open(file_path) as config_ini:
config.read_file(config_ini)
return config
def write_file(self, inifile):
ini = inifile
file_path = join(self.data_folder, ini)
if exists(file_path):
self.logger.debug('Writing to %s', inifile)
with open(file_path, 'w') as config_ini:
self.config.write(config_ini)
else:
self.logger.error('File missing (%s) in %s', ini, self.data_folder)
exit(1)
def url_check(self, url=None, include_port=True, section=None):
@ -91,8 +114,31 @@ class INIParser(object):
self.logger.debug('%s is a valid URL in module [%s].', url_check, module)
return url_check
def parse_opts(self):
self.read_file()
def rectify_ini(self):
self.logger.debug('Rectifying varken.ini with varken.example.ini')
current_ini = self.config
example_ini = self.read_file('varken.example.ini')
for name, section in example_ini.items():
if name not in current_ini:
self.logger.debug('Section %s missing. Adding...', name)
current_ini[name] = {}
for key, value in section.items():
if not current_ini[name].get(key):
self.logger.debug('%s is missing in %s. Adding defaults...', key, name)
current_ini[name][key] = value
self.config = current_ini
self.write_file('varken.ini')
self.parse_opts()
def parse_opts(self, read_file=False):
for service in self.services:
setattr(self, f'{service}_servers', [])
if read_file:
self.config = self.read_file('varken.ini')
self.config_blacklist()
# Parse InfluxDB options
url = self.url_check(self.config.get('influxdb', 'url'), include_port=False, section='influxdb')
@ -105,7 +151,12 @@ class INIParser(object):
# Check for all enabled services
for service in self.services:
setattr(self, f'{service}_enabled', self.enable_check(f'{service}_server_ids'))
try:
setattr(self, f'{service}_enabled', self.enable_check(f'{service}_server_ids'))
except NoOptionError as e:
self.logger.error('Missing global %s. Error: %s', f'{service}_server_ids', e)
self.rectify_ini()
return
service_enabled = getattr(self, f'{service}_enabled')
if service_enabled:
@ -169,6 +220,12 @@ class INIParser(object):
get_stats_run_seconds = self.config.getint(section, 'get_stats_run_seconds')
invalid_wan_ip = rfc1918_ip_check(fallback_ip)
if invalid_wan_ip:
self.logger.error('Invalid failback_ip [%s] set for %s-%s!', fallback_ip, service, server_id)
exit(1)
server = TautulliServer(id=server_id, url=scheme + url, api_key=apikey,
verify_ssl=verify_ssl, get_activity=get_activity,
fallback_ip=fallback_ip, get_stats=get_stats,
@ -222,5 +279,6 @@ class INIParser(object):
getattr(self, f'{service}_servers').append(server)
except NoOptionError as e:
setattr(self, f'{service}_enabled', False)
self.logger.error('%s disabled. Error: %s', section, e)
self.logger.error('Missing key in %s. Error: %s', section, e)
self.rectify_ini()
return

View file

@ -1,3 +1,4 @@
from os import _exit
from logging import getLogger
from requests import Session, Request
from datetime import datetime, timezone
@ -45,13 +46,19 @@ class TautulliAPI(object):
getattr(session, 'ip_address_public')
except AttributeError:
self.logger.error('Public IP attribute missing!!! Do you have an old version of Tautulli (v1)?')
exit(1)
_exit(1)
try:
geodata = self.geoiphandler.lookup(session.ip_address_public)
except (ValueError, AddressNotFoundError):
if self.server.fallback_ip:
geodata = self.geoiphandler.lookup(self.server.fallback_ip)
# Try the failback ip in the config file
try:
geodata = self.geoiphandler.lookup(self.server.fallback_ip)
except AddressNotFoundError as e:
self.logger.error('%s', e)
_exit(1)
else:
my_ip = self.session.get('http://ip.42.pl/raw').text
geodata = self.geoiphandler.lookup(my_ip)