#!/usr/bin/env python
# coding: utf-8

# # Logistic Regression with Scikit Learn - Machine Learning with Python
#
# ![](https://i.imgur.com/N8aIuRK.jpg)

# The following topics are covered in this tutorial:
#
# - Downloading a real-world dataset from Kaggle
# - Exploratory data analysis and visualization
# - Splitting a dataset into training, validation & test sets
# - Filling/imputing missing values in numeric columns
# - Scaling numeric features to a $(0,1)$ range
# - Encoding categorical columns as one-hot vectors
# - Training a logistic regression model using Scikit-learn
# - Evaluating a model using a validation set and test set
# - Saving a model to disk and loading it back
#

# ## Problem Statement
#
# This tutorial takes a practical and coding-focused approach. We'll learn how to apply _logistic regression_ to a real-world dataset from [Kaggle](https://kaggle.com/datasets):
#
# > **QUESTION**: The [Rain in Australia dataset](https://kaggle.com/jsphyg/weather-dataset-rattle-package) contains about 10 years of daily weather observations from numerous Australian weather stations. Here's a small sample from the dataset:
# >
# > ![](https://i.imgur.com/5QNJvir.png)
# >
# > As a data scientist at the Bureau of Meteorology, you are tasked with creating a fully-automated system that can use today's weather data for a given location to predict whether it will rain at the location tomorrow.
# >
# >
# > ![](https://i.imgur.com/KWfcpcO.png)
#

# ## Linear Regression vs. Logistic Regression
# In this tutorial, we'll use _logistic regression_, which is better suited for _classification_ problems like predicting whether it will rain tomorrow. Identifying whether a given problem is a _classfication_ or _regression_ problem is an important first step in machine learning.
#

# ### Classification Problems
#
#
# Problems where each input must be assigned a discrete category (also called label or class) are known as _classification problems_.
#
# Here are some examples of classification problems:
#
# - [Rainfall prediction](https://kaggle.com/jsphyg/weather-dataset-rattle-package): Predicting whether it will rain tomorrow using today's weather data (classes are "Will Rain" and "Will Not Rain")
# - [Breast cancer detection](https://www.kaggle.com/uciml/breast-cancer-wisconsin-data): Predicting whether a tumor  is "benign" (noncancerous) or "malignant" (cancerous) using information like its radius, texture etc.
# - [Loan Repayment Prediction](https://www.kaggle.com/c/home-credit-default-risk) - Predicting whether applicants will repay a home loan based on factors like age, income, loan amount, no. of children etc.
# - [Handwritten Digit Recognition](https://www.kaggle.com/c/digit-recognizer) - Identifying which digit from 0 to 9 a picture of handwritten text represents.
#
# Can you think of some more classification problems?
#
# > **EXERCISE**: Replicate the steps followed in this tutorial with each of the above datasets.
#
#
# Classification problems can be binary (yes/no) or multiclass (picking one of many classes).
#

# ### Regression Problems
#
# Problems where a continuous numeric value must be predicted for each input are known as _regression problems_.
#
# Here are some example of regression problems:
#
# - [Medical Charges Prediction](https://www.kaggle.com/subhakarks/medical-insurance-cost-analysis-and-prediction)
# - [House Price Prediction](https://www.kaggle.com/c/house-prices-advanced-regression-techniques)
# - [Ocean Temperature Prediction](https://www.kaggle.com/sohier/calcofi)
# - [Weather Temperature Prediction](https://www.kaggle.com/budincsevity/szeged-weather)
#
# Can you think of some more regression problems?

# ### Linear Regression for Solving Regression Problems
#
# Linear regression is a commonly used technique for solving regression problems. In a linear regression model, the target is modeled as a linear combination (or weighted sum) of input features. The predictions from the model are evaluated using a loss function like the Root Mean Squared Error (RMSE).
#
#
# Here's a visual summary of how a linear regression model is structured:
#
#
#
#
# For a mathematical discussion of linear regression, watch [this YouTube playlist](https://www.youtube.com/watch?v=kHwlB_j7Hkc&list=PLJs7lEb1U5pYnrI0Wn4mzPmppVqwERL_4&index=1)
#

# ### Logistic Regression for Solving Classification Problems
#
# Logistic regression is a commonly used technique for solving binary classification problems. In a logistic regression model:
#
# - we take linear combination (or weighted sum of the input features)
# - we apply the sigmoid function to the result to obtain a number between 0 and 1
# - this number represents the probability of the input being classified as "Yes"
# - instead of RMSE, the cross entropy loss function is used to evaluate the results
#
#
# Here's a visual summary of how a logistic regression model is structured ([source](http://datahacker.rs/005-pytorch-logistic-regression-in-pytorch/)):
#
#
#
#
# The sigmoid function applied to the linear combination of inputs has the following formula:
#
#
#
#
# The output of the sigmoid function is called a logistic, hence the name _logistic regression_. For a mathematical discussion of logistic regression, sigmoid activation and cross entropy, check out [this YouTube playlist](https://www.youtube.com/watch?v=-la3q9d7AKQ&list=PLNeKWBMsAzboR8vvhnlanxCNr2V7ITuxy&index=1). Logistic regression can also be applied to multi-class classification problems, with a few modifications.
#

# ### Machine Learning Workflow
#
# Whether we're solving a regression problem using linear regression or a classification problem using logistic regression, the workflow for training a model is exactly the same:
#
# 1. We initialize a model with random parameters (weights & biases).
# 2. We pass some inputs into the model to obtain predictions.
# 3. We compare the model's predictions with the actual targets using the loss function.
# 4. We use an optimization technique (like least squares, gradient descent etc.) to reduce the loss by adjusting the weights & biases of the model
# 5. We repeat steps 1 to 4 till the predictions from the model are good enough.
#
#
#
#
#
# Classification and regression are both supervised machine learning problems, because they use labeled data. Machine learning applied to unlabeled data is known as unsupervised learning ([image source](https://au.mathworks.com/help/stats/machine-learning-in-matlab.html)).
#
#
#
#
# In this tutorial, we'll train a _logistic regression_ model using the Rain in Australia dataset to predict whether or not it will rain at a location tomorrow, using today's data. This is a _binary classification_ problem.
#
#

# ## Downloading the Data
#
# We'll use the [`opendatasets` library](https://github.com/JovianML/opendatasets) to download the data from Kaggle directly within Jupyter. Let's install and import `opendatasets`.

# In[1]:

import opendatasets as od

# In[2]:

od.version()

# The dataset can now be downloaded using `od.download`. When you execute `od.download`, you will be asked to provide your Kaggle username and API key. Follow these instructions to create an API key: http://bit.ly/kaggle-creds

# In[1]:

dataset_url = 'https://www.kaggle.com/jsphyg/weather-dataset-rattle-package'

# In[4]:

od.download(dataset_url)

# Once the above command is executed, the dataset is downloaded and extracted to the the directory `weather-dataset-rattle-package`.

# In[5]:

import os

# In[6]:

data_dir = './weather-dataset-rattle-package'

# In[7]:

os.listdir(data_dir)

# In[8]:

train_csv = data_dir + '/weatherAUS.csv'

#
# Let's load the data from `weatherAUS.csv` using Pandas.

# In[9]:

import pandas as pd

# In[10]:

raw_df = pd.read_csv(train_csv)

# In[11]:

raw_df

# The dataset contains over 145,000 rows and 23 columns. The dataset contains date, numeric and categorical columns. Our objective is to create a model to predict the value in the column `RainTomorrow`.
#
# Let's check the data types and missing values in the various columns.

# In[12]:

raw_df.info()

# While we should be able to fill in missing values for most columns, it might be a good idea to discard the rows where the value of `RainTomorrow` or `RainToday` is missing to make our analysis and modeling simpler (since one of them is the target variable, and the other is likely to be very closely related to the target variable).

# In[13]:

raw_df.dropna(subset=['RainToday', 'RainTomorrow'], inplace=True)

# In[14]:

raw_df.info()

# How would you deal with the missing values in the other columns?
#

# ## Exploratory Data Analysis and Visualization
#
# Before training a machine learning model, its always a good idea to explore the distributions of various columns and see how they are related to the target column. Let's explore and visualize the data using the Plotly, Matplotlib and Seaborn libraries.

# In[15]:

import plotly.express as px
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
get_ipython().run_line_magic('matplotlib', 'inline')

sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (10, 6)
matplotlib.rcParams['figure.facecolor'] = '#00000000'

# In[16]:

px.histogram(raw_df, x='Location', title='Location vs. Rainy Days', color='RainToday')

# In[17]:

px.histogram(raw_df,
             x='Temp3pm',
             title='Temperature at 3 pm vs. Rain Tomorrow',
             color='RainTomorrow')

# In[18]:

px.histogram(raw_df,
             x='RainTomorrow',
             color='RainToday',
             title='Rain Tomorrow vs. Rain Today')

# In[19]:

px.scatter(raw_df.sample(2000),
           title='Min Temp. vs Max Temp.',
           x='MinTemp',
           y='MaxTemp',
           color='RainToday')

# In[20]:

px.scatter(raw_df.sample(2000),
           title='Temp (3 pm) vs. Humidity (3 pm)',
           x='Temp3pm',
           y='Humidity3pm',
           color='RainTomorrow')

# What interpertations can you draw from the above charts?
#
# > **EXERCISE**: Visualize all the other columns of the dataset and study their relationship with the `RainToday` and `RainTomorrow` columns.

# ## (Optional) Working with a Sample
#
# When working with massive datasets containing millions of rows, it's a good idea to work with a sample initially, to quickly set up your model training notebook. If you'd like to work with a sample, just set the value of `use_sample` to `True`.

# In[21]:

#use_sample = False

# In[22]:

#sample_fraction = 0.1

# In[23]:

#if use_sample:
    #raw_df = raw_df.sample(frac=sample_fraction).copy()

# Make sure to set `use_sample` to `False` and re-run the notebook end-to-end once you're ready to use the entire dataset.

# ## Training, Validation and Test Sets
#
# While building real-world machine learning models, it is quite common to split the dataset into three parts:
#
# 1. **Training set** - used to train the model, i.e., compute the loss and adjust the model's weights using an optimization technique.
#
#
# 2. **Validation set** - used to evaluate the model during training, tune model hyperparameters (optimization technique, regularization etc.), and pick the best version of the model. Picking a good validation set is essential for training models that generalize well. [Learn more here.](https://www.fast.ai/2017/11/13/validation-sets/)
#
#
# 3. **Test set** - used to compare different models or approaches and report the model's final accuracy. For many datasets, test sets are provided separately. The test set should reflect the kind of data the model will encounter in the real-world, as closely as feasible.
#
#
#
#
#
# As a general rule of thumb you can use around 60% of the data for the training set, 20% for the validation set and 20% for the test set. If a separate test set is already provided, you can use a 75%-25% training-validation split.
#
#
# When rows in the dataset have no inherent order, it's common practice to pick random subsets of rows for creating test and validation sets. This can be done using the `train_test_split` utility from `scikit-learn`. Learn more about it here: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

# In[24]:

from sklearn.model_selection import train_test_split

# In[25]:

temp_df, test_df = train_test_split(raw_df, test_size=0.2, random_state=42)
train_df, val_df = train_test_split(temp_df, test_size=0.25, random_state=42)

# In[26]:

print('train_df.shape :', train_df.shape)
print('val_df.shape :', val_df.shape)
print('test_df.shape :', test_df.shape)

# However, while working with dates, it's often a better idea to separate the training, validation and test sets with time, so that the model is trained on data from the past and evaluated on data from the future.
#
# For the current dataset, we can use the Date column in the dataset to create another column for year. We'll pick the last two years for the test set, and one year before it for the validation set.

# In[27]:

plt.title('No. of Rows per Year')
sns.countplot(x=pd.to_datetime(raw_df.Date).dt.year);

# In[28]:

year = pd.to_datetime(raw_df.Date).dt.year

train_df = raw_df[year < 2015]
val_df = raw_df[year == 2015]
test_df = raw_df[year > 2015]

# In[29]:

print('train_df.shape :', train_df.shape)
print('val_df.shape :', val_df.shape)
print('test_df.shape :', test_df.shape)

# While not a perfect 60-20-20 split, we have ensured that the test validation and test sets both contain data for all 12 months of the year.

# In[30]:

train_df

# In[31]:

val_df

# In[32]:

test_df

# ## Identifying Input and Target Columns
#
# Often, not all the columns in a dataset are useful for training a model. In the current dataset, we can ignore the `Date` column, since we only want to weather conditions to make a prediction about whether it will rain the next day.
#
# Let's create a list of input columns, and also identify the target column.

# In[33]:

input_cols = list(train_df.columns)[1:-1]
target_col = 'RainTomorrow'

# In[34]:

print(input_cols)

# In[35]:

target_col

# We can now create inputs and targets for the training, validation and test sets for further processing and model training.

# In[36]:

train_inputs = train_df[input_cols].copy()
train_targets = train_df[target_col].copy()

# '.copy' is used to make a copy of original data frame. We are making a copy here because we don't want our main data frame to be disturbed.

# In[37]:

val_inputs = val_df[input_cols].copy()
val_targets = val_df[target_col].copy()

# In[38]:

test_inputs = test_df[input_cols].copy()
test_targets = test_df[target_col].copy()

# In[39]:

train_inputs

# In[40]:

train_targets

# Let's also identify which of the columns are numerical and which ones are categorical. This will be useful later, as we'll need to convert the categorical data to numbers for training a logistic regression model.

# In[41]:

get_ipython().system('pip install numpy --quiet')

# In[42]:

import numpy as np

# In[43]:

numeric_cols = train_inputs.select_dtypes(include=np.number).columns.tolist()
categorical_cols = train_inputs.select_dtypes('object').columns.tolist()

# In[44]:

numeric_cols

# In[45]:

categorical_cols

# Let's view some statistics for the numeric columns.

# In[46]:

train_inputs[numeric_cols].describe()

# Do the ranges of the numeric columns seem reasonable? If not, we may have to do some data cleaning as well.
#
# Let's also check the number of categories in each of the categorical columns.

# In[47]:

train_inputs[categorical_cols].nunique()

# ## Imputing Missing Numeric Data
#
# Machine learning models can't work with missing numerical data. The process of filling missing values is called imputation.
#
#
#
# There are several techniques for imputation, but we'll use the most basic one: replacing missing values with the average value in the column using the `SimpleImputer` class from `sklearn.impute`.

# In[48]:

from sklearn.impute import SimpleImputer

# In[49]:

imputer = SimpleImputer(strategy = 'mean')

# Before we perform imputation, let's check the no. of missing values in each numeric column.

# In[50]:

raw_df[numeric_cols].isna().sum()

# These values are spread across the training, test and validation sets. You can also check the no. of missing values individually for `train_inputs`, `val_inputs` and `test_inputs`.

# In[51]:

train_inputs[numeric_cols].isna().sum()

# The first step in imputation is to `fit` the imputer to the data i.e. compute the chosen statistic (e.g. mean) for each column in the dataset.

# In[52]:

imputer.fit(raw_df[numeric_cols])

# After calling `fit`, the computed statistic for each column is stored in the `statistics_` property of `imputer`.

# In[53]:

list(imputer.statistics_)

# The missing values in the training, test and validation sets can now be filled in using the `transform` method of `imputer`.

# In[54]:

train_inputs[numeric_cols] = imputer.transform(train_inputs[numeric_cols])
val_inputs[numeric_cols] = imputer.transform(val_inputs[numeric_cols])
test_inputs[numeric_cols] = imputer.transform(test_inputs[numeric_cols])

# The missing values are now filled in with the mean of each column.

# In[55]:

train_inputs[numeric_cols].isna().sum()

# > **EXERCISE**: Apply some other imputation techniques and observe how they change the results of the model. You can learn more about other imputation techniques here: https://scikit-learn.org/stable/modules/impute.html

# ## Scaling Numeric Features
#
# Another good practice is to scale numeric features to a small range of values e.g. $(0,1)$ or $(-1,1)$. Scaling numeric features ensures that no particular feature has a disproportionate impact on the model's loss. Optimization algorithms also work better in practice with smaller numbers.
#
# The numeric columns in our dataset have varying ranges.

# In[56]:

raw_df[numeric_cols].describe()

# Let's use `MinMaxScaler` from `sklearn.preprocessing` to scale values to the $(0,1)$ range.

# In[3]:

from sklearn.preprocessing import MinMaxScaler

# In[4]:

get_ipython().run_line_magic('pinfo', 'MinMaxScaler')

# In[59]:

scaler = MinMaxScaler()

# First, we `fit` the scaler to the data i.e. compute the range of values for each numeric column.

# In[60]:

scaler.fit(raw_df[numeric_cols])

# We can now inspect the minimum and maximum values in each column.

# In[61]:

print('Minimum:')
list(scaler.data_min_)

# In[62]:

print('Maximum:')
list(scaler.data_max_)

# We can now separately scale the training, validation and test sets using the `transform` method of `scaler`.

# In[63]:

train_inputs[numeric_cols] = scaler.transform(train_inputs[numeric_cols])
val_inputs[numeric_cols] = scaler.transform(val_inputs[numeric_cols])
test_inputs[numeric_cols] = scaler.transform(test_inputs[numeric_cols])

# We can now verify that values in each column lie in the range $(0,1)$

# In[64]:

train_inputs[numeric_cols].describe()

# Learn more about scaling techniques here: https://machinelearningmastery.com/standardscaler-and-minmaxscaler-transforms-in-python/

# ## Encoding Categorical Data
#
# Since machine learning models can only be trained with numeric data, we need to convert categorical data to numbers. A common technique is to use one-hot encoding for categorical columns.
#
#
#
# One hot encoding involves adding a new binary (0/1) column for each unique category of a categorical column.

# In[65]:

raw_df[categorical_cols].nunique()

# We can perform one hot encoding using the `OneHotEncoder` class from `sklearn.preprocessing`.

# In[66]:

from sklearn.preprocessing import OneHotEncoder

# In[67]:

#?OneHotEncoder

# In[68]:

encoder = OneHotEncoder(sparse=False, handle_unknown='ignore')

# First, we `fit` the encoder to the data i.e. identify the full list of categories across all categorical columns.

# In[69]:

encoder.fit(raw_df[categorical_cols])

# In[70]:

encoder.categories_

# The encoder has created a list of categories for each of the categorical columns in the dataset.
#
# We can generate column names for each individual category using `get_feature_names`.

# In[71]:

encoded_cols = list(encoder.get_feature_names_out(categorical_cols))
print(encoded_cols)

# All of the above columns will be added to `train_inputs`, `val_inputs` and `test_inputs`.
#
# To perform the encoding, we use the `transform` method of `encoder`.

# In[72]:

train_inputs[encoded_cols] = encoder.transform(train_inputs[categorical_cols])
val_inputs[encoded_cols] = encoder.transform(val_inputs[categorical_cols])
test_inputs[encoded_cols] = encoder.transform(test_inputs[categorical_cols])

# We can verify that these new columns have been added to our training, test and validation sets.

# In[73]:

pd.set_option('display.max_columns', None)

# In[74]:

test_inputs

# ## Saving Processed Data to Disk
#
# It can be useful to save processed data to disk, especially for really large datasets, to avoid repeating the preprocessing steps every time you start the Jupyter notebook. The parquet format is a fast and efficient format for saving and loading Pandas dataframes.

# In[75]:

print('train_inputs:', train_inputs.shape)
print('train_targets:', train_targets.shape)
print('val_inputs:', val_inputs.shape)
print('val_targets:', val_targets.shape)
print('test_inputs:', test_inputs.shape)
print('test_targets:', test_targets.shape)

# In[76]:

get_ipython().system('pip install pyarrow --quiet')

# In[77]:

train_inputs.to_parquet('train_inputs.parquet')
val_inputs.to_parquet('val_inputs.parquet')
test_inputs.to_parquet('test_inputs.parquet')

# In[78]:

get_ipython().run_cell_magic('time', '', "pd.DataFrame(train_targets).to_parquet('train_targets.parquet')\npd.DataFrame(val_targets).to_parquet('val_targets.parquet')\npd.DataFrame(test_targets).to_parquet('test_targets.parquet')")

# We can read the data back using `pd.read_parquet`.

# In[79]:

get_ipython().run_cell_magic('time', '', "\ntrain_inputs = pd.read_parquet('train_inputs.parquet')\nval_inputs = pd.read_parquet('val_inputs.parquet')\ntest_inputs = pd.read_parquet('test_inputs.parquet')\n\ntrain_targets = pd.read_parquet('train_targets.parquet')[target_col]\nval_targets = pd.read_parquet('val_targets.parquet')[target_col]\ntest_targets = pd.read_parquet('test_targets.parquet')[target_col]")

# Let's verify that the data was loaded properly.

# In[80]:

print('train_inputs:', train_inputs.shape)
print('train_targets:', train_targets.shape)
print('val_inputs:', val_inputs.shape)
print('val_targets:', val_targets.shape)
print('test_inputs:', test_inputs.shape)
print('test_targets:', test_targets.shape)

# In[81]:

val_inputs

# In[82]:

val_targets

# ## Training a Logistic Regression Model
#
# Logistic regression is a commonly used technique for solving binary classification problems. In a logistic regression model:
#
# - we take linear combination (or weighted sum of the input features)
# - we apply the sigmoid function to the result to obtain a number between 0 and 1
# - this number represents the probability of the input being classified as "Yes"
# - instead of RMSE, the cross entropy loss function is used to evaluate the results
#
#
# Here's a visual summary of how a logistic regression model is structured ([source](http://datahacker.rs/005-pytorch-logistic-regression-in-pytorch/)):
#
#
#
#
# The sigmoid function applied to the linear combination of inputs has the following formula:
#
#
#
# To train a logistic regression model, we can use the `LogisticRegression` class from Scikit-learn.

# In[83]:

from sklearn.linear_model import LogisticRegression

# In[84]:

#?LogisticRegression

# In[85]:

model = LogisticRegression(solver='liblinear')

# We can train the model using `model.fit`.

# In[86]:

get_ipython().run_cell_magic('time', '', 'model.fit(train_inputs[numeric_cols + encoded_cols], train_targets)')

# `model.fit` uses the following workflow for training the model ([source](https://www.deepnetts.com/blog/from-basic-machine-learning-to-deep-learning-in-5-minutes.html)):
#
# 1. We initialize a model with random parameters (weights & biases).
# 2. We pass some inputs into the model to obtain predictions.
# 3. We compare the model's predictions with the actual targets using the loss function.
# 4. We use an optimization technique (like least squares, gradient descent etc.) to reduce the loss by adjusting the weights & biases of the model
# 5. We repeat steps 1 to 4 till the predictions from the model are good enough.
#
#
#
#
# For a mathematical discussion of logistic regression, sigmoid activation and cross entropy, check out [this YouTube playlist](https://www.youtube.com/watch?v=-la3q9d7AKQ&list=PLNeKWBMsAzboR8vvhnlanxCNr2V7ITuxy&index=1). Logistic regression can also be applied to multi-class classification problems, with a few modifications.
#

# Let's check the weights and biases of the trained model.

# In[87]:

print(numeric_cols + encoded_cols)

# In[88]:

print(model.coef_.tolist())

# In[89]:

pd.DataFrame({
    'features':(numeric_cols + encoded_cols),
    'weights': model.coef_.tolist()[0]
})

# In[90]:

print(model.intercept_)

# #### Make bar plot to see the 'features' & 'weights' relation and then get the top 10 'features' with strong relation with 'weights'.

# In[91]:

# step 1

weiht_df = pd.DataFrame({
    'features':(numeric_cols + encoded_cols),
    'weights': model.coef_.tolist()[0]
})

# In[92]:

# step 2
plt.figure(figsize=(10,50))
sns.barplot(data=weiht_df,x='weights',y= 'features' )

# In[93]:

# Step 3 Get the top 10 features.

sns.barplot(data=weiht_df.sort_values('weights',ascending=False).head(10),x='weights',y= 'features' )

# Each weight is applied to the value in a specific column of the input. **Higher the weight, greater the impact of the column on the prediction**.

# ## Making Predictions and Evaluating the Model
#
# We can now use the trained model to make predictions on the training, test

# In[94]:

X_train = train_inputs[numeric_cols + encoded_cols]
X_val = val_inputs[numeric_cols + encoded_cols]
X_test = test_inputs[numeric_cols + encoded_cols]

# In[95]:

train_preds = model.predict(X_train)

# In[96]:

train_preds

# In[97]:

train_targets

# We can output a probabilistic prediction using `predict_proba`.

# In[98]:

train_probs = model.predict_proba(X_train)
train_probs

# The numbers above indicate the probabilities for the target classes "No" and "Yes".

# In[99]:

model.classes_

# We can test the accuracy of the model's predictions by computing the percentage of matching values in `train_preds` and `train_targets`.
#
# This can be done using the `accuracy_score` function from `sklearn.metrics`.

# In[100]:

from sklearn.metrics import accuracy_score

# In[101]:

accuracy_score(train_targets, train_preds)

# The model achieves an accuracy of 85.1% on the training set. We can visualize the breakdown of correctly and incorrectly classified inputs using a confusion matrix.
#
#

# In[102]:

from sklearn.metrics import confusion_matrix

# In[103]:

confusion_matrix(train_targets, train_preds, normalize='true')

# Let's define a helper function to generate predictions, compute the accuracy score and plot a confusion matrix for a given st of inputs.

# In[104]:

def predict_and_plot(inputs, targets, name=''):
    preds = model.predict(inputs)

    accuracy = accuracy_score(targets, preds)
    print("Accuracy: {:.2f}%".format(accuracy * 100))

    cf = confusion_matrix(targets, preds, normalize='true')
    plt.figure()
    sns.heatmap(cf, annot=True)
    plt.xlabel('Prediction')
    plt.ylabel('Target')
    plt.title('{} Confusion Matrix'.format(name));

    return preds

# In[105]:

train_preds = predict_and_plot(X_train, train_targets, 'Training')

# Let's compute the model's accuracy on the validation and test sets too.

# In[106]:

val_preds = predict_and_plot(X_val, val_targets, 'Validatiaon')

# In[107]:

test_preds = predict_and_plot(X_test, test_targets, 'Test')

# The accuracy of the model on the test and validation set are above 84%, which suggests that our model generalizes well to data it hasn't seen before.
#
# But how good is 84% accuracy? While this depends on the nature of the problem and on business requirements, a good way to verify whether a model has actually learned something useful is to compare its results to a "random" or "dumb" model.
#
# Let's create two models: one that guesses randomly and another that always return "No". Both of these models completely ignore the inputs given to them.

# In[108]:

def random_guess(inputs):
    return np.random.choice(["No", "Yes"], len(inputs))

# In[109]:

def all_no(inputs):
    return np.full(len(inputs), "No")

# Let's check the accuracies of these two models on the test set.

# In[110]:

accuracy_score(test_targets, random_guess(X_test))

# In[111]:

accuracy_score(test_targets, all_no(X_test))

# Our random model achieves an accuracy of 50% and our "always No" model achieves an accuracy of 77%.
#
# Thankfully, our model is better than a "dumb" or "random" model! This is not always the case, so it's a good practice to benchmark any model you train against such baseline models.

# > **EXERCISE**: Initialize the `LogisticRegression` model with different arguments and try to achieve a higher accuracy. The arguments used for initializing the model are called hyperparameters (to differentiate them from weights and biases - parameters that are learned by the model during training). You can find the full list of arguments here: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

# In[ ]:

# In[ ]:

# > **EXERCISE**: Train a logistic regression model using just the numeric columns from the dataset. Does it perform better or worse than the model trained above?

# In[ ]:

# In[ ]:

# > **EXERCISE**: Train a logistic regression model using just the categorical columns from the dataset. Does it perform better or worse than the model trained above?

# In[ ]:

# In[ ]:

# > **EXERCISE**: Train a logistic regression model without feature scaling. Also try a different strategy for missing data imputation. Does it perform better or worse than the model trained above?

# In[ ]:

# In[ ]:

# ## Making Predictions on a Single Input
#
# Once the model has been trained to a satisfactory accuracy, it can be used to make predictions on new data. Consider the following dictionary containing data collected from the Katherine weather department today.

# In[112]:

new_input = {'Date': '2021-06-19',
             'Location': 'Katherine',
             'MinTemp': 23.2,
             'MaxTemp': 33.2,
             'Rainfall': 10.2,
             'Evaporation': 4.2,
             'Sunshine': np.nan,
             'WindGustDir': 'NNW',
             'WindGustSpeed': 52.0,
             'WindDir9am': 'NW',
             'WindDir3pm': 'NNE',
             'WindSpeed9am': 13.0,
             'WindSpeed3pm': 20.0,
             'Humidity9am': 89.0,
             'Humidity3pm': 58.0,
             'Pressure9am': 1004.8,
             'Pressure3pm': 1001.5,
             'Cloud9am': 8.0,
             'Cloud3pm': 5.0,
             'Temp9am': 25.7,
             'Temp3pm': 33.0,
             'RainToday': 'Yes'}

# The first step is to convert the dictionary into a Pandas dataframe, similar to `raw_df`. This can be done by passing a list containing the given dictionary to the `pd.DataFrame` constructor.

# In[113]:

new_input_df = pd.DataFrame([new_input])

# In[114]:

new_input_df

# We've now created a Pandas dataframe with the same columns as `raw_df` (except `RainTomorrow`, which needs to be predicted). The dataframe contains just one row of data, containing the given input.
#
#
# We must now apply the same transformations applied while training the model:
#
# 1. Imputation of missing values using the `imputer` created earlier
# 2. Scaling numerical features using the `scaler` created earlier
# 3. Encoding categorical features using the `encoder` created earlier

# In[115]:

new_input_df[numeric_cols] = imputer.transform(new_input_df[numeric_cols])
new_input_df[numeric_cols] = scaler.transform(new_input_df[numeric_cols])
new_input_df[encoded_cols] = encoder.transform(new_input_df[categorical_cols])

# In[116]:

X_new_input = new_input_df[numeric_cols + encoded_cols]
X_new_input

# We can now make a prediction using `model.predict`.

# In[117]:

prediction = model.predict(X_new_input)[0]

# In[118]:

prediction

# Our model predicts that it will rain tomorrow in Katherine! We can also check the probability of the prediction.

# In[119]:

prob = model.predict_proba(X_new_input)[0]

# In[120]:

prob

# Looks like our model isn't too confident about its prediction!

# Let's define a helper function to make predictions for individual inputs.

# In[121]:

def predict_input(single_input):
    input_df = pd.DataFrame([single_input])
    input_df[numeric_cols] = imputer.transform(input_df[numeric_cols])
    input_df[numeric_cols] = scaler.transform(input_df[numeric_cols])
    input_df[encoded_cols] = encoder.transform(input_df[categorical_cols])
    X_input = input_df[numeric_cols + encoded_cols]
    pred = model.predict(X_input)[0]
    prob = model.predict_proba(X_input)[0][list(model.classes_).index(pred)]
    return pred, prob

# We can now use this function to make predictions for individual inputs.

# In[122]:

new_input = {'Date': '2021-06-19',
             'Location': 'Launceston',
             'MinTemp': 23.2,
             'MaxTemp': 33.2,
             'Rainfall': 10.2,
             'Evaporation': 4.2,
             'Sunshine': np.nan,
             'WindGustDir': 'NNW',
             'WindGustSpeed': 52.0,
             'WindDir9am': 'NW',
             'WindDir3pm': 'NNE',
             'WindSpeed9am': 13.0,
             'WindSpeed3pm': 20.0,
             'Humidity9am': 89.0,
             'Humidity3pm': 58.0,
             'Pressure9am': 1004.8,
             'Pressure3pm': 1001.5,
             'Cloud9am': 8.0,
             'Cloud3pm': 5.0,
             'Temp9am': 25.7,
             'Temp3pm': 33.0,
             'RainToday': 'Yes'}

# In[123]:

predict_input(new_input)

# > **EXERCISE**: Try changing the values in `new_input` and observe how the predictions and probabilities change. Try different values of location, temperature, humidity, pressure etc. Try to get an _intuitive feel_ of which columns have the greatest effect on the result of the model.

# In[124]:

raw_df.Location.unique()

# In[ ]:

# In[ ]:

# ## Saving and Loading Trained Models
#
# We can save the parameters (weights and biases) of our trained model to disk, so that we needn't retrain the model from scratch each time we wish to use it. Along with the model, it's also important to save imputers, scalers, encoders and even column names. Anything that will be required while generating predictions using the model should be saved.
#
# We can use the `joblib` module to save and load Python objects on the disk.

# In[125]:

import joblib

# Let's first create a dictionary containing all the required objects.

# In[126]:

aussie_rain = {
    'model': model,
    'imputer': imputer,
    'scaler': scaler,
    'encoder': encoder,
    'input_cols': input_cols,
    'target_col': target_col,
    'numeric_cols': numeric_cols,
    'categorical_cols': categorical_cols,
    'encoded_cols': encoded_cols
}

# We can now save this to a file using `joblib.dump`

# In[127]:

joblib.dump(aussie_rain, 'aussie_rain.joblib')

# The object can be loaded back using `joblib.load`

# In[128]:

aussie_rain2 = joblib.load('aussie_rain.joblib')

# Let's use the loaded model to make predictions on the original test set.

# In[129]:

test_preds2 = aussie_rain2['model'].predict(X_test)
accuracy_score(test_targets, test_preds2)

# As expected, we get the same result as the original model.

# ## Putting it all Together
#
# While we've covered a lot of ground in this tutorial, the number of lines of code for processing the data and training the model is fairly small. Each step requires no more than 3-4 lines of code.

# ### Data Preprocessing

# In[130]:

import opendatasets as od
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import MinMaxScaler, OneHotEncoder

# Download the dataset
od.download('https://www.kaggle.com/jsphyg/weather-dataset-rattle-package')
raw_df = pd.read_csv('weather-dataset-rattle-package/weatherAUS.csv')
raw_df.dropna(subset=['RainToday', 'RainTomorrow'], inplace=True)

# Create training, validation and test sets
year = pd.to_datetime(raw_df.Date).dt.year
train_df, val_df, test_df = raw_df[year < 2015], raw_df[year == 2015], raw_df[year > 2015]

# Create inputs and targets
input_cols = list(train_df.columns)[1:-1]
target_col = 'RainTomorrow'
train_inputs, train_targets = train_df[input_cols].copy(), train_df[target_col].copy()
val_inputs, val_targets = val_df[input_cols].copy(), val_df[target_col].copy()
test_inputs, test_targets = test_df[input_cols].copy(), test_df[target_col].copy()

# Identify numeric and categorical columns
numeric_cols = train_inputs.select_dtypes(include=np.number).columns.tolist()[:-1]
categorical_cols = train_inputs.select_dtypes('object').columns.tolist()

# Impute missing numerical values
imputer = SimpleImputer(strategy = 'mean').fit(raw_df[numeric_cols])
train_inputs[numeric_cols] = imputer.transform(train_inputs[numeric_cols])
val_inputs[numeric_cols] = imputer.transform(val_inputs[numeric_cols])
test_inputs[numeric_cols] = imputer.transform(test_inputs[numeric_cols])

# Scale numeric features
scaler = MinMaxScaler().fit(raw_df[numeric_cols])
train_inputs[numeric_cols] = scaler.transform(train_inputs[numeric_cols])
val_inputs[numeric_cols] = scaler.transform(val_inputs[numeric_cols])
test_inputs[numeric_cols] = scaler.transform(test_inputs[numeric_cols])

# One-hot encode categorical features
encoder = OneHotEncoder(sparse=False, handle_unknown='ignore').fit(raw_df[categorical_cols])
encoded_cols = list(encoder.get_feature_names(categorical_cols))
train_inputs[encoded_cols] = encoder.transform(train_inputs[categorical_cols])
val_inputs[encoded_cols] = encoder.transform(val_inputs[categorical_cols])
test_inputs[encoded_cols] = encoder.transform(test_inputs[categorical_cols])

# Save processed data to disk
train_inputs.to_parquet('train_inputs.parquet')
val_inputs.to_parquet('val_inputs.parquet')
test_inputs.to_parquet('test_inputs.parquet')
pd.DataFrame(train_targets).to_parquet('train_targets.parquet')
pd.DataFrame(val_targets).to_parquet('val_targets.parquet')
pd.DataFrame(test_targets).to_parquet('test_targets.parquet')

# Load processed data from disk
train_inputs = pd.read_parquet('train_inputs.parquet')
val_inputs = pd.read_parquet('val_inputs.parquet')
test_inputs = pd.read_parquet('test_inputs.parquet')
train_targets = pd.read_parquet('train_targets.parquet')[target_col]
val_targets = pd.read_parquet('val_targets.parquet')[target_col]
test_targets = pd.read_parquet('test_targets.parquet')[target_col]

# > **EXERCISE**: Try to explain each line of code in the above cell in your own words. Scroll back to relevant sections of the notebook if needed.

# ### Model Training and Evaluation
#

# In[131]:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import joblib

# Select the columns to be used for training/prediction
X_train = train_inputs[numeric_cols + encoded_cols]
X_val = val_inputs[numeric_cols + encoded_cols]
X_test = test_inputs[numeric_cols + encoded_cols]

# Create and train the model
model = LogisticRegression(solver='liblinear')
model.fit(X_train, train_targets)

# Generate predictions and probabilities
train_preds = model.predict(X_train)
train_probs = model.predict_proba(X_train)
accuracy_score(train_targets, train_preds)

# Helper function to predict, compute accuracy & plot confustion matrix
def predict_and_plot(inputs, targets, name=''):
    preds = model.predict(inputs)
    accuracy = accuracy_score(targets, preds)
    print("Accuracy: {:.2f}%".format(accuracy * 100))
    cf = confusion_matrix(targets, preds, normalize='true')
    plt.figure()
    sns.heatmap(cf, annot=True)
    plt.xlabel('Prediction')
    plt.ylabel('Target')
    plt.title('{} Confusion Matrix'.format(name));
    return preds

# Evaluate on validation and test set
val_preds = predict_and_plot(X_val, val_targets, 'Validation')
test_preds = predict_and_plot(X_test, test_targets, 'Test')

# Save the trained model & load it back
aussie_rain = {'model': model, 'imputer': imputer, 'scaler': scaler, 'encoder': encoder,
               'input_cols': input_cols, 'target_col': target_col, 'numeric_cols': numeric_cols,
               'categorical_cols': categorical_cols, 'encoded_cols': encoded_cols}
joblib.dump(aussie_rain, 'aussie_rain.joblib')
aussie_rain2 = joblib.load('aussie_rain.joblib')

# > **EXERCISE**: Try to explain each line of code in the above cell in your own words. Scroll back to relevant sections of the notebook if needed.

# ### Prediction on Single Inputs

# In[132]:

def predict_input(single_input):
    input_df = pd.DataFrame([single_input])
    input_df[numeric_cols] = imputer.transform(input_df[numeric_cols])
    input_df[numeric_cols] = scaler.transform(input_df[numeric_cols])
    input_df[encoded_cols] = encoder.transform(input_df[categorical_cols])
    X_input = input_df[numeric_cols + encoded_cols]
    pred = model.predict(X_input)[0]
    prob = model.predict_proba(X_input)[0][list(model.classes_).index(pred)]
    return pred, prob

new_input = {'Date': '2021-06-19',
             'Location': 'Launceston',
             'MinTemp': 23.2,
             'MaxTemp': 33.2,
             'Rainfall': 10.2,
             'Evaporation': 4.2,
             'Sunshine': np.nan,
             'WindGustDir': 'NNW',
             'WindGustSpeed': 52.0,
             'WindDir9am': 'NW',
             'WindDir3pm': 'NNE',
             'WindSpeed9am': 13.0,
             'WindSpeed3pm': 20.0,
             'Humidity9am': 89.0,
             'Humidity3pm': 58.0,
             'Pressure9am': 1004.8,
             'Pressure3pm': 1001.5,
             'Cloud9am': 8.0,
             'Cloud3pm': 5.0,
             'Temp9am': 25.7,
             'Temp3pm': 33.0,
             'RainToday': 'Yes'}

predict_input(new_input)

# ## Summary and References
#
# Logistic regression is a commonly used technique for solving binary classification problems. In a logistic regression model:
#
# - we take linear combination (or weighted sum of the input features)
# - we apply the sigmoid function to the result to obtain a number between 0 and 1
# - this number represents the probability of the input being classified as "Yes"
# - instead of RMSE, the cross entropy loss function is used to evaluate the results
#
#
# Here's a visual summary of how a logistic regression model is structured ([source](http://datahacker.rs/005-pytorch-logistic-regression-in-pytorch/)):
#
#
#
#
#
# To train a logistic regression model, we can use the `LogisticRegression` class from Scikit-learn. We covered the following topics in this tutorial:
#
# - Downloading a real-world dataset from Kaggle
# - Exploratory data analysis and visualization
# - Splitting a dataset into training, validation & test sets
# - Filling/imputing missing values in numeric columns
# - Scaling numeric features to a $(0,1)$ range
# - Encoding categorical columns as one-hot vectors
# - Training a logistic regression model using Scikit-learn
# - Evaluating a model using a validation set and test set
# - Saving a model to disk and loading it back
#
#
#
# Check out the following resources to learn more:
#
# * https://www.youtube.com/watch?v=-la3q9d7AKQ&list=PLNeKWBMsAzboR8vvhnlanxCNr2V7ITuxy&index=1
# * https://www.kaggle.com/prashant111/extensive-analysis-eda-fe-modelling
# * https://www.kaggle.com/willkoehrsen/start-here-a-gentle-introduction#Baseline
#
#
# Try training logistic regression models on the following datasets:
#
# - [Breast cancer detection](https://www.kaggle.com/uciml/breast-cancer-wisconsin-data): Predicting whether a tumor  is "benign" (noncancerous) or "malignant" (cancerous) using information like its radius, texture etc.
# - [Loan Repayment Prediction](https://www.kaggle.com/c/home-credit-default-risk) - Predicting whether applicants will repay a home loan based on factors like age, income, loan amount, no. of children etc.
# - [Handwritten Digit Recognition](https://www.kaggle.com/c/digit-recognizer) - Identifying which digit from 0 to 9 a picture of handwritten text represents.
#

# ## Revision Questions
# 1.	What is the Machine Learning workflow?
# 2.	What is supervised machine learning? Give an example.
# 3.	What is an unsupervised machine learning? Give an example.
# 4.	What is a semi-supervised machine learning? Give an example.
# 5.	What is Logistic regression?
# 6.	Why is it called Logistic regression?
# 7.	What is the function we use in Logistic regression?
# 8.	What is the difference between Linear regression and Logistic regression?
# 9.	What is a regression problem? Give some examples.
# 10.	What is a classification problem? Give some examples.
# 11.	What is a clustering problem? Give some examples.
# 12.	Why is it recommended to work with a sample when training a model?
# 13.	What is a train set?
# 14.	What is a validation set?
# 15.	Why is it recommended to split train data in to train-validation set?
# 16.	How do we impute missing data?
# 17.	Why do we scale numeric data? What are the different scalers?
# 18.	What is solver=’liblinear’ in LogisticRegression()?
# 19.	What is cross entropy loss function?
# 20.	What is accuracy score?
# 21.	What is a confusion matrix? How does it help in evaluating the model?
# 22.	What is a random model?
# 23.	How do you save model to the disk?

Logistic Regression/Australia Rain Data Set.py


#!/usr/bin/env python

import os
import sys
from config import Config, ConfigMerger
from sortedcontainers import SortedDict
import logging.handlers
import signal
import glob2
import psutil
import time
import subprocess
import requests
import traceback
import schedule
import daemon
import argparse

cfg_dir = 'cfg'
cfg_filename = 'main.cfg'

class Cam:
    cfg = Config()
    cam_cfg = []
    cam_cfg_resolver_dict = {}
    cam_streamer = []
    cam_streamer_pid = []
    cam_streamer_start_time = []
    cam_streamer_start_flag = []
    cam_streamer_poll_flag = []
    cam_capturer = []
    cam_capturer_pid = []
    cam_capturer_start_flag = []
    cam_capturer_check_flag = []
    log = logging.getLogger()
    log_handler_file = None
    main_loop_active_flag = True
    signals_name = {}

    def __init__(self, config_dir, config_filename, log_level=None):
        self.cfg_dir = config_dir
        self.cfg_filename = config_filename
        self.log_level = log_level
        self.cfg_file = os.path.join(self.cfg_dir, self.cfg_filename)

        self.read_main_config()
        self.create_dirs()
        self.setup_logging()
        self.pid_file = os.path.join(self.cfg['pid_dir'], self.cfg['pid_filename'])

        for sig in dir(signal):
            if sig.startswith('SIG') and not sig.startswith('SIG_'):
                self.signals_name[getattr(signal, sig)] = sig

    def read_main_config(self):
        if os.path.isfile(self.cfg_file):
            self.cfg.load(open(self.cfg_file))
        else:
            print('Failed to open the file: ' + self.cfg_file)
            sys.exit(1)

    def create_dirs(self):
        if not os.path.exists(self.cfg['log_dir']):
            os.makedirs(self.cfg['log_dir'])

        if not os.path.exists(self.cfg['pid_dir']):
            os.makedirs(self.cfg['pid_dir'])

    @staticmethod
    def get_log_level(log_level):
        log_levels = {
            'DEBUG': logging.DEBUG,
            'INFO': logging.INFO,
            'WARNING': logging.WARNING,
            'ERROR': logging.ERROR,
            'CRITICAL': logging.CRITICAL
        }

        if log_level is None:
            log_level = 'INFO'

        return log_levels.get(log_level.strip().upper(), logging.INFO)

    def setup_logging(self):
        if self.log_level is None:
            self.log_level = self.cfg['log_level']

        self.log_level = self.get_log_level(self.log_level)
        self.log.setLevel(self.log_level)

        logging_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')

        log_handler_stream = logging.StreamHandler()
        log_handler_stream.setFormatter(logging_formatter)
        log_handler_stream.setLevel(self.log_level)

        self.log_handler_file = logging.handlers.TimedRotatingFileHandler(
            os.path.join(self.cfg['log_dir'], self.cfg['log_filename']),
            when='midnight')
        self.log_handler_file.setFormatter(logging_formatter)
        self.log_handler_file.setLevel(self.log_level)

        self.log.addHandler(log_handler_stream)
        self.log.addHandler(self.log_handler_file)

        logging.getLogger('requests').setLevel(logging.WARNING)
        logging.getLogger('schedule').setLevel(logging.WARNING)
        sys.excepthook = self.exception_handler

    def write_main_pid(self):
        if os.path.isfile(self.pid_file):
            pid_file_content = open(self.pid_file, 'r').read()

            if len(pid_file_content.strip()):
                pid = int(pid_file_content)

                if psutil.pid_exists(pid):
                    print('Error. Already running, PID: %i' % pid)
                    sys.exit(1)

        open(self.pid_file, 'w').write(str(os.getpid()))

    def exit_handler(self, s, frame, log_signal=True, exit_code=0):
        if log_signal:
            self.log.warning('Caught signal: ' + self.signals_name[s])

        self.kill_cams_process()

        self.log.debug('Remove own PID file: ' + self.pid_file)
        if os.path.isfile(self.pid_file):
            os.remove(self.pid_file)
        else:
            self.log.warning('PID file not found: ' + self.pid_file)

        if exit_code == 0:
            self.main_loop_active_flag = False
        else:
            sys.exit(exit_code)

    def exit_child(self, s, frame, log_signal=True):
        while True:
            try:
                pid, status = os.waitpid(-1, os.WNOHANG)

                if pid != 0:
                    logging.warning('Received "%s" signal for "%s" PID with "%s" status' %
                                    (self.signals_name[s], pid, status))
                else:
                    break
            except ChildProcessError:
                self.log.debug('ChildProcessError: No child processes')
                break

    def exception_handler(self, *exception_data):
        self.log.critical('Unhandled exception:\n%s', ''.join(traceback.format_exception(*exception_data)))
        self.exit_handler(None, None, log_signal=False, exit_code=1)

    def kill_process(self, pid_file, remove_pid_file):
        pid = None

        if os.path.isfile(pid_file):
            pid_file_content = open(pid_file, 'r').read()
            self.log.debug('PID file content: "%s"' % pid_file_content)

            if len(pid_file_content.strip()):
                pid = int(pid_file_content)
                self.log.debug('Process id: %i' % pid)

                if psutil.pid_exists(pid):
                    self.log.debug('Kill process: %i' % pid)
                    try:
                        os.kill(pid, signal.SIGTERM)
                    except OSError:
                        self.log.warning('Failed to kill process: %i' % pid)
                else:
                    self.log.info('Process not found: %i' % pid)
            else:
                self.log.warning('PID is empty')

            if remove_pid_file:
                os.remove(pid_file)
        else:
            self.log.debug('Skip kill, PID file not found: ' + pid_file)

        return pid

    def kill_cam_processes(self, cam_index, cam_reset_flag=False, kill_streamer_flag=True, kill_capturer_flag=True):
        self.log.info('Stop cam: ' + self.cam_cfg[cam_index]['name'])

        if kill_capturer_flag:
            self.log.debug('Kill %s capturer' % self.cam_cfg[cam_index]['name'])
            self.kill_process(self.cam_capturer_pid[cam_index], True)

        if kill_streamer_flag:
            self.log.debug('Kill %s streamer' % self.cam_cfg[cam_index]['name'])
            self.kill_process(self.cam_streamer_pid[cam_index], True)

        if cam_reset_flag:
            try:
                self.cam_cfg[cam_index]['reset_cmd']
            except AttributeError:
                self.log.debug('Cam reset command not found. Skip reset')
            else:
                self.log.info('Resetting cam: ' + self.cam_cfg[cam_index]['name'])
                self.log.debug('Reset command: ' + self.cam_cfg[cam_index]['reset_cmd'])
                return_code = subprocess.call(self.cam_cfg[cam_index]['reset_cmd'], shell=True)
                self.log.info('Reseted with exit code: %s' % return_code)

    def kill_cams_process(self, cam_reset_flag=False):
        for iterator, _ in enumerate(self.cam_cfg):
            self.kill_cam_processes(iterator, cam_reset_flag=cam_reset_flag)

    def bg_run(self, cmd, pid_file=None):
        self.log.debug('Running:\n' + cmd)

        subproc = subprocess.Popen(cmd, shell=True,
                                   stdout=subprocess.DEVNULL,
                                   stdin=subprocess.DEVNULL,
                                   stderr=subprocess.DEVNULL)

        self.log.debug('Started PID: %s' % subproc.pid)

        if pid_file is not None:
            open(pid_file, 'w').write(str(subproc.pid))

        return subproc

    def replacer(self, s, cam_index):
        s = s.replace('[cam_name]', self.cam_cfg[cam_index]['name'])
        s = s.replace('[cams_number]', str(len(self.cam_cfg)))
        return s

    def cleaner(self):
        self.log.debug('Cleaner started')
        clean_flag = False
        store_file_list = None

        if int(self.cfg['cleaner_store_max_gb']) != 0:
            store_file_list = glob2.glob(os.path.join(self.cfg['cap_dir'], '**'))
            self.log.debug('Found files: %s' % store_file_list)

            store_file_total_size_bytes = 0
            for store_file in store_file_list:
                store_file_total_size_bytes += os.path.getsize(store_file)

            store_file_total_size_gigabytes = 1.0 * store_file_total_size_bytes / 1024 / 1024 / 1024
            self.log.debug('Store files size, Gb: %f' % store_file_total_size_gigabytes)

            if store_file_total_size_gigabytes > float(self.cfg['cleaner_store_max_gb']):
                self.log.info('Current store size / Configured max store size, Gb: %.3f/%.3f' %
                              (store_file_total_size_gigabytes, self.cfg['cleaner_store_max_gb']))
                clean_flag = True

        if int(self.cfg['cleaner_store_keep_free_gb']) != 0:
            store_stat = os.statvfs(self.cfg['cap_dir'])
            store_free_gb = 1.0 * store_stat.f_bavail * store_stat.f_frsize / 1024 / 1024 / 1024
            self.log.debug('Store free space, Gb: %f' % store_free_gb)

            if store_free_gb < float(self.cfg['cleaner_store_keep_free_gb']):
                self.log.info('Current store free space / Configured keep store free space, Gb: %.3f/%.3f' %
                              (store_free_gb, self.cfg['cleaner_store_keep_free_gb']))
                clean_flag = True

        if clean_flag:
            self.log.info('Clean is active')

            if store_file_list is None:
                store_file_list = glob2.glob(os.path.join(self.cfg['cap_dir'], '**'))

            store_file_list_sorted = SortedDict()
            for store_file in store_file_list:
                store_file_list_sorted.update({os.path.getmtime(store_file): store_file})

            self.log.debug('Sorted files list (with last modification date): %s' % store_file_list_sorted)
            self.log.debug('Sorted files list (by last modification date): %s' % store_file_list_sorted.values())

            removes = 0
            for file_name in store_file_list_sorted.values():
                if os.path.isfile(file_name):
                    file_size = os.path.getsize(file_name)
                    self.log.info('Remove file: ' + file_name)
                    os.remove(file_name)

                    if file_size > int(self.cfg['cleaner_force_remove_file_less_bytes']):
                        removes += 1
                    else:
                        self.log.warning('Removed "%s" file with the "%s" bytes size' % (file_name, file_size))

                    if removes == int(self.cfg['cleaner_max_removes_per_run']):
                        self.log.debug('Max removes reached: ' + self.cfg['cleaner_max_removes_per_run'])
                        break

        self.log.debug('Cleaner finished')

    def configs_resolver(self, map1, map2, key):
        self.cam_cfg_resolver_dict[key] = map1[key]
        return "overwrite"

    def main(self):
        self.log.info('Start')
        self.log.debug('Started: ' + os.path.abspath(__file__))
        self.log.debug('Setting SIGTERM, SIGINT, SIGCHLD handlers')
        signal.signal(signal.SIGTERM, self.exit_handler)
        signal.signal(signal.SIGINT, self.exit_handler)
        signal.signal(signal.SIGCHLD, self.exit_child)

        # Read cam configs
        cam_cfg_dir = os.path.join(self.cfg_dir, self.cfg['cam_cfg_mask'])
        self.log.debug('Configs search path: ' + cam_cfg_dir)

        cam_cfg_list = glob2.glob(os.path.join(self.cfg_dir, self.cfg['cam_cfg_mask']))
        cam_cfg_list.remove(self.cfg_file)
        self.log.debug('Found configs: %s' % cam_cfg_list)

        if len(cam_cfg_list) == 0:
            self.log.critical('No cam config found. Exit')
            sys.exit(0)

        for cur_cam_cfg in cam_cfg_list:
            self.log.debug('Read cam config: ' + cur_cam_cfg)
            tmp_cfg = Config(open(cur_cam_cfg))
            cur_cam_cfg_active_flag = True

            try:
                tmp_cfg['active']
            except AttributeError:
                self.log.debug('active flag not found')
            else:
                cur_cam_cfg_active_flag = tmp_cfg['active']

            if cur_cam_cfg_active_flag:
                self.cam_cfg.append(tmp_cfg)
                self.cam_cfg_resolver_dict.clear()
                merger = ConfigMerger(resolver=self.configs_resolver)
                merger.merge(self.cam_cfg[-1], self.cfg)

                for key in self.cam_cfg_resolver_dict:
                    self.cam_cfg[-1][key] = self.cam_cfg_resolver_dict[key]

                self.log.debug('Loaded settings for: ' + self.cam_cfg[-1]['name'])
            else:
                self.log.debug('Cam config is skipped due active flag: ' + cur_cam_cfg)
        # End Read cam configs

        # Cleaner
        self.cfg['cleaner_max_removes_per_run'] = self.replacer(str(self.cfg['cleaner_max_removes_per_run']), 0)
        schedule.every(self.cfg['cleaner_run_every_minutes']).minutes.do(self.cleaner)
        # End Cleaner

        # PIDs full path
        for iterator, cam in enumerate(self.cam_cfg):
            try:
                pid_streamer = cam['pid_streamer']
            except AttributeError:
                self.log.debug('pid_streamer not found for cam: ' + cam['name'])
                try:
                    pid_streamer = self.cfg['pid_streamer']
                except AttributeError:
                    self.log.critical("Can't find pid_streamer in config")
                    sys.exit(1)

            try:
                pid_capturer = cam['pid_capturer']
            except AttributeError:
                self.log.debug('pid_capturer not found for cam: ' + cam['name'])
                try:
                    pid_capturer = self.cfg['pid_capturer']
                except AttributeError:
                    self.log.critical("Can't find pid_capturer in config")
                    sys.exit(1)

            self.cam_streamer_pid.append(self.replacer(os.path.join(self.cfg['pid_dir'], pid_streamer), iterator))
            self.cam_capturer_pid.append(self.replacer(os.path.join(self.cfg['pid_dir'], pid_capturer), iterator))
        # End PIDs full path

        self.kill_cams_process()
        self.write_main_pid()

        while self.main_loop_active_flag:
            for iterator, cam in enumerate(self.cam_cfg):
                if len(self.cam_streamer) == iterator:

                    # Create cam cap dir only if cap_cmd is not False
                    try:
                        cap_cmd = self.cam_cfg[iterator]['cap_cmd']
                    except AttributeError:
                        cap_cmd = None
                        self.log.debug('Capture command not found')

                    if cap_cmd is not False:
                        cap_dir_cam = self.replacer(self.cfg['cap_dir_cam'], iterator)
                        if not os.path.exists(cap_dir_cam):
                            try:
                                os.makedirs(cap_dir_cam)
                            except OSError:
                                self.log.critical('Failed to create directory: ' + cap_dir_cam)
                                sys.exit(1)
                    # End Create cam cap dir

                    self.cam_streamer_start_flag.append(True)

                    self.cam_streamer.append(None)
                    self.cam_streamer_start_time.append(0)
                    self.cam_streamer_poll_flag.append(False)

                    self.cam_capturer.append(None)
                    self.cam_capturer_start_flag.append(False)
                    self.cam_capturer_check_flag.append(False)
                else:
                    if self.cam_streamer[iterator].poll() is None:
                        self.log.debug('Streamer "%s" is alive' % cam['name'])
                    else:
                        self.log.warning('Streamer "%s" is dead (exit code: %s)' %
                                         (cam['name'], self.cam_streamer[iterator].returncode))
                        self.cam_streamer_start_flag[iterator] = True

                # Capturer alive check
                if self.cam_capturer_check_flag[iterator]:
                    if self.cam_capturer[iterator].poll() is None:
                        self.log.debug('Capturer "%s" is alive' % cam['name'])
                    else:
                        self.log.warning('Capturer "%s" is dead (exit code: %s)' %
                                         (cam['name'], self.cam_capturer[iterator].returncode))
                        self.cam_streamer_poll_flag[iterator] = True
                        self.cam_capturer_check_flag[iterator] = False
                # End Capturer alive check

                # Run streamer
                if self.cam_streamer_start_flag[iterator]:
                    self.log.info('Run "%s" streamer in background' % cam['name'])
                    self.cam_streamer[iterator] = self.bg_run(cam['cmd'].strip(), self.cam_streamer_pid[iterator])
                    self.cam_streamer_start_time[iterator] = time.time()
                    self.cam_streamer_poll_flag[iterator] = True
                    self.cam_streamer_start_flag[iterator] = False
                # End Run streamer

                # Poll streamer
                if self.cam_streamer_poll_flag[iterator]:
                    cap_url = self.cfg['cap_url']
                    cap_url = self.replacer(cap_url, iterator)

                    self.log.debug('Getting HTTP status: ' + cap_url)
                    http_code = 0

                    try:
                        http_code = requests.head(cap_url, timeout=1).status_code
                    except requests.exceptions.RequestException:
                        self.log.warning('Failed to connect: ' + cap_url)

                    if http_code != 0:
                        self.log.info('Checked "%s", status: %s' % (cam['name'], http_code))

                        if http_code == 200:
                            self.cam_capturer_start_flag[iterator] = True
                            self.cam_streamer_poll_flag[iterator] = False

                    start_time_delta = time.time() - self.cam_streamer_start_time[iterator]
                    if self.cam_streamer_poll_flag[iterator]:
                        if start_time_delta > cam['max_start_seconds']:
                            self.log.warning('Time outed waiting data from: ' + cam['name'])
                            self.log.info('Kill: ' + cam['name'])
                            self.kill_cam_processes(iterator, cam_reset_flag=True)
                            self.cam_streamer_start_flag[iterator] = True
                        else:
                            self.log.info('Attempt "%s": [%i/%i]' %
                                          (cam['name'], start_time_delta, cam['max_start_seconds']))
                # End Poll streamer

                # Run capturer
                if self.cam_capturer_start_flag[iterator]:
                    if self.cam_capturer[iterator] is not None and self.cam_capturer[iterator].poll() is None:
                        self.log.warning('Capturer "%s" is STILL alive' % cam['name'])
                    else:
                        cap_cmd = None
                        try:
                            cap_cmd = self.cam_cfg[iterator]['cap_cmd']
                        except AttributeError:
                            self.log.debug('Capture command not found in cam config. Using global')
                            try:
                                cap_cmd = self.cfg['cap_cmd']
                            except AttributeError:
                                self.log.critical('Capture command not found. Exit')
                                self.exit_handler(None, None, log_signal=False, exit_code=1)

                        if cap_cmd is not False:
                            cap_cmd = self.replacer(cap_cmd, iterator)

                            self.log.info('Run "%s" capturer in background' % cam['name'])
                            self.cam_capturer[iterator] = self.bg_run(cap_cmd, self.cam_capturer_pid[iterator])
                            self.cam_capturer_check_flag[iterator] = True
                        else:
                            self.log.info('Capturer "%s" is turned off' % cam['name'])

                    self.cam_capturer_start_flag[iterator] = False
                # End Run capturer

            schedule.run_pending()
            time.sleep(1)

        self.log.info('Finish')

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-daemon', choices=['start', 'stop', 'restart'],
                        help='Daemon mode startup options')
    parser.add_argument('-log_level', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'],
                        help='Override config log_level')
    args = parser.parse_args()

    script_dir = os.path.dirname(os.path.abspath(__file__))
    os.chdir(script_dir)

    if args.log_level:
        c = Cam(cfg_dir, cfg_filename, args.log_level)
    else:
        c = Cam(cfg_dir, cfg_filename)

    if args.daemon:
        if args.daemon == 'stop' or args.daemon == 'restart':
            c.log.debug('[Daemon] Stopping')
            main_pid = c.kill_process(c.pid_file, False)

            if main_pid:
                kill_time = time.time()
                killed_flag = False

                timeout = 20

                while time.time() - kill_time < timeout:
                    if not psutil.pid_exists(main_pid):
                        killed_flag = True
                        break

                if not killed_flag:
                    c.log.warning('[Daemon] Time outed waiting process to exit (timeout: "%i" seconds). PID: "%i"' %
                                  (timeout, main_pid))

        if args.daemon == 'start' or args.daemon == 'restart':
            c.log.debug('[Daemon] Starting from working directory: ' + script_dir)
            with daemon.DaemonContext(working_directory=script_dir, files_preserve=[c.log_handler_file.stream]):
                c.main()
    else:
        c.main()


# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import migrations, models

class Migration(migrations.Migration):

    dependencies = [
        ('app', '0002_auto_20151129_1222'),
    ]

    operations = [
        migrations.AddField(
            model_name='feature',
            name='project',
            field=models.ForeignKey(to='app.Project', default=1),
            preserve_default=False,
        ),
    ]


#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Tests for the importer functions and classes."""

import os
import yaml
import xarray as xr
import numpy as np
import dask.array as da
from datetime import datetime
from satpy import DatasetID, Scene
from pyresample.geometry import AreaDefinition
from uwsift.workspace.importer import available_satpy_readers, SatpyImporter
from uwsift.common import Info

def test_available_satpy_readers_defaults():
    """Test default behavior of the satpy reader function."""
    readers = available_satpy_readers()
    assert isinstance(readers, list)
    assert len(readers) != 0
    assert isinstance(readers[0], str)  # names

def test_available_satpy_readers_as_dict():
    """Test asking for reader info as dictionaries."""
    readers = available_satpy_readers(as_dict=True)
    assert isinstance(readers, list)
    assert len(readers) != 0
    assert isinstance(readers[0], dict)

def test_available_satpy_readers_no_cache(tmpdir, monkeypatch):
    """Test loading the satpy readers when we know the cache is missing."""
    p = tmpdir.join('satpy_available_readers.yaml')
    monkeypatch.setattr('uwsift.workspace.importer.SATPY_READER_CACHE_FILE', str(p))
    readers = available_satpy_readers()
    assert isinstance(readers, list)
    assert len(readers) != 0
    assert isinstance(readers[0], str)

def test_available_satpy_readers_empty_cache(tmpdir, monkeypatch):
    """Test loading the satpy readers when the cache exists but is empty."""
    p = tmpdir.join('satpy_available_readers.yaml')
    with open(p, 'w') as cfile:
        yaml.dump({}, cfile)
    monkeypatch.setattr('uwsift.workspace.importer.SATPY_READER_CACHE_FILE', str(p))
    readers = available_satpy_readers()
    assert isinstance(readers, list)
    assert len(readers) != 0
    assert isinstance(readers[0], str)

def test_available_satpy_readers_known_cache(tmpdir, monkeypatch):
    """Test loading the satpy readers when the cache exists."""
    p = tmpdir.join('satpy_available_readers.yaml')
    with open(p, 'w') as cfile:
        yaml.dump({}, cfile)
    monkeypatch.setattr('uwsift.workspace.importer.SATPY_READER_CACHE_FILE', str(p))
    monkeypatch.setattr('uwsift.workspace.importer._SATPY_READERS', None)
    # create the cache
    _ = available_satpy_readers()
    mod_time1 = os.stat(p).st_mtime

    # load from the cache
    monkeypatch.setattr('uwsift.workspace.importer._SATPY_READERS', None)
    _ = available_satpy_readers()
    mod_time2 = os.stat(p).st_mtime
    assert mod_time1 == mod_time2, "Cache was not reused"

    # force recreation of the cache
    readers = available_satpy_readers(force_cache_refresh=True)
    mod_time3 = os.stat(p).st_mtime
    assert mod_time3 != mod_time1, "Cache was reused when it should have been recreated"
    assert isinstance(readers, list)
    assert len(readers) != 0
    assert isinstance(readers[0], str)

def _get_data_array_generator(data_arrs):
    """Help mimic what a real Satpy Scene would do."""
    yield from data_arrs

def test_satpy_importer_basic(tmpdir, monkeypatch, mocker):
    """Basic import test using Satpy."""
    db_sess = mocker.MagicMock()
    attrs = {
        'name': 'C01',
        'wavelength': (1.0, 2.0, 3.0),
        'area': AreaDefinition(
            'test', 'test', 'test',
            {
                'proj': 'geos',
                'sweep': 'x',
                'lon_0': -75,
                'h': 35786023,
                'ellps': 'GRS80',
                'units': 'm',
            }, 5, 5,
            (-5434894.885056, -5434894.885056, 5434894.885056, 5434894.885056)
        ),
        'start_time': datetime(2018, 9, 10, 17, 0, 31, 100000),
        'end_time': datetime(2018, 9, 10, 17, 11, 7, 800000),
        'standard_name': 'toa_bidirectional_reflectance',
    }
    data_arr = xr.DataArray(da.from_array(np.empty((5, 5), dtype=np.float64), chunks='auto'),
                            attrs=attrs)
    scn = Scene()
    scn['C01'] = data_arr
    scn.load = mocker.MagicMock()  # don't do anything on load

    imp = SatpyImporter(['/test/file.nc'], tmpdir, db_sess,
                        scene=scn,
                        reader='abi_l1b',
                        dataset_ids=[DatasetID(name='C01')])
    imp.merge_resources()
    assert imp.num_products == 1
    products = list(imp.merge_products())
    assert len(products) == 1
    assert products[0].info[Info.CENTRAL_WAVELENGTH] == 2.0
    assert products[0].info[Info.STANDARD_NAME] == 'toa_bidirectional_reflectance'


import os, argparse

def main():
    parser = argparse.ArgumentParser('Remove all files of according extension')
    parser.add_argument('--fold_A', dest='fold_A', help='input directory for image A', type=str, default='./A')
    parser.add_argument('--fold_B', dest='fold_B', help='input directory for image B', type=str, default='./B')
    parser.add_argument('--ext', dest='ext', help='file extension you want removed', type=str, default='.png')
    args = parser.parse_args()
    for root, dirs, files in os.walk(args.fold_A):
        for file in files:
            if file.endswith(args.ext):
                os.remove(os.path.join(root,file))
    for root, dirs, files in os.walk(args.fold_B):
        for file in files:
            if file.endswith(args.ext):
                os.remove(os.path.join(root,file))
    print(args.ext + "removed")

if __name__ == "__main__":
    main()


import pandas as pd
import numpy as np
import sklearn.model_selection
import sklearn.linear_model
import statsmodels.api as sm

df=pd.read_csv('C:/Users/mayij/Desktop/DOC/GITHUB/MLGH/linear regression/car.csv')
df['sell']=df['Selling_Price']
df['original']=df['Present_Price']
df['age']=2019-df['Year']
df['km']=df['Kms_Driven']/1000
df['petrol']=np.where(df['Fuel_Type']=='Petrol',1,0)
df['dealer']=np.where(df['Seller_Type']=='Dealer',1,0)
df['manual']=np.where(df['Transmission']=='Manual',1,0)
df=df[['sell','original','age','km','petrol','dealer','manual']].reset_index(drop=True)

df.plot(x='original',y='sell',style='o',title='sell vs original')

xtrain,xtest,ytrain,ytest=sklearn.model_selection.train_test_split(df[['original','age','km','petrol','dealer','manual']],df['sell'],test_size=0.2)
xtrain,xval,ytrain,yval=sklearn.model_selection.train_test_split(xtrain,ytrain,test_size=0.25)

moriginal=np.random.randint(0,1)
mage=np.random.randint(0,1)
mkm=np.random.randint(0,1)
mpetrol=np.random.randint(0,1)
mdealer=np.random.randint(0,1)
mmanual=np.random.randint(0,1)
c=np.random.randint(0,10)
a=0.0001
epochs=10000
n=len(xtrain)

for i in range(epochs):
    ypred=moriginal*xtrain['original']+mage*xtrain['age']+mkm*xtrain['km']+mpetrol*xtrain['petrol']+mdealer*xtrain['dealer']+mmanual*xtrain['manual']+c
    dmoriginal=(-2/n)*sum(xtrain['original']*(ytrain-ypred))
    dmage=(-2/n)*sum(xtrain['age']*(ytrain-ypred))
    dmkm=(-2/n)*sum(xtrain['km']*(ytrain-ypred))
    dmpetrol=(-2/n)*sum(xtrain['petrol']*(ytrain-ypred))
    dmdealer=(-2/n)*sum(xtrain['dealer']*(ytrain-ypred))
    dmmanual=(-2/n)*sum(xtrain['manual']*(ytrain-ypred))
    dc=(-2/n)*sum(ytrain-ypred)
    moriginal=moriginal-a*dmoriginal
    mage=mage-a*dmage
    mkm=mkm-a*dmkm
    mpetrol=mpetrol-a*dmpetrol
    mdealer=mdealer-a*dmdealer
    mmanual=mmanual-a*dmmanual
    c=c-a*dc

print('moriginal:'+str(moriginal)+'\n'+
      'mage:'+str(mage)+'\n'+
      'mkm:'+str(mkm)+'\n'+
      'mpetrol:'+str(mpetrol)+'\n'+
      'mdealer:'+str(mdealer)+'\n'+
      'mmanual:'+str(mmanual)+'\n'+
      'c:'+str(c))

sm.OLS(ytrain,sm.add_constant(xtrain)).fit().summary()


"""travelcave URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/2.2/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  path('', views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  path('', Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.urls import include, path
    2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
"""
from django.urls import path

from . import views

app_name = 'blog'

urlpatterns = [
    path('', views.index, name='home'),
    path('home/', views.index, name='home'),
    path('register/', views.register, name='register'),
    path('login/', views.login_view, name='login'),
    path('logout/', views.logout_view, name='logout'),
    path('posts/create/', views.create_post, name="create_post"),
    path('posts/publish/', views.publish_new_post, name='publish_new_post'),
    path('posts/<int:pk>/edit/', views.edit_post, name="edit_post"),
    path('posts/<int:pk>/publish/', views.publish_post, name="publish_post"),
    path('posts/<int:pk>/delete/', views.delete_post, name="delete_post"),
    path('posts/<int:pk>/view/', views.view_post, name="view_post"),
    path('posts/<int:pk>/like/', views.like_post, name="like_post"),
    path('posts/<int:pk>/unlike/', views.unlike_post, name="unlike_post"),
    path('posts/<int:pk>/comment/', views.comment_post, name="comment_post"),
    path('posts/<int:pk>/share/', views.share_post, name="share_post"),
    path('comment/<int:pk>/reply/', views.reply_comment, name='reply_comment'),
    path('account/posts/', views.my_posts, name='my_posts'),
    path('users/<str:username>/follow/', views.follow_user,
         name='follow_user'),
    path('users/<str:username>/unfollow/', views.unfollow_user,
         name='unfollow_user'),
    path('users/<str:username>/', views.view_user, name='view_user'),
    path('users/', views.users, name='users'),
    path('locations/<int:pk>/', views.view_location, name='view_location'),
    path('locations/<int:pk>/follow/', views.follow_location,
         name='follow_location'),
    path('locations/<int:pk>/unfollow/', views.unfollow_location,
         name='unfollow_location'),
    path('tags/<int:pk>/', views.view_tag, name='view_tag'),
    path('explore/', views.explore, name='explore'),
]


#!/usr/bin/python

import argparse
import bareos.bsock
from bareos.bsock.filedaemon import FileDaemon
import logging
import sys

def getArguments():
    argparser = argparse.ArgumentParser(description="Connect to Bareos File Daemon.")
    argparser.add_argument(
        "-d", "--debug", action="store_true", help="enable debugging output"
    )
    bareos.bsock.FileDaemon.argparser_add_default_command_line_arguments(argparser)
    argparser.add_argument(
        "command", nargs="*", help="Command to send to the Bareos File Daemon"
    )
    args = argparser.parse_args()
    return args

if __name__ == "__main__":
    logging.basicConfig(
        format="%(levelname)s %(module)s.%(funcName)s: %(message)s", level=logging.INFO
    )
    logger = logging.getLogger()

    args = getArguments()
    if args.debug:
        logger.setLevel(logging.DEBUG)

    bareos_args = bareos.bsock.FileDaemon.argparser_get_bareos_parameter(args)
    logger.debug("options: %s" % (bareos_args))
    try:
        bsock = FileDaemon(**bareos_args)
    except (bareos.exceptions.Error) as e:
        print(str(e))
        sys.exit(1)
    logger.debug("authentication successful")
    if args.command:
        print(bsock.call(args.command))
    else:
        bsock.interactive()


"""FastAPI routes for comments."""

from datetime import datetime, timezone
from typing import List

from bson import ObjectId
from fastapi import APIRouter, Depends, HTTPException, status
from fastapi.encoders import jsonable_encoder

from server.config import settings
from server.models.comments import Comment, CommentResponse, UpdateComment
from server.util import jwt
from server.util.common import check_course_in_db

router = APIRouter()

@router.post(
    "/teams/{course_name}/comments",
    description="Add a comment to a team.",
    status_code=status.HTTP_204_NO_CONTENT,
    responses={
        204: {"description": "The comment was added successfully"},
        500: {"description": "Internal server error"},
    },
    dependencies=[Depends(jwt.get_current_user_email)],
)
def add_comment(course_name: str, comment: Comment):
    """Add a TA comment for a team into the database."""
    coll = settings.database[course_name].comments
    comment_json = jsonable_encoder(comment)
    # Timestamp the comment.
    comment_json["created_at"] = comment_json["last_modified_at"] = datetime.now(
        timezone.utc
    ).isoformat()
    coll.insert_one(comment_json)

@router.get(
    "/teams/{course_name}/{sprint}/{team}/comments",
    description="Get comments for a team for a specific sprint. Specify sprint 0 to get all sprints",
    status_code=status.HTTP_200_OK,
    responses={
        200: {"description": "Comments are returned successfully"},
        404: {"description": "Course, sprint, or team not found"},
    },
    response_model=List[CommentResponse],
    dependencies=[Depends(jwt.get_current_user_email)],
)
def get_comments(course_name: str, sprint: int, team: str):
    """Get all comments for a team for a specific sprint."""
    check_course_in_db(course_name, [".comments", ".sprints"])

    if sprint == 0:
        # Get all comments for a team.
        comments_cur = settings.database[course_name].comments.find({"team": team})
    else:
        # First check if sprint exists.
        verify_sprint = settings.database[course_name].sprints.find_one(
            {"sprint_number": sprint}
        )
        if verify_sprint is None:
            raise HTTPException(
                status_code=status.HTTP_404_NOT_FOUND,
                detail=f"Sprint {sprint} does not exist",
            )

        comments_cur = settings.database[course_name].comments.find(
            {"$and": [{"sprint_number": sprint}, {"team": team}]}
        )

    comments = []
    for comment in comments_cur:
        comment_detail = CommentResponse(
            id=str(comment["_id"]),
            message=comment["message"],
            team=comment["team"],
            sprint_number=comment["sprint_number"],
            created_at=comment["created_at"],
            last_modified_at=comment["last_modified_at"],
        )
        comments.append(comment_detail)

    return comments

@router.patch(
    "/teams/{course_name}/comments/{comment_id}",
    description="Edit a comment for a team.",
    status_code=status.HTTP_204_NO_CONTENT,
    responses={
        204: {"description": "The comment was edited successfully"},
        400: {"description": "The ID was invalid"},
        404: {"description": "The course or comment was not found"},
        500: {"description": "Internal server error"},
    },
    dependencies=[Depends(jwt.get_current_user_email)],
)
def edit_comment(course_name: str, comment_id: str, comment: UpdateComment):
    """Edit a TA comment for a team in the database."""
    if not ObjectId.is_valid(comment_id):
        raise HTTPException(status_code=400, detail="Invalid comment ID")
    check_course_in_db(course_name, [".comments"])

    comment_json = jsonable_encoder(comment, exclude_none=True)
    # Timestamp the comment update.
    comment_json["last_modified_at"] = datetime.now(timezone.utc).isoformat()

    coll = settings.database[course_name].comments
    res = coll.update_one({"_id": ObjectId(comment_id)}, {"$set": comment_json})
    if res.matched_count == 0:
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND, detail="Comment not found"
        )

@router.delete(
    "/teams/{course_name}/comments/{comment_id}",
    description="Delete a comment for a team.",
    status_code=status.HTTP_204_NO_CONTENT,
    responses={
        204: {"description": "The comment was deleted successfully"},
        400: {"description": "The ID was invalid"},
        404: {"description": "The course or comment was not found"},
        500: {"description": "Internal server error"},
    },
    dependencies=[Depends(jwt.get_current_user_email)],
)
def delete_comment(course_name: str, comment_id: str):
    """Delete a TA comment for a team in the database."""
    if not ObjectId.is_valid(comment_id):
        raise HTTPException(status_code=400, detail="Invalid comment ID")
    check_course_in_db(course_name, [".comments"])

    coll = settings.database[course_name].comments
    res = coll.delete_one({"_id": ObjectId(comment_id)})
    if res.deleted_count == 0:
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND,
            detail="Comment not found",
        )


#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
 created by gjwei on 7/14/18

"""
import torch
import torch.nn as nn
import numpy as np
import torch.nn.functional as F
import ipdb

class CharCNNEmbedding(nn.Module):

    def __init__(self,
                 char_vocab,
                 char_dim,
                 word_max_length=10,
                 embed_dropout=0.2,
                 kernel_sizes=[3, 4, 5],
                 feature_maps=[128, 128, 128],
                 output_dim=128):
        super(CharCNNEmbedding, self).__init__()
        self.char_embedding = nn.Embedding(char_vocab, char_dim,
                                           padding_idx=0)
        self.embed_dropout = nn.Dropout(embed_dropout)
        self.cnns = nn.ModuleList(
                nn.Sequential(
                        nn.Conv1d(in_channels=char_dim,
                                  out_channels=fm,
                                  kernel_size=ks),
                        nn.Tanh(),
                        nn.MaxPool1d(kernel_size=word_max_length - ks + 1)
                )
                for fm, ks in zip(feature_maps, kernel_sizes)
        )
        self.dropout = nn.Dropout(0.5)

        self.output_layer = nn.Linear(sum(feature_maps), output_dim)

        self.init_weights()

    def init_weights(self):
        nn.init.normal_(self.char_embedding.weight, mean=0, std=0.01)
        nn.init.constant_(self.char_embedding.weight.data[0], 0.0)

        nn.init.xavier_normal_(self.output_layer.weight)

    def forward(self, x):
        """
        :param x: batch_size, seq_max_length, word_max_length
        :return: batch_size , seq_max_length, output_dim
        """
        batch_size, seq_max_length = x.size(0), x.size(1)
        x = x.view(-1, x.size(2))

        x = self.char_embedding(x)  # bs, ml, dim
        x = self.embed_dropout(x)

        x = x.permute(0, 2, 1)

        raw_cnn_ouputs = []

        for cnn in self.cnns:
            raw_cnn_ouputs.append(torch.squeeze(cnn(x)))

        # ipdb.set_trace()

        flatten = torch.cat(raw_cnn_ouputs, dim=-1)  # bs, sum(feature_maps)

        flatten = self.dropout(flatten)

        output = self.output_layer(flatten)  # bs output_dim

        output = output.view(batch_size, seq_max_length, -1)  # bs, seq_max_length, output_dim

        return output

if __name__ == '__main__':
    input = torch.from_numpy(np.random.randint(0, 99, size=(32, 20, 15)))
    model = CharCNNEmbedding(char_vocab=100, char_dim=50)
    output = model(input)

    print(output.size())


# -*- coding: utf-8 -*-
"""
Four Box Model Use Tutorial
=============

* **ACTM Performer:** JHU-APL;
* **Author:** Chace Ashcraft (chace.ashcraft@jhuapl.edu)

PyBAMOCS Tutorial

PyBAMOCS uses a simplified box model to simulate the Atlantic meridional overturning circulation (AMOC). The
implementation is based on the paper by Gnanadesikan et al.
(https://journals.ametsoc.org/view/journals/clim/31/22/jcli-d-18-0388.1.xml), which examined the stability of the AMOC.
This tutorial assumes that you have downloaded the PyBAMOCs code from the PACMANs repository on GitHub and installed
PyBAMOCS into your Python environment. Installation instructions for the PyBAMOCS module is provided below. For
additional information about installing PyBAMOCS and the PyBAMOCS API, please see the PACMANS repository
(https://github.com/JHUAPL/PACMANs), and navigate to `box_model/python/`.

This tutorial is meant to be a tool for getting started using the PyBAMOCS package and the included `box_model`
functionality, and is not meant to be a comprehensive overview of all functions or capabilities of the included code.
For a more thorough understanding, please consider viewing the code itself, and the examples in the `scripts` folder.
There is also a more extensive tutorial in the form of a Jupyter Notebook in the `notebooks` directory of the PyBAMOCS
module.

# PyBAMOCS Installation

We recommend installing PyBAMOCS in its own environment using `venv`
(https://docs.python.org/3/library/venv.html) or `conda`
(https://conda.io/projects/conda/en/latest/user-guide/install/index.html). Once in your new environment (if desired),
perform the following steps to install the module.

* Clone the PACMANS repository (https://github.com/JHUAPL/PACMANs)
    * `git clone https://github.com/JHUAPL/PACMANs.git`
* Change directories to `box_model/python/`
    * `cd box_model/python/`
* Install locally using `pip`
    * `pip install -e .`

To execute the snippet:

**Step 1:** Follow the instructions above to install the PyBAMOCS module.

**Step 2:** The snippet itself may be run from the command line as follows:

```
% python pybamocs_tutorial.py
```

We encourage the interested user to examine the code below to identify the relevant objects and methods required to
implement their specific use case.

"""

# .............................................
# IMPORT STATEMENTS
# .............................................

import time
from matplotlib import pyplot as plt
from typing import Union

# Import the `box_model` function and its arguments
from pybamocs.box_model import box_model
from pybamocs.box_model import NORTH_IDX, SOUTH_IDX, LOW_IDX, DEEP_IDX
from pybamocs.box_model_args import (
    BoxModelBoxDimensions,
    BoxModelInitConditions,
    BoxModelTimeStep,
    BoxModelParameters,
)

# Convenience function
def print_box_model_argument_settings(argument: Union[BoxModelBoxDimensions, BoxModelInitConditions,
                                                      BoxModelTimeStep, BoxModelParameters]) -> None:
    """
    Print box model settings to console for a given box_model argument.
    :param argument: One of the box_model argument objects.
    """
    settings_dict = argument.to_dict()  # Get the settings as a python dictionary

    print("vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv")
    # Print each key, value pair in the dictionary
    for key, value in settings_dict.items():
        print(f"{key}={value}")
    print("^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^")
    print()

def main():
    print("Box model settings are divided into five groups:")
    # Each object represents a set of parameters to the model
    box_model_dimensions = BoxModelBoxDimensions()
    box_model_initial_conditions = BoxModelInitConditions()
    box_model_time_settings = BoxModelTimeStep()
    box_model_parameters = BoxModelParameters()

    # And there is one object for each group as shown above.
    print("------------------------------------------------------------")
    print("Box Dimensions:")
    print_box_model_argument_settings(box_model_dimensions)

    print("------------------------------------------------------------")
    print("Initial Conditions")
    print_box_model_argument_settings(box_model_initial_conditions)

    print("------------------------------------------------------------")
    print("Model Time Step Parameters:")
    print_box_model_argument_settings(box_model_time_settings)

    print("------------------------------------------------------------")
    print("Model Parameters")
    print_box_model_argument_settings(box_model_parameters)

    # To run the box model, simply call the `box_model` function with your argument objects
    print("Running the box model...")
    start = time.time()
    results = box_model(box_model_dimensions,
                        box_model_initial_conditions,
                        box_model_parameters,
                        box_model_time_settings)
    print(f"Run complete. Time: {time.time() - start} seconds.")

    # Note that the results are contained in a BoxModelResult object
    print("Box model output object:")
    print(results)
    print()

    # You can access specific results directly from the object
    print("Result details:")
    print("------------------------------------------------------")
    print("Shape of M_n for run:", results.M_n.shape)
    print()

    # or you can "unpack" them all at once
    M_n, M_upw, M_eddy, D_low, T, S, sigma0 = results.unpack()

    print("------------------------------------------------------")
    print("First 3 values of each output variable:")
    print("M_n:", M_n[:3])
    print("M_upw:", M_upw[:3])
    print("M_eddy:", M_eddy[:3])
    print("D_low:", D_low[:3])
    print("T_north:", T[NORTH_IDX, :3])
    print("T_south:", T[SOUTH_IDX, :3])
    print("T_low:", T[LOW_IDX, :3])
    print("T_deep:", T[DEEP_IDX, :3])
    print("S_north:", S[NORTH_IDX, :3])
    print("S_south:", S[SOUTH_IDX, :3])
    print("S_low:", S[LOW_IDX, :3])
    print("S_deep:", S[DEEP_IDX, :3])
    print("sigma0_north:", sigma0[NORTH_IDX, :3])
    print("sigma0_south:", sigma0[SOUTH_IDX, :3])
    print("sigma0_low:", sigma0[LOW_IDX, :3])
    print("sigma0_deep:", sigma0[DEEP_IDX, :3])

    print("Plotting results...")
    plt.plot(M_n, label='M_n')
    plt.plot(M_upw, label='M_upw')
    plt.plot(M_eddy, label='M_eddy')
    plt.plot(D_low, label='Dlow')
    plt.legend()
    plt.title("1D Box Model Outputs")

    fig, ax = plt.subplots(nrows=2, ncols=2)
    ax[0, 0].plot(T[NORTH_IDX], label='North')
    ax[0, 1].plot(T[SOUTH_IDX], label='South')
    ax[1, 0].plot(T[LOW_IDX], label='Low')
    ax[1, 1].plot(T[DEEP_IDX], label='Deep')
    ax[0, 0].legend()
    ax[0, 1].legend()
    ax[1, 0].legend()
    ax[1, 1].legend()
    plt.suptitle("Temperature in the 4 different boxes")
    plt.tight_layout()

    fig1, ax1 = plt.subplots(nrows=2, ncols=2)
    ax1[0, 0].plot(S[NORTH_IDX], label='North')
    ax1[0, 1].plot(S[SOUTH_IDX], label='South')
    ax1[1, 0].plot(S[LOW_IDX], label='Low')
    ax1[1, 1].plot(S[DEEP_IDX], label='Deep')
    ax1[0, 0].legend()
    ax1[0, 1].legend()
    ax1[1, 0].legend()
    ax1[1, 1].legend()
    plt.suptitle("Salinity in the 4 different boxes")
    plt.tight_layout()

    fig2, ax2 = plt.subplots(nrows=2, ncols=2)
    ax2[0, 0].plot(sigma0[NORTH_IDX], label='North')
    ax2[0, 1].plot(sigma0[SOUTH_IDX], label='South')
    ax2[1, 0].plot(sigma0[LOW_IDX], label='Low')
    ax2[1, 1].plot(sigma0[DEEP_IDX], label='Deep')
    ax2[0, 0].legend()
    ax2[0, 1].legend()
    ax2[1, 0].legend()
    ax2[1, 1].legend()
    plt.suptitle("Density in the 4 different boxes")
    plt.tight_layout()
    plt.show(block=False)

    # Consider an example of some data collection
    Fwn_values_to_test = [10000, 50000, 100000, 500000, 1000000]
    alternate_north_starting_temp = 4.0

    box_dims = BoxModelBoxDimensions()
    params = BoxModelParameters()
    time_step = BoxModelTimeStep()
    init = BoxModelInitConditions(T_north0=alternate_north_starting_temp)

    # Collect the data... (should take a few seconds)
    print(f"Generating data for alternate parameter settings: T_north={alternate_north_starting_temp}")
    start_time = time.time()
    results = []
    for fwn in Fwn_values_to_test:
        print(f"Fwn={fwn}")
        params.Fwn = fwn
        results.append(box_model(box_dims, init, params, time_step))
    time_to_collect_data = time.time() - start_time
    print(f"Total time to collect data: {time_to_collect_data} seconds")

    # Let's look at the difference in M_n for each run
    print("Plotting results...")
    plt.figure()
    for i in range(len(Fwn_values_to_test)):
        plt.plot(results[i].M_n, label=f"Fwn={Fwn_values_to_test[i]}")
    plt.title(f"M_n for different Northern Fluxes and T_north0={alternate_north_starting_temp}")
    plt.legend()
    plt.show()  # suspends script until plot windows are closed
    print("Done")

if __name__ == "__main__":
    main()


#from django.http import HttpResponse
from django.shortcuts import render
import jieba

def home(request):
	return render(request,'home.html')

def count(request):
	txt=request.GET['text']
	dic={}
	text = jieba.lcut(txt)
	for word in text:
		dic[word]=dic.get(word,0)+1
	sortedDic=sorted(dic.items(),key=lambda x:x[1],reverse=True)
	length=len(text)
	return render(request,'count.html',{'lenText':length,'T':text,
		'D':dic,'S':sortedDic}
		)


from typing import Optional

import numpy as np
import scipy.sparse
from scipy.sparse import csr_matrix
from sqlalchemy import text

class Popularity:
    def __init__(self, window_size: int = 10, retrain_interval: int = 10):
        self.window_size = window_size
        self.retrain_interval = retrain_interval
        # Popularity matrix
        self.items_popularity: Optional[scipy.sparse.csr_matrix] = None

    def run(self, x: scipy.sparse.csr_matrix) -> Optional[csr_matrix]:
        return self.__fit(x).__predict(x)

    def retrain(self, date, dataset, k):
        # Return a query that'll return the Top-K most bought items of the past {window_size} days
        return text(
            f'SELECT item_id FROM (\n'
            f'SELECT item_id,\n' +
            f'       sum(item_count) OVER (PARTITION BY item_id ORDER BY purchase_date\n' +
            f'           ROWS BETWEEN {self.window_size} PRECEDING AND CURRENT ROW)\n' +
            f'           as total_interactions\n' +
            f'FROM items_day_count\n' +
            f'WHERE purchase_date = \'{date}\' AND dataset_name=\'{dataset}\'\n'
            f'ORDER BY total_interactions DESC LIMIT {k}) as tmp;'
        )

    def __fit(self, x: scipy.sparse.csr_matrix) -> 'Popularity':
        # Get popularity of each item
        items_popularity = np.asarray(x.sum(0))
        # Create index for each item
        col_index: np.ndarray = np.arange(items_popularity.shape[1])
        # Add index row as 2'nd row to array
        items_popularity: np.ndarray = np.vstack([items_popularity, col_index])
        # Sort array by values in first row in descending order
        result: np.ndarray = items_popularity[:, np.flip(np.argsort(items_popularity[0]))]
        # Applying window size
        result = result[:, :self.window_size]
        # result = np.resize(result, (result.shape[0], self.window_size))  # Cutting all products that aren't wanted
        self.items_popularity = result
        return self

    def __predict(self, histories: scipy.sparse.csr_matrix) -> Optional[csr_matrix]:
        return self.items_popularity

if __name__ == "__main__":
    a = Popularity(window_size=5)
    print(a.run(scipy.sparse.csr_matrix([[10, 2, 12], [1, 2, 3], [1, 2, 3], [5, 1, 8], [1, 9, 4], [9, 9, 9]])))