Mini Automation Project 2

Almost five years ago I have developed a mini automation project for automating turning things on and off at certain times. This project can be found here: http://www.etedal.net/2014/01/mini-automation-project.html

I have done a very similar project using BeagleBone and Python which the related code can be found here:
https://github.com/mahyaret/MiniAutomation

Raspberry Pi 4 as a HiFi Music Server

I've bought Raspberry Pi 4 to use it as a DaaS (Desktop as a Service). I've needed a VPN server to securely connect to when I am traveling. I've needed a Cloud Server to replace Dropbox to have unlimited space. I've needed a NAS (Network-attached storage) to share files among my laptops and computers. I've also needed a Music Player Server to manage my music files and serve as a source for my amplifier. I've also needed a media player like Apple TV to play my movies, youtube etc.

Note: the sound quality was not satisfying for me.

In summary Raspberry Pi as:
1. Cloud server
2. VPN server
3. NAS
4. Music server (this post) (not recommended)
5. Media Player

Raspberry Pi 4 as a Media Player

I've bought Raspberry Pi 4 to use it as a DaaS (Desktop as a Service). I've needed a VPN server to securely connect to when I am traveling. I've needed a Cloud Server to replace Dropbox to have unlimited space. I've needed a NAS (Network-attached storage) to share files among my laptops and computers. I've also needed a Music Player Server to manage my music files and serve as a source for my amplifier. I've also needed a media player like Apple TV to play my movies, youtube etc.

In summary Raspberry Pi as:
1. Cloud server
2. VPN server
3. NAS
4. Music server
5. Media Player (current post)

Raspberry Pi 4 as a NAS

I've bought Raspberry Pi 4 to use it as a DaaS (Desktop as a Service). I've needed a VPN server to securely connect to when I am traveling. I've needed a Cloud Server to replace Dropbox to have unlimited space. I've needed a NAS (Network-attached storage) to share files among my laptops and computers. I've also needed a Music Player Server to manage my music files and serve as a source for my amplifier. I've also needed a media player like Apple TV to play my movies, youtube etc.

In summary Raspberry Pi as:
1. Cloud server
2. VPN server
3. NAS (this post)
4. Music server
5. Media Player

Raspberry Pi 4 as a VPN Server

I've bought Raspberry Pi 4 to use it as a DaaS (Desktop as a Service). I've needed a VPN server to securely connect to when I am traveling. I've needed a Cloud Server to replace Dropbox to have unlimited space. I've needed a NAS (Network-attached storage) to share files among my laptops and computers. I've also needed a Music Player Server to manage my music files and serve as a source for my amplifier. I've also needed a media player like Apple TV to play my movies, youtube etc.

In summary Raspberry Pi as:
1. Cloud server
2. VPN server (this post)
3. NAS
4. Music server
5. Media Player

Raspberry Pi 4 as a Cloud Server

I've bought Raspberry Pi 4 to use it as a DaaS (Desktop as a Service). I've needed a VPN server to securely connect to when I am traveling. I've needed a Cloud Server to replace Dropbox to have unlimited space. I've needed a NAS (Network-attached storage) to share files among my laptops and computers. I've also needed a Music Player Server to manage my music files and serve as a source for my amplifier. I've also needed a media player like Apple TV to play my movies, youtube etc.

In summary Raspberry Pi as:
1. Cloud server (this post)
2. VPN server
3. NAS
4. Music server
5. Media Player

Create a Bootable Windows 7 or 10 USB Drive in Linux

source: https://thornelabs.net/posts/create-a-bootable-windows-7-or-10-usb-drive-in-linux.html


# fdisk /dev/sdY

create single partition type 7+bootable partition

# mkfs.ntfs -f /dev/sdY1
# ms-sys -7 /dev/sdY

# mount -o loop win7.iso /mnt/iso
# mount /dev/sdY1 /mnt/usb
# cp -r /mnt/iso/* /mnt/usb/

Updating modules using Elpy

When a self written module gets updated, reevaluating the buffer and the module in the python shell inside Emacs/Elpy doesn't get updated. For solving this issue, add the following to your Emacs configuration file:

(defun my-restart-python-console ()
  "Restart python console before evaluate buffer or region to avoid various uncanny conflicts, like not reloding modules even when they are changed"
  (interactive)
  (if (get-buffer "*Python*")
      (let ((kill-buffer-query-functions nil)) (kill-buffer "*Python*")))
  (elpy-shell-send-region-or-buffer))

(global-set-key (kbd "C-c C-x C-c") 'my-restart-python-console)

restart your Emacs run your code using ```C-c C-x C-c```

In short, this code has the "if clause" for checking if *Python* buffer is open. This will help to be able to run ```C-c C-x C-c``` at any time of development even when there is no Python process already open. Another part is ```kill-buffer-query-functions``` which neglects the prompt for killing the *Python* buffer.

Matplotlib plot not appearing with elpy in Emacs

For solving this, you can use different back-end:

import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt

Other GUI backends:

• TkAgg
• WX
• QTAgg
• QT4Agg

Run your code using: C-u C-c C-c

Fix tmux issue with conda

source
Problem: When running a conda environment and opening tmux on macOS, a utility called path_helper is run again. Essentially, the shell is initialized twice which messes up the ${PATH} so that the wrong Python version shows up within tmux.

Solution  If using bash, edit /etc/profile and add one line. (For zsh, edit /etc/zprofile)

Emacs as Python3 IDE in Mac

Install the following using your preferred method:

pip3 install rope jedi importmagic autopep8 flake8

In Emacs:

M-x package-install RET elpy RET
M-x package-install RET exec-path-from-shell RET
M-x package-install RET pyenv RET
M-x package-install RET anaconda-mode RET

Edit .emacs as following:

(elpy-enable)
(when (memq window-system '(mac ns x))
    (exec-path-from-shell-initialize))
(add-hook 'python-mode-hook 'anaconda-mode)
(setenv "WORKON_HOME" "/anaconda3/envs")
(pyvenv-mode 1)

Restart your emacs and enjoy using the best Python IDE

Use M-x pyvenv-workon to switch between anaconda environments

Mosh

Mosh is similar to SSH, remote terminal application but it allows roaming, and supports intermittent connectivity. Its usage is similar to SSH. It has to be installed both on the server and client.

On RedHat/CentOS remote server:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

sudo yum install mosh

sudo firewall-cmd --zone=public --permanent --add-port=60000-61000/udp

sudo firewall-cmd --reload

Azure:

If you're using Azure, make sure to allow the inbound port:

On local Mac:

brew install mosh

How to recover a shell after a disconnection?

It is not possible! but the cure is tmux. I start tmux, start the operation and go on my way. If I return and find the connection has been broken, all I have to do is reconnect and type tmux attach

For installing tmux on Mac:
   brew install tmux
tmux in the top image is running with vtop on the top pane and PM2 on the bottom.
Here is the cheatsheet for tmux:

Machine Learning vs. Optimization

Although it may be an odd comparison, sometimes this can be confusing for approaching problems. On one hand, mathematical optimization is used in machine learning during model training, when we are trying to minimize the cost of errors between our model and our data points. On the other hand, machine learning can be used to solve optimization problems. For many actual problems, the data cannot be known accurately for a variety of reasons. The first reason is due to simple measurement error. The second and more fundamental reason is that some data represent information about the future (e.g., product demand or price for a future time period) and simply cannot be known with certainty. The third reason is when obtaining the data is computationally intensive, even close to impossible; for instance, forecasting travel times between every two locations in a city. In these cases, we can use the power of machine learning to do the forecasting job for us. Or Robust optimization techniques can be used when the parameters are known only within certain bounds; the goal is to find a solution that is feasible for all data and optimal in some senses. Stochastic programming models take advantage of the fact that probability distributions governing the data are known or can be estimated; the goal is to find some policy that is feasible for all (or almost all) the possible data instances and optimize the expected performance of the model.

Use Azure as your Jupyter notebook Server

Make sure to follow the steps from this post to setup your Azure Virtual Machine:
http://www.etedal.net/2019/03/setting-up-azure-nc6-for-kaggle.html

Setup Jupyter remote ssh connection:

On Azure: jupyter notebook --no-browser --port=2001

On your machine: ssh -N -f -L localhost:2002:localhost:2001 remoteuser@remotehost

(Optional if you use mosh instead): mosh -ssh "ssh -N -f -L localhost:2002:localhost:2001" remoteuser@remotehost
On your browser: localhost:2002

Windows Subsystem for Debian

If you're frustrated using Windows and there is no way around not using it use the following:
https://docs.microsoft.com/en-us/windows/wsl/install-win10

Open PowerShell as Administrator:
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

Install Debian:
https://www.microsoft.com/en-ca/p/debian/9msvkqc78pk6?rtc=1&activetab=pivot:overviewtab

sudo apt update

Downloading Debian!

This is my favorite Debian:
http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/weekly-builds/amd64/iso-cd/

Setting up Azure NC6 for Kaggle Competitions

I have chosen NC6 and Ubuntu 18.04

Install the following using Lambda Stack:
Deep Learning frameworks: TensorFlow, Keras, PyTorch, Caffe, Caffe 2
GPU software: CUDA, cuDNN, NVIDIA drivers
Development tools: git, tmux, screen, vim, emacs, htop, valgrind, build-essential
Using Kaggle's beta API, you can interact with Competitions and Datasets to download data, make submissions, and more via the command line. 

Google Test (gtest) on Debian

Here is the link to the project source https://github.com/google/googletest

Here is the code for installing Google Test (gtest)

Initial Steps for Python

Change python version system-wide

This page has a great explanation for switching between Python and Python3:
https://linuxconfig.org/how-to-change-from-default-to-alternative-python-version-on-debian-linux

Installing Jupyter

Since in Python 3, ConfigParser has been renamed to configparser I switched to Python using:
Then for installing and running a Notebook: