Django migrate Sqllite to Mysql

Steps to migrate Django Sqllite DB to MySql DB

1. python manage.py dumpdata  -o datadump.json
2. Change settings.py to your mysql

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'database',
        'USER': 'username',
        'PASSWORD': 'password',
        'HOST': 'localhost',   # Or an IP Address that your DB is hosted on
        'PORT': '3306',
    }
}

3. Check you have mysqlclient, else run below command: 

       apt-get install python-dev libmysqlclient-dev 

       pip install mysqlclient

4. python manage.py migrate --run-syncdb5. python manage.py loaddata datadump.json

Google BigQuery - Calculating Minutes and Hours Differences

Assume you have a date column "OrderedDate", then we can calculate the total difference in minute and hour with respect to current date time:

SELECT
(CURRENT_TIMESTAMP() - TIMESTAMP_TO_USEC(OrderedDate)) / 1000000/ 3600 AS Diff_Hour,
 (CURRENT_TIMESTAMP() - TIMESTAMP_TO_USEC(OrderedDate)) / 1000000/ 60 AS Diff_Minute
FROM [Project:Dataset.Table]

General Copy Command to load S3 file data to Redshift

General Copy Command to load S3 file data to Redshift table:
COPY <tablename> from 's3://bucket/folder/file.txt'
CREDENTIALS 'aws_access_key_id=**********;aws_secret_access_key=*********'
DELIMITER   '|' IGNOREHEADER 1;

AWS Redshift: 'String contains invalid or unsupported UTF8 codepoints. Bad UTF8 hex sequence: b6

When you receive below error message from AWS redshift while executing use copy command then use 'ACCEPTINVCHARS ESCAPE' syntax in copy command:

Error Code: 'String contains invalid or unsupported UTF8 codepoints. Bad UTF8 hex sequence: b6 (error 3) '

COPY <tablename> from 's3://bucket/folder/file.txt'
CREDENTIALS '**********;aws_secret_access_key=******'
DELIMITER   '|' ACCEPTINVCHARS ESCAPE IGNOREHEADER 1;

AWS String contains invalid or unsupported UTF8 codepoints. Bad UTF8 hex sequence: b6

When you receive below error message from AWS redshift while executing use copy command then use 'ACCEPTINVCHARS ESCAPE' syntax in copy command:

Error Code: 'String contains invalid or unsupported UTF8 codepoints. Bad UTF8 hex sequence: b6 (error 3) '

COPY <tablename> from 's3://bucket/folder/file.txt'
CREDENTIALS '**********;aws_secret_access_key=******'
DELIMITER   '|' ACCEPTINVCHARS ESCAPE IGNOREHEADER 1;

General COPY command to load data from S3 file

General Copy Command trom to load S3 file data to Redshift table:
COPY <tablename> from 's3://bucket/folder/file.txt'
CREDENTIALS 'aws_access_key_id=**********;aws_secret_access_key=*********'
DELIMITER   '|' IGNOREHEADER 1;

PYODBC Connection in Python

Creating ODBC Connection in Python

STEP 1: Installing pypyodbc can be done via the commandline:

           >>C:\Python34\Scripts>pip install pypyodbc


Using the below command in your code for better naming conventions:

>> import pypyodbc as pyodbc