Category Archives: Generic

Postgres Fedora 28 Workstation

Posted on by

This is is probably not the best way to fix the problem I am having.

I tend to use postgreSQL when I an faffing with some data. The data I am usually faffing with is huge and on Fedora the problem is that the var directory is in the 50Gb of data provided as part of a separate disk partition used to hold the system files. Don’t know why, but there you go.

So ideally I need to move the data files to somewhere else.

Install postgreSQL from the nice people over at PostgreSQL. Once installed make sure that the postgres user exists.
Create a folder for the user in the “/home/” directory and chown it to the user using:

chown postgres:postgres /home/postgres

…then change the postgres-10.service file using:

sudo su
systemctl edit --full postgresql-10.service

…change the Environment line to use your path as the PGDATA:

Environment=PGDATA=/home/postgres/DATA

then run:

/usr/pgsql-10/bin/postgresql-10-setup initdb

…and make sure that the service is running using:

sudo systemctl enable postgres-10.service
sudo systemctl start postgres-10.service
sudo systemctl status postgres-10.service

Once you have it up and running you can upload files (from somewhere that postgres user can access – like tmp) using:

psql -d police postgres -c "\copy street FROM '/tmp/streetdata.tsv' delimiter E'\t' csv header"

Fedora Extra Backgrounds

Posted on by


sudo dnf install f23-backgrounds-extras-gnome f24-backgrounds-extras-gnome f25-backgrounds-extras-gnome f26-backgrounds-extras-gnome f27-backgrounds-extras-gnome f28-backgrounds-extras-gnome

Some Basic DNF

Posted on by 
sudo dnf repolist
sudo dnf config-manager --enablerepo=fedora-multimedia
sudo dnf config-manager --disablerepo=fedora-multimedia
sudo dnf upgrade --exclude=mono*

That last one will ignore all mono packages from your currently identified upgrade list.

NuGet on Rider C# IDE from JetBrains

Posted on by

You can find NuGet under the Tools menu, then select NuGet and then Manage Nuget Packages.

Once you have the packages dialog open type in the name of the package, choose the package you want and use the green cross button (on the right hand side) to add it to your project.
That should add it to your project; nice and easy.

Wales Pharmacy Data. Load unto mariadb.

Posted on by

You can download some data from here.
Unzip the files into your data directory. The GPDataYYYYMM.csv files are the ones you want.
This assumes you have mariadb installed and all working OK.

Unfortunately the structure of the csv files changes between 2013 to 2015 has one set of columns and 2016 onward have another set.
as per Blue Peter I’ve created a database called pharmacy before hand.

create table pharmacy.GPData_OLD
(
	HB text null,
	Locality text null,
	PracticeID text null,
	BNFCode text null,
	BNFName text null,
	Items int null,
	NIC double null,
	ActCost double null,
	Quantity int null,
	Period int null
);

The new structure has a couple of extra columns: DDD and ADQ.

create table pharmacy.GPData_NEW
(
	HB text null,
	Locality text null,
	PracticeID text null,
	BNFCode text null,
	BNFName text null,
	Items int null,
	NIC double null,
	ActCost double null,
	Quantity int null,
	DDD double null,
	ADQ double null,
	Period int null
);

Now we’ve created the tables we can use a bash script to load all the csv files into the tables we’ve just knocked up.
This from something pulled from Stack Overflow:

for f in /DATA/mysql/pharmacy/GPData201[345]*.csv
do
    mysql -e "LOAD DATA INFILE '"$f"' INTO TABLE  GPData_OLD FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES" -u root --password="yourrootpassword" pharmacy
echo "Done: '"$f"' at $(date)"
done

save it as something called “something” and then run it by invoking like ./something. Remember chmod u+x something

for f in /DATA/mysql/pharmacy/GPData201[67]*.csv
do
    mysql -e "LOAD DATA INFILE '"$f"' INTO TABLE  GPData201706 FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES" -u root --password="TreeOfLife123$" pharmacy
echo "Done: '"$f"' at $(date)"
done

as something like “something2016” invoking with ./something2016 again make it run-able using chmod u+x something2016

Chug chug chug and you get something like 33 million rows.

Moving Files By Name

Posted on by

…where the name is made up of the date and time.

I’ve got a lot of image files that I tend to dump into whatever directory I feel like.
This simple python script will create directories and move files around based on the file name, as long as the filename follows the following format YYYYMM. Bascially as long as the first 6 characters are integers it should work. In my file system that’s fine In yours the same may not be true. If the first 6 characters is not an integer then it will leave them alone.

Notice I’m not checking for JPG/MP4 files only; I’m not checking that the year or month are valid values; I’m not checking the EXIF values and basing it on them if the filename is not valid; there are lots of really cool things I could do as well but I don’t need those because that’s how my files are set out. I’m not building in lots of features that don’t touch my use-case.

import os
import shutil

path = '/home/mike/TestPython/'
os.chdir(path)

for file in os.walk(path):
    for name in file[2]:
        year = name[:4]
        month = name[4:6]
        if name[:6].isdigit():
            if not os.path.exists(year):
                os.makedirs(year)
            if not os.path.exists(year + "/" + month):
                os.makedirs(year + "/" + month)
            src = file[0] + "/" + name
            dst = path + year + "/" + month + "/"+name
            shutil.move(src, dst)

Once you’ve moved all the files into their new homes delete all the empty directories. This code has been ever so slightly modifed from this StakOverflow post. It was modified for Python 3, by adding brackets to the print syntax:

import os
currentDir = '/home/mike/TestPython/'

index = 0
for root, dirs, files in os.walk(currentDir):
    for dir in dirs:
        newDir = os.path.join(root, dir)
        index += 1
        print (str(index) + " ---> " + newDir)
        try:
            os.removedirs(newDir)
            print("Directory empty! Deleting...")
            print(" ")
        except:
            print("Directory not empty and will not be removed")
            print(" ")