Tiny struggles

Python dla naukowca

• NumPy - wydajne typy danych oraz tablice i funkcje do operowania na nich
• SciPy - biblioteka różnych przydatnych naukowych funckji (fft, integrals, odes)
• Matplotlib - wykresy w 2D i 3D
• IPython - interaktywna konsola pythona, notebook i więcej
• SymPy - biblioteka do obliczeń symbolicznych (jak mathematica tylko otwarta i fajniejsza)
• pandas - biblioteka do operacji na danych (przypomina pakiet R)

NumPy

• User Guide
• typy danych
• tworzenie array'ów
• szybki kurs tentative numpy tutorial

Typy danych

Po co nam typy danych?

Jakie są wspierane?

Przykład:

>>> import numpy as np
>>> x = np.float32(1.0)
>>> x
1.0
>>> y = np.int_([1,2,4])
>>> y
array([1, 2, 4])
>>> z = np.arange(3, dtype=np.uint8)
>>> z
array([0, 1, 2], dtype=uint8)

źródło

Tworzenie tablic (1)

• z pythonowych obiektów
• wbudowane funkcje

>>> from numpy  import *
>>> a = array( [2,3,4] )
>>> a
array([2, 3, 4])
>>> a.dtype
dtype('int32')
>>> b = array([1.2, 3.5, 5.1])
>>> b.dtype
dtype('float64')

Tworzenie tablic (2)

Z listy list powstanie tablica dwuwymiarowa, z listy list list trzywymiarowa ;) i tak dalej.

>>> b = array( [ (1.5,2,3), (4,5,6) ] )
>>> b
array([[ 1.5,  2. ,  3. ],
       [ 4. ,  5. ,  6. ]])

Typ tablicy można podać przy tworzeniu:

>>> c = array( [ [1,2], [3,4] ], dtype=complex )
>>> c
array([[ 1.+0.j,  2.+0.j],
       [ 3.+0.j,  4.+0.j]])

Specjalne funkcje do tworzenia tablic:

>>> zeros( (3,4) )
array([[0.,  0.,  0.,  0.],
       [0.,  0.,  0.,  0.],
       [0.,  0.,  0.,  0.]])
>>> ones( (2,3,4), dtype=int16 )                # dtype can also be specified
array([[[ 1, 1, 1, 1],
        [ 1, 1, 1, 1],
        [ 1, 1, 1, 1]],
       [[ 1, 1, 1, 1],
        [ 1, 1, 1, 1],
        [ 1, 1, 1, 1]]], dtype=int16)
>>> empty( (2,3) )
array([[  3.73603959e-262,   6.02658058e-154,   6.55490914e-260],
       [  5.30498948e-313,   3.14673309e-307,   1.00000000e+000]])

Indeksowanie

>>> a = arange(10) ** 3
>>> a
array([  0,   1,   8,  27,  64, 125, 216, 343, 512, 729])
>>> a[2]
8
>>> a[2:5]
array([ 8, 27, 64])
>>> a[:6:2] = -1000  # equivalent to a[0:6:2] = -1000; from start to position 6, exclusive, set every 2nd element to -1000
>>> a
array([-1000,     1, -1000,    27, -1000,   125,   216,   343,   512,   729])
>>> a[ : :-1]                                 # reversed a
array([  729,   512,   343,   216,   125, -1000,    27, -1000,     1, -1000])

Algebra liniowa

• w numpy są zaimplementowane macierze, które są bardzo proste w użyciu
• poza tym można wykonywać obliczenia związane z algebrą liniową na zwykłych numpajowych tablicach: reference

Przykład kodu z macierzami

>>> A = mat([[3, 1, 4], [1, 5, 9], [2, 6, 5]])
>>> A = mat('3 1 4; 1 5 9; 2 6 5') # equivalent
>>> print A
[[3 1 4]
 [1 5 9]
 [2 6 5]]

>>> A
matrix([[3, 1, 4],
        [1, 5, 9],
        [2, 6, 5]])

>>> A.T # transpose
matrix([[3, 1, 2],
        [1, 5, 6],
        [4, 9, 5]])

>>> A.H # Hermitian transpose (same for real matrix)
matrix([[3, 1, 2],
        [1, 5, 6],
        [4, 9, 5]])

>>> A.I # matrix inverse
matrix([[ 0.32222222, -0.21111111,  0.12222222],
        [-0.14444444, -0.07777778,  0.25555556],
        [ 0.04444444,  0.17777778, -0.15555556]])

Scipy

scipy lecture notes, dokumentacja scipy, scipy cookbook

Co można zrobić używając scipy?

• scipy.cluster Vector quantization / Kmeans
• scipy.constants Physical and mathematical constants
• scipy.fftpack Fourier transform
• scipy.integrate Integration routines
• scipy.interpolate Interpolation
• scipy.io Data input and output
• scipy.linalg Linear algebra routines
• scipy.ndimage n-dimensional image package
• scipy.odr Orthogonal distance regression
• scipy.optimize Optimization
• scipy.signal Signal processing
• scipy.sparse Sparse matrices
• scipy.spatial Spatial data structures and algorithms
• scipy.special Any special mathematical functions
• scipy.stats Statistics

Matplotlib

Przykład kodu generaującego wykres funkcji

import numpy as np
import matplotlib.pyplot as plt


font = {'family' : 'serif',
        'color'  : 'darkred',
        'weight' : 'normal',
        'size'   : 16,
        }

x = np.linspace(0.0, 5.0, 100)
y = np.cos(2 * np.pi * x) * np.exp(-x)

plt.plot(x, y, 'k')
plt.title('Damped exponential decay', fontdict=font)
plt.text(2, 0.65, r'$\cos(2 \pi t) \exp(-t)$', fontdict=font)
plt.xlabel('time (s)', fontdict=font)
plt.ylabel('voltage (mV)', fontdict=font)

# Tweak spacing to prevent clipping of ylabel
plt.subplots_adjust(left=0.15)
plt.show()

Histogram

Kod generujący histogram

import numpy as np
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt


# example data
mu = 100 # mean of distribution
sigma = 15 # standard deviation of distribution
x = mu + sigma * np.random.randn(10000)

num_bins = 50
# the histogram of the data
n, bins, patches = plt.hist(x, num_bins, normed=1, facecolor='green', alpha=0.5)
# add a 'best fit' line
y = mlab.normpdf(bins, mu, sigma)
plt.plot(bins, y, 'r--')
plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.title(r'Histogram of IQ: $\mu=100$, $\sigma=15$')

# Tweak spacing to prevent clipping of ylabel
plt.subplots_adjust(left=0.15)
plt.show()

pandas

pandas in 10 minutes

Czym jest pandas?

Do czego się przydaje?

Sympy

• analiza z sympy
• obliczenia symboliczne
• całki i pochodne

Notebooki

• galeria interesujących notebooków
• nbviewer

ćwiczenie:

Opis problemu:

Pierwiastek A rozpada się na pierwiastek B, a pierwiastek B rozpada się do pierwiastka C. Pierwiastek C nie ulega rozpadowi radioaktywnemu.

A -> B -> C

Znamy stałą rozpadu lambda-a i lambda-b. Jak zamodelować taki proces?

Modelowanie w pythonie

Na przykładzie zombie apokalipsy

kod

Świetne strony:

• scipy lectures
• python crash course for scientists

Świetne blogi

• kwantowy python
• python perambulations

Python jest fantastyczny!

IPython? What's that?

IPython provides a rich architecture for interactive computing with:

• Powerful interactive shells (terminal and Qt-based).
• A browser-based notebook with support for code, text, mathematical expressions, inline plots and other rich media.
• Support for interactive data visualization
• Flexible, embeddable interpreters to load into your own projects.
• Easy to use, high performance tools for parallel computing.

Let's grab some screenshots, not to just talk about imaginative things:

Terminal client

Browser client (called notebook)

Czytaj więcej…

I feel like a cyborg

I love searching the net. It gives me inspiration, it feeds my desire for knowledge and entertainment, it make me feel connected to people I like or with with people with whom I share similar interests.

Being able to search and check information instantly is amazing. You don't have to remember things. Because you can always check them, and they will be more to date and probably more abundant than before.

Internet grows very fast. But how fast exactly? There are some statistics about amount of users and about traffic

Lack of connection

Today I came home after work and my internet connection was gone. It still is, actually. And I feel pretty awful, I feel strugling. But suddenly I have lot's of time, but it's just a regular evening.

One of symptoms of my deprivation is a feeling of innertion. I don't know, but it pretty hard to get things done. Sitting in front of my computer, I also have troubles getting things done. I tend to miss my working out routine, cleaning house, doing actual work etc. But I don't see it in such a clear manner, because I'm so engaged in the discussions over xmpp, mailing, browsing the news, or reading another newsletter, article or documentation page. And there is more and more tabs and apps opened.

I feel pretty overhelmed sometimes. I tend to get analog for a while, but not for long. It's a routine.

And where a connection doesn't work, my routine breaks.

Inner doubts

I don't really know, what is better for me. I love the internet, I love computers, I love instant knowledge, a power that software gives me, the enterntainment. But I'm only human. My capacity is constrained. I have not so much time for everything I would like to do. And engaging things tend to break my life balance. It's like a severe addiction.

But I know, that without the internet, I wouldn't be able to do, what I do, and I wouldn't be the same person. It's hard to say, but I often feel that I and probably lots of other people are not really prepared for richness of the internet. Our poor souls. But we have to learn or cease.

It's all about tradeoffs. But hopefully I would be able to make smart decisions.

Evolution

The internet changed my perception of the world. Using my computer approximately half a day (or more) changed my interaction with the world.

We should be scared with it, but embrace the advantages such as production of big data, which can give us better understanding of what we do, how we do this and why as a community or agents.

I hate closed science

I study physics at WUT. I want to read articles about subjects which interest me or just to learn if I'm really interested in them or not.

But I usually can't.

And I feel bad about it. Today I was interested in researching some various fields of physics, because I have to choose my next project. There are various options, many of which I haven't heard of before.

First step is usually googling new thing. There is something there usually. But there is a wall. Wall of paywall which is pretty outrageus. I want to check out 30 subjects (it's a number of basic options I have). But one article costs aproximatelly $40. How am I supposed to read this? I don't have 30 * $40 = $1200 just to check out some new fields of interest which could change my carrier.

With computer science/engineering it's a completely different matter. At least in my past experience, I could easily learn amazingly lot of different things without paying a cent. And I actually don't mind paying, if the sum isn't bizzare of course. But, as young person I didn't have many possibilities to earn enough money in the past. And my learning/researching desires are insatiable and usually imminent.

There are amazing resources about learning programming languages, you can learn this way python, erlang, haskell, lisp, lua and many others. Also - there are amazing courses at coursera, udacity, edx, and similar platforms.

Open science

There are some initiatives to open our closed science such as: open-science.pen.io, opensience.org

But shortly, what is open science:

Open science is the umbrella term of the movement to make scientific research, data and dissemination accessible to all levels of an inquiring society, amateur or professional. It encompasses practices such as publishing open research, campaigning for open access, encouraging scientists to practice open notebook science, and generally making it easier to publish and communicate scientific knowledge.

Quote from wikipedia.

Why is it important?

• Everybody should be able to check if what other people tell her is true or even make sense.

  • evaluation of methods
  • evaluation of data
  • evaluation of conclusions

• People can learn from other researchers

  • less reinventing the wheel
  • good practices emerge
    • know how of research
  • sharing tools

• Everyone should be able to easily find information about results, both summary and whole information needed to understand and reproduce given research

Without it:

• doing proper science is much more difficult for begginners and not affiliated groups
• there is a space for malpractice
• waste of resources (not reproducible research, lots of time spent on developing poor tools)
• unhealthy environment
• blocking development

What could we do?

• Make our research open.

• We need github for science.

• Publish on open access - and there is simply explained why.

• Create open notebooks.

From wikipedia:

Open Notebook Science is the practice of making the entire primary record of a research project publicly available online as it is recorded. This involves placing the personal, or laboratory, notebook of the researcher online along with all raw and processed data, and any associated material, as this material is generated. The approach may be summed up by the slogan 'no insider information'.

I love it. It's the way I would really enjoy to learn how experiments and research are done in such a way. I just didn't even know that such notebooks exit. Now I plan to use it for my future research to learn some good practices.

Summary

I started writing this article being outraged by closed science around me. Now I'm pretty excited about open science, which we can develop together.

Starting something new or just different is usually pretty hard. However, we are not alone, we have amazing tools to our disposal such as:

• github
• scipy stack
• R
• wikipedia

And lots of others, it would be impossible to list them all.

Probably the hardest thing is changing people's habits.

Remote with root access

It's just amazing to have your own remote server, where you can do whatever you like. I'm used to use servers set up by someone else, which is usually pretty comfortable - everything works and if not, someone else is worrying about it.

However, the day with my own servers has come.

Where

In short period of time i tried: Amazon AWS, OVH, DigitalOcean and Hetzner. I had problems on each of listed above. In the nearest future I'm going to set up a virtual machine on Linode.

EC2

Amazon AWS gives everyone free tier for one year. This is pretty cool. You can have a small instance for free, running all the time! However, bigger instances are getting pretty expensive, I think that I would recommend EC2 for ad hoc big needs, rather than normal servers.

Installing Ubuntu is pretty simple there. User interfaces aren't the most user friendly, but i'm sure, that you can handle them after reading the manuals (lots of manuals!).

OVH

They couldn't set up my machine. That sucks. I cancelled the deal after nearly a month of waiting.

Digital ocean

I was pretty unlucky and set an instance on which I couldn't ssh. I tried a lots of times. Further investigation proved the problem with Digital ocean routing. I asked support for help and got answer pretty soon. There was an error during setup. Changing location to Amsterdam helped.

Over all, the digital ocean droplet(their naming scheme for virtual servers is pretty strange, but you would quickly get used to it) just works nicely and I'm really satisfied for what I get, paying so little.

Hetzner

They try to be nice, but have some serious problems with their infrastructure. I need rock-solid service and they wouldn't able to provide it. I run into problems with my server after trying to install gentoo from rescue cd. It didn't boot. Probably some issues with network configuration.

Probably... I wasn't able to check. Their console, which would be a real salvation, if it worked, was just failing and failing. I had problems with everything, using interface, passwords and logging to my own dedicated server.

Rough times.

Is there any hope?

Overall it was a good idea to try all this providers. I gained some experience, I have configured instances where I can deploy whatever I want and just try things out.

I plan to try out linode now. But having a good, cheap instance on digital ocean is real win. I also enjoy my free tier on amazon aws (and other bonuses).

I also tried collocation by friends. Yep... my server 'skarpetka' hosted in Warsaw Hackerspace is currently down. God knows why.

One or many

Recently I've come accross issue of managment of side projects. I love creating new things and pursuing my ideas.

They are different, there are lots of them and they aren't usually extremely successful, but:

• I love creation
• I learn new things this way
• I build things which I can later on use or show (or laugh at ;)

However, sometimes I can't decide what I want to do. My projects aren't taken care of and I'm in doubt.

The more the merier

I prefer doing more than one projects because, I cannot be stuck on one thing for too long (asynchronous io). I can switch focus between them and gain fresh energy. It's more stimulating to have to tackle different kinds of problems.

Less is more

However it has many drawbacks such us:

• loss of focus
• projects not progressing (other projects take my time or I cannot do anything because I'm out of focus, whatever)
• tending to start infinite number of projects
• when other things are around it's hard to have time for them all

Discussion

I talked with a friend about having many projects. Actually he has one project and he has been spending time on it for at least half a year without any big sign of progress. This project actually totally doesn't appeal to me, it's an analog electronics projects with trivial application.

I couldn't do such thing. Simply, I would defenestrate the thing after few days without a real progress (maybe it's a reason, why I don't end many electronics projects).

I need many, various stimulating ideas, because getting bored or tired is a real issue for me.

It's not that I cannot focus on anything a bit deeper, it's not. I love reading about complicated stuff - mathematics, CS, physics. But I get sleepy If I pour into myself too much knowledge, I need changes, creation.

My friend argued about being able to focus. He claimed that real focus is possible with only one project.

It's a strange approach to me, because I was always tackling with many things at once (which was usually much more stressful than normal execution) and didn't have issues with changing focus.

However, recently I've fallen in love in taking my time. 8 hours in a row, mainly for one project is so damn awesome. The awareness, that I don't have to care about anything except from my projects is amazing.

Sadly, shortly I'll start my studies after a summer break and time without worries will be gone, but I plan to take my time on my projects (and real work and local hackerspace and lots of other important things... (focusing issues? - No!)) no matter what.

And there will be lot's of them.

My other friend has many projects and does amazing things with them. However he does not progress so fast, as one would like. He is a very bright man and has a wide variety of skills and interests.

Summary

Probably everyone has different project capacity. If you want to do only one, do one. But if you feel, that you lack something or feel dissatisfied, try different strategy. Maybe it's time to implement some concurrency pattern in your life?

So many task, so many different resources and possibilites, but noone said that life would be easier.

Why do I need a text editor?

I earn a living writing the code. I want to make my programming workflow as smooth as possible. I used several IDE's and text editors and I usually wasn't happy about my productivity and comfort during writing and developing.

Sophisiticated IDE's usually try to do too much and they aren't perfect in what they're doing. I used to develop on JVM and I used Netbeans, Intellij Idea and Eclipse and had problems on all of them. Some where rather tough and took a lot of my time. I agree that smart IDE's can be really helpful - especially with java - handling all dependecies, running your tests and app, and suggesting you what should you write - autocompletion, spotting your coding errors. It makes your life somewhat easier, but can make you pretty stupid. You can program and don't care about errors, just looking if there is something red there. You shouldn't! And running your code only in your ide separates you from your OS, which isn't good at all.

There is a simpler alternative - text editors. Let's begin with the best known ones - vim and emacs. I like Vim, but I never mastered it. Emacs scares me out. Maybe I'll give it a try one day. But not know - know I just want to write text.

Sublime text is a text editor for coders - but it's also very good for general purpose writing. I write this blogpost in it! I also write some latex here.

Get your free sublime text!

Package control

It's probably the first plugin you should install in your Sublime Text editor - if you haven't - go on this page and do it now, you won't regret.

It makes easy organizing your plugins. After installing type: 'Ctrl+Shift+P inst [tab]' and browse lots of available packages. Plugins in sublime text are pretty cool.

Shortcuts

The most important shortcuts in sublime starts with control key:

• Ctrl+Shift+P - command prompt
• Ctrl+P - go to any file (just type some words you are looking for in filepath)
• Ctrl+F - find in open file
• Ctrl+Shift+F - find in all files - opens file with results - snippets with line numbers you can just click and they redirect you to file with text you wanted to find
• Ctrl+G - go to line
• Ctrl+R - go to definition
• Alt+Number - switching between tabs

More of useful shortcuts

Snippets

Have you written your snippets already?

Snippets are small amounts of code you can write by triggering a trigger, they're great if you using some code over and over again like: "import ipdb; ipdb.set_trace()".

They are really easy to use, and you can use snippets written by somebody else to - I have lots of installed html snippets.

Summary

That's all for now. I wouldn't recommend installing too many plugins at once, because they can cause real problems with your editor - such as sudden segmentation faults or hanging.,

The course

Few weeks ago I started a hack design course. It looked like a startup-ish thing, but I thought that I would give it a chance... Yep, I'm a bit sceptical to the startup rave.

Course is relly nice. It gives a lot of insight for real beginners and is pretty easy to follow. The main material is a bunch of articles every week (however from time to time there is a video or interactive game/site).

There are some of my reflections about the design as a whole and the course.

Helvetica and world of typefaces

Typefaces are fun. There are many of them and some of them are really bad. Actually I wonder if it would be a good idea to have a website with a worst typefaces I could find. Like a font of the week, but instead of the best one, the worst one. It would be a real challenge to comic sans. You know what? I really liked comic sans when I was a kid, it looked relaxed and friendly, not so oficial and boring as Arial or Times New Roman in times when I used Microsoft Word.

Now I cannot see comic sans and not think that the design is broken. I also start to recognize the most popular and trendy fonts nowadays. Letters have more shape now, whatever it means. I started to think about it. There are so many possibilities to make something look somehow :). Inspiring.

I watched a documentary about helvetica. It's pretty brainwashing, but I liked it, however I don't like small a in helvetica. It isn't perfect, really. It's the worst letter in a set.

Design as a whole

However really important nowadays, design will always be pretty superficial to me. It has just not enough complexity and layers as I would want it to have, to make it interesting for me. It's an art thing. Art things are important and very hard to master, but they really aren't my field.

I would do some web design as a kind of relax activity, but making it a way of making a living for me would be a real disaster. However I don't really know why.

Importance of cryptography

I really haven't thought much about cryptography untill now. Also about a security as a whole. I treated paranoia of some of my sysops friend as just another quirk of their unusual behaviour or whatever.

I just push it out of my mind unconsiously. It's not my job. I don't know it, I don't really want to worry about and event try to think about what bad could happen. It would stress me out, wouldn't it? I'm not a good Dr Doom.

However it's extremly naive and I can see that. I encountered this behaviour in different fields so many times - I wanted to be in my comfort zone. Luckilly, my different part of personality loves challenges and rationalism too much to have stable comfort zone.

Recently I read an article on hackernews about cryptography and it inspired me, to change my attitude, learn about cryptography and think about security of systems I build and maintain.

Cryptography enineering

I started reading this book. I read first chapter and it sounded pretty sane and interesting too me. This book can be treated as a course material, there are even recomendations of how to divide in in 10, 12 or 6 weeks.

I enjoyed the text. It's pleasure to read. It's clear and well structured.

Excercises! Yes, this book offers nice excercises to practice your new acquired skills. I was surprised by the elaborate manual of writing report for excercise and at least a bit overhelmed by it and amount of excercises, because I'm naturally lazy and new too subject - so it's double hard. But I plan to take my time and do them, if not all, at least a representive half.

Thinking about methods to break securities and commit crimes without punishment and not feeling wrong about it (think like your enemy) could be really refreshing and insightful.

World isn't only books

The real test of thinking well in security domain would be building real system that would satisfy at least basic security demands.

I don't like most of mail clients

Why is it so? They are usually overcomplicated and break my workflow. My main mail client is Thunderbird (I haven't done complete transition yet). I use it to check mails from all of my email addresses. And there are actually lots of them, five that I use nearly everyday. My primary adress is a gmail one. I wanted to get rid of google gmail web app, but it was so much more intuitive and better designed than thunderbird, that I ceased to do it.

Actually thunderbird is a real riddle for me. I now how to change them, install plug-ins, create new events in calendar, configure facebook-chat and so on, but I am still unable to make my email browsing comfortable. A disaster!

Why not gmail then? I don't want to give all of my correspondence into google hands. I wouldn't feel good with it. Next thing is - browsing emails in web browser is pretty distracting for me. I want to chat to my friends, I click on google plus. I also tend to stay on google gmail page way to long.

I use KDE as primary windows manager and default mail client is a kmail. I admit that I haven't tried it. However I heard that it is similar to thunderbird(some of my friend claim that it is even worse), but I don't want to be forced to use a specific windows manager in the future, because I don't really know how would my future configuration look like.

It turns out that among my linux-using friends the highest esteem has a mutt.

What is mutt?

Mutt is a console mail client which is really good at what it does and nothing else. Its homepage isn't really appealing. More useful for beginners can be this overview

Actually to handle emails with mutt, you need something more than mutt. That might be surprising, because we are used to all-doing applications, but if we look at it from linux perspective it makes perfect sense.

You will need to:

• fetch your emails
• sort/categorize/filter your emails
• see your emails
• edit them
• send

and mutt of course :). Pretty much stuff and every element might fail.

Mutt with gmail

I wanted to configure my mutt to work with emails, that I get on my gmail account. I was looking for some easy solution in the internet, because it's much easier to start with something than doing everything from scratch.

This tutorial was most helpful for me. I did all those steps and ended up with pretty usable mutt but without my emails. The biggest problem for me was actually configuring procmail. It has regular expressions to sort emails and path to directories which aren't really intuitive - I've spent too much time to do it right and actually get some emails. One source of my problems was just plain ignorance - like using '~/' instead of '/home/me/'. This magic simply didn't work.

Elements

So I ended up with fetchmail, procmail, mutt, msmtp and some not so bad color configuration.

Some people recommend maildrop instead of procmail. I don't know yet what is better. However I know that procmail configuration syntax can be ugly and isn't intuitive at all, at least for me, so maybe in the future I would migrate to maildrop.

Conclusion

If you use your console terminal a lot, you would probably really enjoy mutt, because it's convenient to use in console pane/window, especially with tmux which I started using recently. Mutt is really nice and does what it is supposed to do. Everything is really intuitive and easy and to do most common task you only use some simple keystrokes, no more unnecessary clicking!